Remote driving control method, remote driving cabin and remote driving control system

Abstract

The invention relates to a remote driving control method, a remote driving cabin and a remote driving control system, and relates to the technical field of mining mechanical equipment. The method comprises the following steps: in response to a connection operation on a target unmanned vehicle, acquiring field information collected by the target unmanned vehicle; and further performing safety analysis based on the field information to enter a takeover mode of the target unmanned vehicle under the condition that the target unmanned vehicle meets takeover conditions. According to the method, on the basis of preliminary connection, on the basis of on-site information of the unmanned vehicle, whether takeover conditions are met or not is judged, then takeover is executed to achieve safe and reliable remote control, and complex working conditions can be better dealt with; and remote control can be performed when parking and vehicle pressing problems caused by poor road conditions or vehicle faults occur, field operation of operators is not needed, and the production safety and the transportation efficiency of the mining area are improved on the basis that the unmanned degree of the mining area is kept.

Description

Background Art

[0002] The remote monitoring and operation of unmanned transport vehicles in mining environments such as open-pit mines and underground mines belongs to the intersection of mining intelligent equipment technology, vehicle automatic control technology, remote control technology and artificial intelligence technology.

[0003] At present, in order to meet the needs of intelligent and unmanned mining operations, unmanned driving has gradually become the mainstream of mining transportation, which not only includes the unmanned driving function of vehicles with cockpits, but also includes the automatic driving function of vehicles without cockpits.

[0004] However, mining transportation has the characteristics of long routes, harsh environment, complex road conditions, and frequent changes in production processes. Unmanned driving cannot adapt to and handle some roads, sites and emergencies in a timely manner, and sometimes requires on-site intervention by operators, affecting production safety and mining transportation efficiency. Summary of the invention

[0005] The purpose of the present disclosure is to provide a remote driving control method, a remote driving cockpit and a remote driving control system, which can improve production safety and mining transportation efficiency while maintaining the degree of unmanned operation in the mining area.

[0006] According to a first aspect of the present disclosure, a remote driving control method is provided, which may include: in response to a connection operation on a target unmanned vehicle, obtaining on-site information collected by the target unmanned vehicle; performing a safety analysis based on the on-site information, and entering a takeover mode for the target unmanned vehicle when the target unmanned vehicle meets takeover conditions.

[0007] Optionally, the takeover condition includes at least one of the following:

[0008] The real-time speed of the target unmanned vehicle is 0;

[0009] The target unmanned vehicle is in N gear;

[0010] The target unmanned vehicle's handbrake is in the engaged state;

[0011] The target unmanned vehicle is in unmanned mode.

[0012] Optionally, the takeover mode includes at least one of the following:

[0013] Control the speed of the target unmanned vehicle to be less than or equal to the remote control speed threshold;

[0014] Obstacle distance recognition is performed based on on-site information. When the distance between the target unmanned vehicle and the obstacle is less than or equal to the remote control distance threshold, the target unmanned vehicle is controlled to stop suddenly.

[0015] When the transmission delay of the on-site information is greater than or equal to the network delay threshold, the control exits the takeover mode of the target unmanned vehicle;

[0016] displaying the on-site information in a rearview mirror mode;

[0017] The on-site information is displayed in a reverse mode.

[0018] Optionally, the on-site information includes front-view video, rear-view video and basic vehicle information.

[0019] Optionally, the basic vehicle information includes at least one of vehicle speed, gear position, speed, steering wheel angle, handbrake status, tire pressure, lighting information, remaining fuel, pitch angle, roll angle, brake pressure, brake sign, lift angle, emergency stop switch status, remaining urea amount, and vehicle number information.

[0020] Optionally, the rearview mirror mode includes displaying the rear-view video in the center of the display interface and displaying the front-view video in a small window at the corner of the display interface when the target unmanned vehicle is in the R gear; and displaying the front-view video in the center of the display interface and displaying the rear-view video in a small window at the corner of the display interface in any other gear position; wherein the basic vehicle information is displayed together with the front-view video.

[0021] Optionally, the reversing mode includes displaying the front video in the center of the display interface when the target unmanned vehicle is in the R gear, displaying the rear video in a small window in the corner of the display interface, and not displaying the rear video in any other gear position; wherein the basic vehicle information is displayed together with the front video.

[0022] Optionally, the number of unmanned vehicles is greater than 1, and obtaining the on-site information collected by the target unmanned vehicle includes: sending a connection request to the target unmanned vehicle; when no other unmanned vehicles are connected, successfully connecting and obtaining the on-site information collected by the target unmanned vehicle; when other unmanned vehicles are connected, failing the connection and prompting that there is a connected unmanned vehicle.

[0023] Optionally, obtaining the on-site information collected by the target unmanned vehicle includes: sending a connection request to the target unmanned vehicle; in the event of a connection failure, sending a restart request to the target unmanned vehicle; and re-sending the connection request to the target unmanned vehicle after a successful restart, until the connection is successful and the on-site information collected by the target unmanned vehicle is obtained.

[0024] According to a second aspect of the present disclosure, a remote driving cockpit is provided, which may include: a remote information collection module, used to obtain on-site information collected by the target unmanned vehicle in response to a connection operation on the target unmanned vehicle; a remote control takeover module, used to perform a safety analysis based on the on-site information, and enter a takeover mode for the target unmanned vehicle when the target unmanned vehicle meets the takeover conditions.

[0025] According to a third method of the present disclosure, a remote driving control system is provided, the system comprising an unmanned vehicle and the remote driving cockpit of the aforementioned second aspect.

[0026] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0027] Processor; and

[0028] a memory for storing a computer program for the processor;

[0029] The processor is configured to implement the remote driving control method of the first aspect by executing a computer program.

[0030] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the remote driving control method of the first aspect is implemented.

[0031] According to a sixth aspect of the present disclosure, a computer program product is provided, which, when executed on an electronic device, enables the electronic device to implement the remote driving control method of the first aspect.

[0032] The present disclosure provides a remote driving control method, a remote driving cockpit and a remote driving control system. The method responds to the connection operation of the target unmanned vehicle, obtains the field information collected by the target unmanned vehicle; further performs a safety analysis based on the field information, so as to enter the takeover mode of the target unmanned vehicle when the target unmanned vehicle meets the takeover conditions. The method can execute corresponding safety protection strategies based on the vehicle and environmental status of the unmanned vehicle in the mining area, and determine whether the takeover conditions are met based on the field information of the unmanned vehicle on the basis of the initial connection, and then perform takeover to achieve safe and reliable remote control, which can better cope with complex working conditions; and in the event of parking or pressing problems caused by poor road conditions or vehicle failure, remote control can be performed without the need for on-site operation by operators, while maintaining the degree of unmanned operation in the mining area, improving production safety and mining transportation efficiency.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0035] Figure 1 A flowchart of the steps of the remote driving control method provided in an embodiment of the present disclosure.

[0036] Figure 2 A structural block diagram of a remote driving cockpit provided in an embodiment of the present disclosure.

[0037] Figure 3 A structural block diagram of a remote driving control system provided in an embodiment of the present disclosure.

[0038] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0040] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0041] It should be noted that the data obtained in this disclosure are accessed, collected, stored and used for subsequent analysis and processing with the consent and authorization of the users or the parties to whom the relevant data belong, after the users or the parties to whom the relevant data belong are clearly informed of the data collection content, data purpose, processing method and other information. The users or the parties to whom the relevant data belong may be sent the ways to access, correct and delete the data, as well as the methods to revoke consent and authorization.

[0042] Figure 1 This is a flowchart of the steps of the remote driving control method provided by the embodiment of the present disclosure. The steps of the method can be shown as follows from step 101 to step 102:

[0043] Step 101: In response to a connection operation to a target unmanned vehicle, obtain on-site information collected by the target unmanned vehicle.

[0044] In the disclosed embodiments, it can be applied to the remote control of one or more unmanned vehicles in a mining area. The unmanned vehicle can be in a normal state, and the unmanned vehicle can be controlled remotely to perform mining transportation operations; it can also be in an abnormal state such as being stuck, deviating from the path, or lifting abnormality, and the unmanned vehicle can be controlled remotely to return to a normal state and join the mining transportation operation. The unmanned vehicle can be a vehicle with a cockpit and enabled with an unmanned driving function, or it can be an unmanned vehicle without a cockpit. Specifically, the unmanned vehicle can be an unmanned wide-body dump truck.

[0045] In the disclosed embodiment, the connection operation to the target unmanned vehicle can be triggered by an external operation, and the connection operation can be a selection operation for any target unmanned vehicle. For example, the vehicle information of one or more unmanned vehicles can be provided to the operator, and when the operator's selection operation for an unmanned vehicle is received, the unmanned vehicle is used as the target unmanned vehicle. The vehicle information can include the vehicle's unique identification information, such as the vehicle number, vehicle name, etc., and can also include simple vehicle status information, such as normal status, abnormal status, etc., so that the operator can intervene remotely in time.

[0046] Further, in response to the connection operation, the collected on-site information can be obtained from the target unmanned vehicle. The on-site information can be data collected on-site by the target unmanned vehicle at the mining site through cameras, lidar, millimeter-wave radar, vehicle-mounted sensors, etc. The specific content of the on-site information can be designed according to the mining environment, operation requirements, hardware conditions, etc., and the embodiments of the present disclosure do not impose specific restrictions on this. At this time, the on-site environment of the unmanned vehicle can be analyzed through the on-site information, but the unmanned vehicle cannot be remotely controlled.

[0047] Among them, a communication connection can be established with the target unmanned vehicle through wireless communication technology, so as to obtain the on-site information collected by the target unmanned vehicle. The wireless communication technology can be a 5G or 5G evolved authorized frequency band communication technology. Those skilled in the art can also choose other communication technologies according to actual needs, and the embodiments of the present disclosure do not specifically limit this.

[0048] In the disclosed embodiment, before receiving the connection operation, the operator's identity may also be authenticated, and the identity authentication may be achieved through account and password verification, or biometric verification such as voice and face.

[0049] Step 102: Perform a safety analysis based on the on-site information, and enter a takeover mode for the target unmanned vehicle if the target unmanned vehicle meets the takeover conditions.

[0050] In the embodiments of the present disclosure, the security analysis based on the on-site information may be performed by displaying or providing the on-site information to the operator, who will determine whether there are security risks that may affect the takeover of control; or it may be performed by configuring security analysis rules, or training a security analysis model based on historical vehicle status data, to perform a security analysis on the on-site information, and determine whether there are security risks that may affect the takeover of control based on the security analysis results; or, it may be performed by configuring security analysis rules, or training a security analysis model based on historical vehicle status data, to perform a security analysis on the on-site information, and provide the security analysis results to the operator, who will determine whether there are security risks that may affect the takeover of control based on the security analysis results. The embodiments of the present disclosure do not impose specific restrictions on this.

[0051] Furthermore, the takeover conditions are used to determine whether the target unmanned vehicle has a safety risk that affects the takeover control. When there is no safety risk, or the safety risk is controllable, it can be considered that the target unmanned vehicle meets the takeover conditions. At this point, the takeover mode for the target unmanned vehicle can be entered. In the disclosed embodiment, when the target unmanned vehicle meets the takeover conditions, a function button for entering the takeover mode may be opened to the operator, such as providing the operator with a virtual button for entering the takeover mode, or activating a physical button for entering the takeover mode, where the physical button is invalid when not activated, and can enter the takeover mode when activated; or the target unmanned vehicle may directly enter the takeover mode when it meets the takeover conditions, so that the operator is not aware of it, and is only prompted or alarmed when the takeover conditions are not met.

[0052] In the disclosed embodiment, the takeover mode is a mode for remote driving and controlling the target unmanned vehicle. In the takeover mode, the target unmanned vehicle collects real-time on-site information of the mining area, and the operator can perform remote driving and control operations based on the on-site information, and send control instructions to the target unmanned vehicle based on the remote driving and control operations to take over and control the direction, speed, gear and other motion parameters of the target unmanned vehicle. In the takeover mode, a reminder that remote takeover is in progress can be provided in the form of voice, icons, lights, etc.

[0053] In an optional embodiment of the present disclosure, the number of unmanned vehicles may be greater than 1, and the step 101 of acquiring the on-site information collected by the target unmanned vehicle includes the following steps A1 to A3.

[0054] Step A1: Send a connection request to the target unmanned vehicle.

[0055] Step A2: When no other unmanned vehicles are connected, the connection is successful and the on-site information collected by the target unmanned vehicle is obtained.

[0056] Step A3: If other unmanned vehicles are already connected, the connection fails and a prompt is given indicating that there is an unmanned vehicle already connected.

[0057] In the disclosed embodiment, it can be set to take over only one unmanned vehicle at a time, and on this basis, only one of two or more unmanned vehicles is used as the target unmanned vehicle at a time. In response to the connection operation of the target unmanned vehicle, a corresponding connection request can be sent to the target unmanned vehicle. At this time, if other unmanned vehicles are not connected, the target unmanned vehicle can be successfully connected, and the field information collected by the target unmanned vehicle can be obtained from the target unmanned vehicle through the established communication connection; if other unmanned vehicles are connected, since only one unmanned vehicle is taken over at a time, the connection to the target unmanned vehicle fails, and a prompt can be given that there is an unmanned vehicle that is already connected, such as through a pop-up window warning "The connection is not disconnected, and other vehicles are prohibited from being connected."

[0058] Furthermore, in the case where other unmanned vehicles are already connected, an operator's disconnection operation on the connected unmanned vehicle can be received, and a connection request can be resent to the target unmanned vehicle after the connection is disconnected. In the embodiment of the present disclosure, after exiting the takeover mode for other unmanned vehicles, even if other unmanned vehicles cannot be remotely controlled, they may be in a communication connection state with other unmanned vehicles, and thus cannot establish a communication connection with the target unmanned vehicle. On this basis, when establishing a communication connection with other unmanned vehicles, a disconnection function button can be provided to the operator to disconnect the connection with other unmanned vehicles when receiving the operator's selection operation on the disconnection function button; on this basis, a new connection operation of the operator can be received to send a connection request to the target unmanned vehicle, or a connection request can be directly triggered to be sent to the target unmanned vehicle, and when the connection is successful, a disconnection function button can be provided to the operator.

[0059] In an optional embodiment of the present disclosure, obtaining the on-site information collected by the target unmanned vehicle in step 101 includes the following steps B1 to B3.

[0060] Step B1: Send a connection request to the target unmanned vehicle.

[0061] Step B2: In case of connection failure, a restart request is sent to the target unmanned vehicle.

[0062] Step B3: After the restart is successful, a connection request is sent to the target unmanned vehicle again until the connection is successful and the on-site information collected by the target unmanned vehicle is obtained.

[0063] In the disclosed embodiment, the target unmanned vehicle can be restarted remotely. In the event of a connection failure, a restart can be attempted to reconnect to the target unmanned vehicle. During the restart process, some information, configurations, or processes can be cleaned up or reset, which may solve the problem that caused the connection failure. After a successful restart, a connection request can be resent to the target unmanned vehicle until the connection is successful and the on-site information collected by the target unmanned vehicle is obtained. Resending the connection request can receive a new connection operation from the operator to send a connection request to the target unmanned vehicle, or it can directly trigger the sending of a connection request to the target unmanned vehicle.

[0064] The number of restarts or the interval can be set. When the number of restarts at the set interval reaches the set number, it can be considered that the final connection has failed, and then the operator is prompted or warned.

[0065] In an optional embodiment of the present disclosure, the takeover condition may include that the real-time vehicle speed of the target unmanned vehicle is 0.

[0066] In an optional embodiment of the present disclosure, the takeover condition may include that the target unmanned vehicle is in N gear.

[0067] In an optional embodiment of the present disclosure, the takeover condition may include that the handbrake of the target unmanned vehicle is in a pulled-up state.

[0068] In an optional embodiment of the present disclosure, the takeover condition may include that the target unmanned driving vehicle is in an unmanned driving mode.

[0069] In the disclosed embodiment, the takeover condition may be a restriction on the vehicle state, including that the real-time speed of the target unmanned vehicle is 0, that is, stationary and not driving; it may also include that the target unmanned vehicle is in N gear, which is neutral gear. In N gear, the engine of the target unmanned vehicle is not locked, but the power output is cut off. At this time, it can be started, or it can be towed or dragged; it may also include that the handbrake of the target unmanned vehicle is in the pulled-up state. The handbrake is also called the parking brake, which is used to apply resistance to the vehicle when parking to prevent slipping. When taking over the unmanned vehicle, it is usually required to enter the driving and transporting state, so the handbrake needs to be in the pulled-up state; it may also include that the target unmanned vehicle is in an unmanned driving mode, wherein the unmanned driving mode may be a control driving mode with no one on site and remote takeover. In addition to the unmanned driving mode, there may also be an on-site operation driving state, or other automatic driving states that do not require remote takeover.

[0070] In the embodiments of the present disclosure, the above takeover conditions are only used as examples. Those skilled in the art may select one or more, or expand more takeover conditions according to actual needs, and the embodiments of the present disclosure do not impose specific limitations on this.

[0071] In an optional embodiment of the present disclosure, the takeover mode may include controlling the speed of the target unmanned vehicle to be less than or equal to a remote control speed threshold.

[0072] In the embodiment of the present disclosure, after entering the takeover mode of the target unmanned vehicle, the target unmanned vehicle can be remotely driven and controlled in the takeover mode, including remotely controlling the driving direction, speed and collection of on-site information of the target unmanned vehicle. For example, in the takeover mode, the speed of the target unmanned vehicle can be controlled to be less than or equal to the remote control speed threshold. Among them, the remote control speed threshold can be pre-set according to the road conditions in the mining area, the model specifications of the target unmanned vehicle, the load of the target unmanned vehicle, and the transportation efficiency requirements of the mining area, or set by real-time dynamic analysis, and can be any size that meets actual needs, such as 10 kilometers per hour, 15 kilometers per hour, etc. The embodiment of the present disclosure does not specifically limit the setting method or actual size of the remote control speed threshold.

[0073] In an optional embodiment of the present disclosure, the takeover mode may include identifying obstacle distance based on on-site information, and controlling the target unmanned vehicle to make an emergency stop when the distance between the target unmanned vehicle and the obstacle is less than or equal to a remote control distance threshold.

[0074] In the disclosed embodiment, the on-site information may include obstacle information collected by the target unmanned vehicle at the mining site, such as location information of obstacles such as rocks, buildings, and other unmanned vehicles in the mining area collected by laser radar, millimeter wave radar, etc. On this basis, the distance of obstacles within a predetermined range in the driving direction of the target unmanned vehicle is identified. The predetermined range may include the driving direction, deflection angle, etc. of the target unmanned vehicle. Obstacles within the predetermined range may collide with the target unmanned vehicle. At this time, the distance of obstacles in the driving direction of the target unmanned vehicle can be identified.

[0075] It can be considered that the smaller the distance between the target unmanned vehicle and the obstacle, the greater the probability of collision. The remote control distance threshold can be set according to the critical distance at which the predicted collision probability is equal to the safety probability. The remote control distance threshold can be greater than or equal to the critical distance to avoid the predicted collision probability being too large. For example, when the critical distance is 3 meters, the remote control distance threshold can be 3 meters, or 4 meters, 5 meters, etc.

[0076] When the distance between the target unmanned vehicle and the obstacle is less than or equal to the remote control distance threshold, it is considered that there is a high risk of collision with the obstacle, and the target unmanned vehicle can be controlled to stop suddenly to avoid collision. On this basis, the target unmanned vehicle can be further controlled to reverse, turn, etc. to avoid the obstacle and continue to drive, or it can exit the takeover mode or perform other processing, which is not specifically limited in the embodiments of the present disclosure.

[0077] In an optional embodiment of the present disclosure, the takeover mode may include controlling the exit of the takeover mode of the target unmanned vehicle when the transmission delay of the on-site information is greater than or equal to the network delay threshold.

[0078] In the disclosed embodiment, the network delay of the transmission of information, instructions, etc. can also be monitored. For example, when the transmission delay of the on-site information is greater than or equal to the network delay threshold, it can be considered that the reference information of the remote control is distorted, and there is also a delay in sending the control instruction to the target unmanned vehicle. At this time, continuing the remote control has a greater safety risk, and the takeover mode of the target unmanned vehicle can be automatically exited. Among them, the network delay threshold can be set according to the type of communication technology, mining area transportation requirements, etc., such as 500 milliseconds, 300 milliseconds, etc. The disclosed embodiment does not specifically limit the size of the network delay threshold.

[0079] In the disclosed embodiment, after entering the takeover mode for the target unmanned vehicle, a function key for exiting the takeover mode may be provided to the operator. The function key for exiting the takeover mode may be the same as or different from the function key for entering the takeover mode in step 102.

[0080] In an optional embodiment of the present disclosure, the takeover mode may include displaying the on-site information in a rearview mirror mode.

[0081] In the disclosed embodiment, the target unmanned vehicle can collect video data from different directions and angles at the driving site in the mining area. The rearview mirror mode can display the video data collected from different directions and angles to the operator.

[0082] In an optional embodiment of the present disclosure, the takeover mode may include displaying the on-site information in a reverse mode.

[0083] In the disclosed embodiment, the reversing mode may be to display video data collected in a required specific direction and angle when the target unmanned vehicle is in a specific driving state.

[0084] In the embodiments of the present disclosure, the content of the above takeover mode is only used as an example. Those skilled in the art may select one or more, or expand more content according to actual needs, and the embodiments of the present disclosure do not impose specific limitations on this.

[0085] In an optional embodiment of the present disclosure, the on-site information includes front-view video, rear-view video and basic vehicle information.

[0086] In the disclosed embodiment, the front-view video may be video data acquired in the same acquisition direction as the driving direction of the target unmanned vehicle, and the rear-view video may be video data acquired in the opposite acquisition direction to the driving direction of the target unmanned vehicle. In addition, the basic vehicle information may include information describing the vehicle's motion state and the vehicle function enabled or disabled state.

[0087] In an optional embodiment of the present disclosure, the basic vehicle information includes at least one of vehicle speed, gear position, speed, steering wheel angle, handbrake status, tire pressure, lighting information, remaining fuel, pitch angle, roll angle, brake pressure, brake sign, lift angle, emergency stop switch status, remaining urea amount, and vehicle license plate information.

[0088] In an optional embodiment of the present disclosure, the rearview mirror mode includes displaying the rear-view video in the center of the display interface and displaying the front-view video in the form of a small window in the corner of the display interface when the target unmanned vehicle is in the R gear; and displaying the front-view video in the center of the display interface and displaying the rear-view video in the form of a small window in the corner of the display interface in any other gear position; wherein the basic vehicle information is displayed together with the front-view video.

[0089] In the disclosed embodiment, a display interface can be provided to the operator to display on-site information, so as to facilitate the operator to make decisions on remote driving control. In the rearview mirror mode, when the target unmanned vehicle is in the R gear, i.e. the reverse gear, the vehicle is in a reversing state, and the rear-view video can be displayed in the center of the display interface, and the front-view video can be displayed in the form of a small window in the corner of the display interface. At this time, the operator can more directly obtain the rear-view on-site environmental information of the target unmanned vehicle in order to reverse safely and smoothly; in any other gear, i.e. the gear other than the R gear, the front-view video and the rear-view video exchange display positions, and the front-view video is displayed in the center of the display interface, and the rear-view video is displayed in the form of a small window in the corner of the display interface. At this time, the operator can more directly obtain the on-site environmental information of the target unmanned vehicle in the driving direction.

[0090] The basic vehicle information can also be displayed together with the front-view video, such as being displayed in the front-view video, either in a fixed position or in an adjustable position. At this time, the type and quantity of the basic vehicle information can be set according to the remote driving control requirements.

[0091] In an optional embodiment of the present disclosure, the reversing mode includes displaying the front video in the center of the display interface when the target unmanned vehicle is in the R gear, displaying the rear video in the form of a small window in the corner of the display interface, and not displaying the rear video in any other gear position; wherein the basic vehicle information is displayed together with the front video.

[0092] In the disclosed embodiment, in the reversing mode, when the target unmanned vehicle is in the R gear, it is in a reversing driving state, and then on the basis of displaying the front video in the center of the display interface, the rear video is displayed in the form of a small window in the corner of the display interface, so that the operator can obtain the rear scene environment information of the target unmanned vehicle; when the target unmanned vehicle is in any other gear, that is, a gear other than the R gear, the rear video is not displayed, and only the front video is displayed in the center of the display interface to avoid interference from excessive information.

[0093] The basic vehicle information can also be displayed together with the front-view video, such as being displayed in the front-view video, either in a fixed position or in an adjustable position. At this time, the type and quantity of the basic vehicle information can be set according to the remote driving control requirements.

[0094] In the disclosed embodiment, in order to improve the transmission efficiency of video data such as front-view video and rear-view video, the target unmanned vehicle can provide the video playback address of the video data in the video background, and transmit the video data to the video background through a dedicated network. At this time, the video data can be pulled from the video background based on the video playback address to provide it to the operator in the display interface. This process can be implemented based on web real-time communication technology, supporting end-to-end real-time audio and video communication across cross-platform browsers.

[0095] The present disclosure provides a remote driving control method, a remote driving cockpit and a remote driving control system. The method responds to the connection operation of the target unmanned vehicle, obtains the field information collected by the target unmanned vehicle; further performs a safety analysis based on the field information, so as to enter the takeover mode of the target unmanned vehicle when the target unmanned vehicle meets the takeover conditions. The method can execute corresponding safety protection strategies based on the vehicle and environmental status of the unmanned vehicle in the mining area, and determine whether the takeover conditions are met based on the field information of the unmanned vehicle on the basis of the initial connection, and then perform takeover to achieve safe and reliable remote control, which can better cope with complex working conditions; and in the event of parking or pressing problems caused by poor road conditions or vehicle failure, remote control can be performed without the need for on-site operation by operators, while maintaining the degree of unmanned operation in the mining area, improving production safety and mining transportation efficiency.

[0096] Figure 2 A structural block diagram of a remote driving cockpit 200 is also provided. Figure 2 As shown, the remote driving cockpit 200 may include: a remote information collection module 201, which is used to obtain on-site information collected by the target unmanned vehicle in response to the connection operation of the target unmanned vehicle; a remote control takeover module 202, which is used to perform safety analysis based on the on-site information and enter the takeover mode of the target unmanned vehicle when the target unmanned vehicle meets the takeover conditions.

[0097] In the disclosed embodiment, the remote driving cockpit 200 may also include a configuration of driving simulation hardware. When the target unmanned vehicle is remotely driven and controlled in the takeover mode, the operator may perform remote driving control operations through the driving simulation hardware, such as controlling the driving direction of the target unmanned vehicle through the steering wheel configured in the remote driving cockpit, controlling the gear position of the target unmanned vehicle through the gear configured in the remote driving cockpit, controlling the speed and start and stop of the target unmanned vehicle through the accelerator pedal and brake pedal configured in the remote driving cockpit, and other functional buttons, such as a rocker switch for entering the takeover mode. The remote takeover control module 202 may receive the operator's operation of the driving simulation hardware in the remote driving cockpit 200 to generate a control instruction and send it to the target unmanned vehicle, so that the target unmanned vehicle changes or maintains the driving state based on the control instruction.

[0098] For example, on the basis that the target unmanned vehicle meets the takeover conditions, the operator steps on the brake and releases the emergency stop button in the driving simulation hardware, then switches to the D gear or R gear, releases the handbrake and slowly releases the brake, and steps on the accelerator pedal after the target unmanned vehicle is driving normally. The remote control takeover module 202 can generate corresponding control instructions from the above remote driving control actions and send them to the target unmanned vehicle to start driving in the takeover mode.

[0099] Alternatively, when exiting the takeover mode of the target unmanned vehicle, the operator switches to N gear in the driving simulation hardware, then applies the parking brake and presses the emergency stop button. The remote control takeover module 202 can generate corresponding control instructions from the above remote driving control actions and send them to the target unmanned vehicle to stop the vehicle in preparation for exiting the takeover mode.

[0100] In an optional embodiment of the present disclosure, the takeover condition includes at least one of the following:

[0101] The real-time speed of the target unmanned vehicle is 0;

[0102] The target unmanned vehicle is in N gear;

[0103] The target unmanned vehicle's handbrake is in the engaged state;

[0104] The target unmanned vehicle is in unmanned mode.

[0105] In an optional embodiment of the present disclosure, the takeover mode includes at least one of the following contents:

[0106] Control the speed of the target unmanned vehicle to be less than or equal to the remote control speed threshold;

[0107] Obstacle distance recognition is performed based on on-site information. When the distance between the target unmanned vehicle and the obstacle is less than or equal to the remote control distance threshold, the target unmanned vehicle is controlled to stop suddenly.

[0108] When the transmission delay of the on-site information is greater than or equal to the network delay threshold, the control exits the takeover mode of the target unmanned vehicle;

[0109] displaying the on-site information in a rearview mirror mode;

[0110] The on-site information is displayed in a reverse mode.

[0111] In an optional embodiment of the present disclosure, the on-site information includes front-view video, rear-view video and basic vehicle information.

[0112] In an optional embodiment of the present disclosure, the basic vehicle information includes at least one of vehicle speed, gear position, speed, steering wheel angle, handbrake status, tire pressure, lighting information, remaining fuel, pitch angle, roll angle, brake pressure, brake sign, lift angle, emergency stop switch status, remaining urea amount, and vehicle license plate information.

[0113] In an optional embodiment of the present disclosure, the rearview mirror mode includes displaying the rear-view video in the center of the display interface and displaying the front-view video in the form of a small window in the corner of the display interface when the target unmanned vehicle is in the R gear; and displaying the front-view video in the center of the display interface and displaying the rear-view video in the form of a small window in the corner of the display interface in any other gear position; wherein the basic vehicle information is displayed together with the front-view video.

[0114] In an optional embodiment of the present disclosure, the reversing mode includes displaying the front video in the center of the display interface when the target unmanned vehicle is in the R gear, displaying the rear video in the form of a small window in the corner of the display interface, and not displaying the rear video in any other gear position; wherein the basic vehicle information is displayed together with the front video.

[0115] In an optional embodiment of the present disclosure, the number of unmanned vehicles is greater than 1, and the remote information collection module 201 is specifically used to send a connection request to the target unmanned vehicle; when other unmanned vehicles are not connected, the connection is successful and the on-site information collected by the target unmanned vehicle is obtained; when other unmanned vehicles are connected, the connection fails and a prompt is given that there is an unmanned vehicle that is already connected.

[0116] In the disclosed embodiment, the number of remote driving cockpits 200 may be one or more than one, each remote driving cockpit 200 is connected to one target unmanned vehicle at a time, and different remote driving cockpits 200 are connected to different target unmanned vehicles.

[0117] In an optional embodiment of the present disclosure, the remote information collection module 201 is specifically used to send a connection request to the target unmanned vehicle; in the event of a connection failure, send a restart request to the target unmanned vehicle; after a successful restart, re-send a connection request to the target unmanned vehicle until the connection is successful and the on-site information collected by the target unmanned vehicle is obtained.

[0118] The remote driving cockpit provided by the present disclosure obtains the field information collected by the target unmanned vehicle in response to the connection operation of the target unmanned vehicle; further performs a safety analysis based on the field information to enter the takeover mode of the target unmanned vehicle when the target unmanned vehicle meets the takeover conditions. The method can execute corresponding safety protection strategies based on the vehicle and environmental status of the unmanned vehicle in the mining area, determine whether the takeover conditions are met based on the field information of the unmanned vehicle on the basis of the initial connection, and then perform takeover to achieve safe and reliable remote control, which can better cope with complex working conditions; and in the event of parking or pressing problems caused by poor road conditions or vehicle failure, remote control can be performed without the need for on-site operation by operators, and production safety and mining transportation efficiency can be improved while maintaining the degree of unmanned operation in the mining area.

[0119] Figure 3 A structural block diagram of a remote driving control system 300 is also provided. Figure 3 As shown, the remote driving control system 300 includes an unmanned vehicle 301 and a remote driving cockpit 302. The number of the unmanned vehicle 301 can be 1 or more than 1, and the number of the remote driving cockpit 302 can be 1 or more than 1. The remote driving cockpit 302 can be Figure 2 To avoid repetition, the related descriptions shown are not repeated here.

[0120] In the disclosed embodiment, the unmanned vehicle 301 can communicate with the remote driving cockpit 302 through the vehicle-side agent. The vehicle-side agent can communicate with the remote driving cockpit 302 through wireless communication technology, send the field information collected by the unmanned vehicle 301 to the remote driving cockpit 302, and receive the control instructions sent by the remote driving cockpit 302. The vehicle-side agent can also distribute the control instructions so that the unmanned vehicle 301 changes or maintains the driving state based on the control instructions.

[0121] On this basis, when the remote driving cockpit 302 fails to connect to the vehicle-side proxy, the vehicle-side proxy can be restarted through a restart request, and then the connection request is resent. For details, please refer to the relevant descriptions of the above steps B1 to B3, which will not be repeated here to avoid repetition.

[0122] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0123] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0124] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0125] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0126] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0127] Refer to the following Figure 4 The electronic device 400 according to this embodiment of the present disclosure is described. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] like Figure 4As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410).

[0129] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0130] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 4201 and / or a cache storage unit 4202 , and may further include a read-only storage unit (ROM) 4203 .

[0131] The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0132] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0133] The electronic device 400 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through a display unit 440 and an input / output (I / O) interface 450 connected to the display unit 440. In addition, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 through a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0135] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0136] In the embodiments of the present disclosure, a program product for implementing the above method is also provided, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0137] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0138] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0140] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0141] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0142] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A remote driving control method, characterized in that: The method comprises: In response to a connection operation to a target unmanned vehicle, acquiring on-site information collected by the target unmanned vehicle; A safety analysis is performed based on the on-site information, and when the target unmanned vehicle meets the takeover conditions, a takeover mode for the target unmanned vehicle is entered.

2. The method according to claim 1, characterized in that The takeover condition includes at least one of the following: The real-time speed of the target unmanned vehicle is 0; The target unmanned vehicle is in N gear; The handbrake of the target unmanned vehicle is in the pulled-up state; The target unmanned vehicle is in an unmanned driving mode.

3. The method according to claim 1, characterized in that The takeover mode includes at least one of the following: Controlling the speed of the target unmanned vehicle to be less than or equal to a remote control speed threshold; Perform obstacle distance identification based on the on-site information, and control the target unmanned vehicle to stop suddenly when the distance between the target unmanned vehicle and the obstacle is less than or equal to a remote control distance threshold; When the transmission delay of the on-site information is greater than or equal to a network delay threshold, controlling the exit of the takeover mode of the target unmanned vehicle; displaying the on-site information in a rearview mirror mode; The on-site information is displayed in a reverse mode.

4. The method according to claim 3, characterized in that The on-site information includes front-view video, rear-view video and basic vehicle information; The basic vehicle information includes at least one of vehicle speed, gear position, speed, steering wheel angle, handbrake status, tire pressure, lighting information, remaining fuel, pitch angle, roll angle, brake air pressure, brake sign, lifting angle, emergency stop switch status, remaining urea amount, and vehicle number information.

5. The method according to claim 4, characterized in that The rearview mirror mode includes displaying the rear-view video in the center of the display interface and displaying the front-view video in a small window at the corner of the display interface when the target unmanned vehicle is in the R gear; And in any other gear position, the front-view video is displayed in the center of the display interface, and the rear-view video is displayed in the form of a small window in the corner of the display interface; The basic vehicle information is displayed together with the forward-facing video.

6. The method according to claim 4, characterized in that The reverse mode includes displaying the front-view video in the center of the display interface when the target unmanned vehicle is in the R gear, displaying the rear-view video in a small window at the corner of the display interface, and not displaying the rear-view video in any other gear position; The basic vehicle information is displayed together with the forward-facing video.

7. The method according to claim 1, characterized in that The number of the unmanned vehicles is greater than 1, and the acquiring of the on-site information collected by the target unmanned vehicles includes: Sending a connection request to the target unmanned vehicle; In the case of no connection to other unmanned vehicles, the connection is successful and the on-site information collected by the target unmanned vehicle is obtained; If another driverless vehicle is already connected, the connection fails and a prompt appears indicating that there is already a connected driverless vehicle.

8. The method according to claim 1, characterized in that The obtaining of the on-site information collected by the target unmanned vehicle includes: Sending a connection request to the target unmanned vehicle; In case of connection failure, sending a restart request to the target unmanned vehicle; After the restart is successful, a connection request is resent to the target unmanned vehicle until the connection is successful and the on-site information collected by the target unmanned vehicle is obtained.

9. A remote control cockpit, characterized in that: The cockpit comprises: A remote information collection module, configured to obtain on-site information collected by a target unmanned vehicle in response to a connection operation to the target unmanned vehicle; The remote control takeover module is used to perform a safety analysis based on the on-site information and enter a takeover mode for the target unmanned vehicle when the target unmanned vehicle meets the takeover conditions.

10. A remote driving control system, characterized in that: The system comprises an unmanned vehicle and the remote driving cockpit according to claim 9.

Images