Remote cockpit control method and control system

Abstract

The present invention discloses a remote cockpit control method and control system, which includes: performing image detection on a preset detection position in the remote cockpit, and determining whether the driver is in place based on the result of the image detection; if it is determined that the driver is in place, inquiring the driver whether to turn on the order-taking status; if the driver turns on the order-taking status, updating the order-taking status of the remote cockpit to an available status, and notifying the cloud-based scheduling system to wait for the cloud-based scheduling system to dispatch an order; when the remote cockpit receives an order, inquiring the driver whether to accept the dispatch task; if the driver accepts the dispatch task, controlling the remote cockpit to establish a remote communication connection with the vehicle, and remotely driving the vehicle through the remote cockpit. The remote cockpit control method provided by the present invention can automatically update the order-taking status of the remote cockpit, reduce invalid dispatches of the cloud-based scheduling system, improve dispatch efficiency, and enhance the driver's operating experience in the order-taking process.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a remote cockpit control method and control system. Background Art

[0002] Remote driving is a unique solution in the field of autonomous driving. Autonomous driving still presents challenges in complex driving scenarios, such as high-risk road conditions, assisted parking, and vehicle breakdowns. Manual remote driving can address these issues. During remote driving, the remote vehicle transmits vehicle status data to the remote cockpit in real time via the 5G network. The remote driver can monitor the remote vehicle's driving status through the cockpit's displays and issue real-time control commands to the remote vehicle using the steering wheel, accelerator, brake, and other equipment on the cockpit, thereby remotely controlling the vehicle's movement.

[0003] During the commercialization process of remote driving, users can be charged for remote driving services. Therefore, a remote driving system generally includes a remote driving service device and a remote driving control device. The remote driving service device is used to determine the target cockpit assigned to the connected vehicle from the stored available cockpit information based on the request of the connected vehicle, and send dispatch information to the remote driving control device via the cloud network. The dispatch information contains the information of the connected vehicle that sent the remote driving request and the assigned target cockpit information. The remote driving control device sends the remote control command of the target cockpit to the connected vehicle via the cloud network to remotely control the connected vehicle through the remote control command. However, the availability of the cockpit often needs to be maintained manually by the driver. In actual use, if the driver forgets to suspend accepting orders when leaving the cockpit, the cockpit will still remain available. When the cloud dispatch system dispatches an order, the driver will actually be unable to accept the order, resulting in the failure of the dispatch. Therefore, it is necessary to provide an improved technical solution to reduce the occurrence of invalid dispatches in the cloud dispatch system. Summary of the Invention

[0004] In view of the above problems, the purpose of the present invention is to provide a remote cockpit control method that can automatically update the order acceptance status of the remote cockpit, reduce invalid orders in the cloud scheduling system, improve the order dispatching efficiency of the remote driving service, and enhance the driver's operating experience during the order acceptance process.

[0005] The present invention provides a remote cockpit control method, comprising: performing image detection on a preset detection position in the remote cockpit, and determining whether a driver is in position based on the result of the image detection; if it is determined that the driver is in position, inquiring the driver whether to turn on the order acceptance status; if the driver turns on the order acceptance status, updating the order acceptance status of the remote cockpit to an available status, and notifying a cloud-based scheduling system to wait for the order dispatched by the cloud-based scheduling system; when the remote cockpit receives the dispatch order, inquiring the driver whether to accept the dispatch task; if the driver accepts the dispatch task, controlling the remote cockpit to establish a remote communication connection with the vehicle, and remotely driving the vehicle through the remote cockpit.

[0006] Furthermore, the image detection is performed on a preset detection position in the remote cockpit, and determining whether the driver is in place based on the result of the image detection includes: obtaining a video stream of the seating area in the remote cockpit through a wide-angle color camera within a preset detection period; performing image detection on the video stream, and when a facial image or a human body image is detected, and the time when the front side of the facial image or the human body image remains facing the wide-angle color camera exceeds a preset in-place detection time threshold, it is determined that the driver is in place.

[0007] Furthermore, the image detection includes: extracting facial targets or human targets from the video stream; performing face detection and positioning on the extracted facial targets or human targets based on a multi-task cascade convolutional neural network algorithm, and identifying whether the detected face remains facing the front side of the wide-angle color camera.

[0008] Furthermore, the method also includes: when the order acceptance status of the remote cockpit is in an available state, if the time for which no face image or body image is detected at the detection position exceeds a preset non-positioned time threshold, the order acceptance status of the remote cockpit is updated to an unavailable state, and the cloud scheduling system is notified.

[0009] Furthermore, the method also includes: when the remote cockpit detects the parking command issued by the driver, asking the driver whether to end the dispatch task; if the driver confirms to end the dispatch task, controlling the remote cockpit to disconnect the remote communication connection with the vehicle.

[0010] Furthermore, the method also includes: when the driver accepts the dispatch task, updating the order acceptance status of the remote cockpit to an unavailable status, and suspending image detection of the detection location; when the driver confirms the completion of the dispatch task, updating the order acceptance status of the remote cockpit to an available status, and restarting image detection of the detection location.

[0011] Furthermore, the method also includes: the driver confirms whether to start the order acceptance status, confirms whether to accept the task of the order, and confirms whether to end the task of the order through voice instructions.

[0012] The present invention also provides a remote cockpit control system, comprising an image detection module, an order acceptance control module and a remote control module arranged in the remote cockpit; the image detection module is used to perform image detection on a preset detection position in the remote cockpit, and determine whether the driver is in place based on the result of the image detection; the order acceptance control module is used to communicate with the cloud scheduling system, and is used to inquire the driver whether to turn on the order acceptance status when it is determined that the driver is in place; if the driver turns on the order acceptance status, the order acceptance status of the remote cockpit is updated to an available status, and the cloud scheduling system is notified to wait for the order from the cloud scheduling system; when the remote cockpit receives the order, it inquires the driver whether to accept the order task; the remote control module is used to control the remote cockpit to establish a remote communication connection with the vehicle after the driver accepts the order task, and remotely drive the vehicle through the remote cockpit.

[0013] Furthermore, the image detection module includes a wide-angle color camera arranged in front of the remote cockpit seat, and the wide-angle color camera is used to obtain a video stream of the seat area in the remote cockpit.

[0014] Furthermore, the remote cockpit control system also includes a voice input device arranged in front of the remote cockpit seat, and the voice input device is used to receive voice operation instructions of the driver during the order acceptance process.

[0015] The remote cockpit control method and remote driving control system provided by the present invention determine whether the driver is in place by performing image detection on the cockpit. If the driver is in place and the order-taking state is turned on, the order-taking state of the remote cockpit is updated to an available state. If it is detected through image detection that the driver has left the remote cockpit seat, it is determined that the driver is not in place and the order-taking state of the remote cockpit is automatically updated to an unavailable state. This avoids the situation where the cloud-based dispatching system continues to dispatch orders due to the driver forgetting to suspend order receipt due to leaving the seat, while the driver cannot receive the orders, resulting in invalid orders. This improves the dispatching efficiency of the remote driving service, and completes the operations in the order-taking process through voice control, thereby improving the driver's operating experience in the order-taking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A schematic flow chart of a remote cockpit control method provided by one embodiment of the present invention;

[0018] Figure 2 A schematic diagram of the control flow of the remote cockpit after completing a dispatch mission provided by one embodiment of the present invention;

[0019] Figure 3 A schematic diagram of a control flow when a pilot leaves a remote cockpit according to an embodiment of the present invention;

[0020] Figure 4 This is a structural block diagram of a remote cockpit control system provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The aforementioned and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of the preferred embodiments with reference to the drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention. Some well-known parts may not be shown. In the various drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the drawings are not drawn to scale.

[0022] Figure 1 A flow chart of a remote cockpit control method according to an embodiment of the present invention is shown.

[0023] like Figure 1 As shown, the remote cockpit control method provided by the embodiment of the present invention includes:

[0024] Step S01: performing image detection on a preset detection position in the remote cockpit, and determining whether the pilot is in position based on the result of the image detection;

[0025] In one embodiment, a wide-angle color camera is provided in front of the driver's seat of the remote cockpit. The wide-angle color camera is used to obtain a video stream of the seating area in the remote cockpit within a preset detection period. Image detection is performed on the video stream. When a facial image or a human body image is detected, and the time when the front side of the facial image or the human body image remains facing the wide-angle color camera exceeds a preset in-position detection time threshold, it is determined that the driver is in position.

[0026] Specifically, when the remote cockpit is activated, a wide-angle color camera is simultaneously activated to capture a video stream of the seating area within the remote cockpit. Images within the video stream are used to detect whether the driver is seated in the remote cockpit. When a face or body image is detected in the driver's seat, image detection is used to determine whether the driver's face is facing the wide-angle color camera for a specified duration. The driver is deemed seated when the driver's face remains facing the wide-angle color camera for longer than a preset in-place detection time threshold. The in-place detection time threshold can be set to 5 seconds based on practical circumstances, and this embodiment does not impose any specific restrictions on the in-place detection time threshold. It will be readily understood that the use of a wide-angle color camera in this embodiment to capture a video stream of the seating area within the remote cockpit allows for images to be captured over a larger spatial range. Furthermore, due to the stronger perspective of a wide-angle lens, the contrast between near objects and far objects is emphasized more than with other lenses, resulting in clearer images of nearby objects. Other types of camera devices may also be used in other embodiments, and this is not a limitation, as long as they can capture video data of the seating area.

[0027] In one embodiment, the image detection includes: extracting facial targets or human targets from the video stream, and segmenting the facial targets or human targets from the background in the image based on the geometric features of the targets and the statistical features of the targets; performing face detection and positioning on the extracted facial targets or human targets based on a multi-task cascaded convolutional neural network algorithm (MTCNN, Multi-task Cascaded Convolutional Networks), and identifying whether the detected face remains facing the wide-angle color camera.

[0028] Step S02: If it is determined that the driver is in place, the driver is asked whether to start the order acceptance state;

[0029] Specifically, the driver can be prompted by voice reminders whether to turn on the order-accepting status. Accordingly, the driver can also directly confirm whether to turn on the order-accepting status by voice control. The voice control method is based on the semantic analysis technology in the natural language processing (NLP) algorithm, which recognizes the intention of the voice content and executes the operation instructions that match the recognized intention, such as confirming the turn on of the order-accepting status through the voice instruction of "turn on the order-accepting status". In some other embodiments, a prompt window can be popped up on the display device to prompt the driver whether to turn on the order-accepting status. Accordingly, the driver can also confirm whether to turn on the order-accepting status by clicking the mouse or touching the screen at the corresponding position in the pop-up prompt window.

[0030] Step S03: If the driver turns on the order acceptance state, the order acceptance state of the remote cockpit is updated to the available state, and the cloud dispatch system is notified to wait for the order to be dispatched by the cloud dispatch system;

[0031] Specifically, when the cloud-based scheduling system of the remote driving service receives a remote driving request from a connected vehicle, it sends a dispatch to an available remote cockpit, where the dispatch includes the registration information of the remote vehicle and the remote driving task requested by the connected vehicle.

[0032] Step S04: When the remote cockpit receives the dispatch order, it asks the driver whether to accept the dispatch order;

[0033] Specifically, when the remote cockpit receives a dispatch, it can ask the driver via voice prompts whether to accept the dispatched task, and the driver can also directly confirm whether to accept the dispatched task via voice commands. In other embodiments, when the remote cockpit receives a dispatch, a prompt window can be popped up on the display device to ask the driver whether to accept the dispatched task, and the driver can also confirm acceptance of the dispatched task by clicking the mouse or touching the screen at the corresponding position in the pop-up prompt window.

[0034] Step S05: If the driver accepts the dispatched task, the remote cockpit is controlled to establish a remote communication connection with the vehicle, and the vehicle is remotely driven through the remote cockpit.

[0035] Specifically, if the driver accepts the dispatch task, the remote cockpit is controlled through the cloud network to establish a remote communication connection with the remote vehicle according to the registration information of the remote vehicle in the dispatch order. The remote vehicle transmits the vehicle body status data to the remote cockpit in real time through the 5G network. The driver in the remote cockpit can understand the driving status of the remote vehicle in real time through the display, and issue control instructions to the remote vehicle in real time through the steering wheel, accelerator, brake and other equipment in the remote cockpit, thereby remotely controlling the vehicle's driving.

[0036] Figure 2 A schematic diagram of a control flow of a remote cockpit ending a dispatching task provided by an embodiment of the present invention. The remote cockpit control method also includes a control method for a remote cockpit ending a dispatching task, such as Figure 2 As shown in the figure, the control process of ending the dispatch task in the remote cockpit includes:

[0037] Step S06: When the remote cockpit detects the parking instruction issued by the driver, it asks the driver whether to end the dispatch task;

[0038] Specifically, after the driver in the remote cockpit controls the remote vehicle to complete the dispatching task, the brake in the remote cockpit is engaged in the parking gear. The remote cockpit detects the parking command issued by the driver, that is, detects that the brake is engaged in the parking gear. It can ask the driver through a voice reminder whether to end the dispatching task, and the driver confirms through a voice command. In some other embodiments, confirmation can also be made by a prompt window popping up on the display device.

[0039] Step S07: If the driver confirms to end the dispatching task, the remote cockpit is controlled to disconnect the remote communication connection with the vehicle.

[0040] In one embodiment, when a driver accepts a dispatch, the remote cockpit's order acceptance status is updated to unavailable, and image detection of the detection location is suspended. When the driver confirms the termination of the dispatch, the remote cockpit's order acceptance status is updated to available, and image detection of the detection location is resumed.

[0041] The above-mentioned remote cockpit control method optimizes the operation process of the driver in the process of accepting orders and improves the driver's operating experience in the process of accepting orders.

[0042] Figure 3 This is a schematic diagram of a control flow when a pilot leaves a remote cockpit according to an embodiment of the present invention. The remote cockpit control method also includes a control method when a pilot leaves a remote cockpit, such as Figure 3 As shown in Figure 1, the control flow when the driver leaves the remote cockpit includes:

[0043] Step S08: When the driver leaves the driver's seat of the remote cockpit and the order acceptance status of the remote cockpit is still available, image detection is performed on a preset detection position in the remote cockpit;

[0044] Step S09: If the time for which no face image or body image is detected at the detection position exceeds a preset non-position time threshold, the order acceptance status of the remote cockpit is updated to an unavailable state, and the cloud scheduling system is notified.

[0045] Specifically, when the driver forgets to suspend accepting orders when leaving the driving seat of the remote cockpit, the order acceptance status of the remote cockpit will remain available, and the image detection function will continue to be executed. When the time when no face image or body image is detected in the driver's seat through image detection exceeds the preset non-seated time threshold, the order acceptance status of the remote cockpit will be automatically updated to an unavailable state, and the cloud-based scheduling system will be notified to stop dispatching orders.

[0046] The above-mentioned embodiment uses image detection to detect in real time whether the driver has left the remote cockpit seat when the remote cockpit is in an available state. Based on the detection results, the remote cockpit can automatically update its order acceptance status to avoid invalid orders caused by the driver leaving and not being able to receive orders from the cloud scheduling system. This improves the dispatching efficiency of the remote driving service, simplifies the operating process, and enhances the driver's operating experience.

[0047] Figure 4 This is a structural block diagram of a remote cockpit control system provided by one embodiment of the present invention. Figure 4 As shown, the remote cockpit control system 200 provided in the embodiment of the present invention includes an image detection module 210, an order control module 220 and a remote control module 230 arranged in the remote cockpit.

[0048] Among them, the image detection module 210 is used to perform image detection on the preset detection position in the remote cockpit, and determine whether the driver is in place based on the result of the image detection; the order control module 220 is used to communicate with the cloud scheduling system, and when it is determined that the driver is in place, ask the driver whether to turn on the order status; if the driver turns on the order status, the order status of the remote cockpit is updated to an available status, and the cloud scheduling system is notified to wait for the order from the cloud scheduling system; when the remote cockpit receives the order, it asks the driver whether to accept the order task; the remote control module 230 is used to control the remote cockpit to establish a remote communication connection with the vehicle after the driver accepts the order task, and remotely drive the vehicle through the remote cockpit. The order acceptance control module 220 is also used to ask the driver whether to end the order dispatching task after the remote cockpit detects the parking command issued by the driver, and to update the order acceptance status of the remote cockpit to an unavailable state and notify the cloud scheduling system when the time when no face image or body image is detected at the detection position exceeds the preset non-position time threshold; the remote control module 230 is also used to control the remote cockpit to disconnect the remote communication connection with the vehicle when the driver confirms the end of the order dispatching task.

[0049] In one embodiment, the image detection module 210 includes a wide-angle color camera disposed in front of a seat in the remote cockpit, and the wide-angle color camera is used to obtain a video stream of the seat area in the remote cockpit.

[0050] In one embodiment, the remote cockpit control system 200 further includes a voice input device disposed in front of the remote cockpit seat, and the voice input device is used to receive voice operation instructions from the driver during the order acceptance process.

[0051] The remote cockpit control system provided in the embodiment of the present invention can perform image detection of the cockpit through the image detection module and determine whether the driver is in place, automatically update the order acceptance status of the remote cockpit through the order acceptance control module and receive the order dispatch from the cloud scheduling system, and then realize the communication connection and remote control between the remote cockpit and the vehicle according to the received order dispatch information through the remote control module, and can complete the operations in the order acceptance process through voice control, which simplifies the operation process in the order acceptance process, improves the order dispatch efficiency of the remote driving service, and enhances the driver's operation experience in the order acceptance process.

[0052] In the description of the present invention, unless otherwise expressly specified or limited, the terms "disposed," "installed," and "connected" should be interpreted broadly. For example, they can refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary. Those skilled in the art will understand the specific meanings of these terms in specific contexts. Terms such as "upper," "lower," "left," "right," "front," "back," "top," "bottom," "inner," and "outer" indicate positions or locations based on those shown in the accompanying drawings or the positions or locations in which the inventive product is typically placed during use. These terms are intended for ease of description and simplification only and are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. The terms "include," "comprising," or any other variations thereof are intended to encompass a non-exclusive inclusion, including elements not specifically listed, in addition to the listed elements.

[0053] The foregoing description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein are intended to be encompassed within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A remote cockpit control method, characterized in that: include: Perform image detection on the preset detection position in the remote cockpit and determine whether the pilot is in position based on the image detection results; If it is determined that the driver is in place, the driver is asked whether to start the order acceptance state; If the driver turns on the order acceptance state, the order acceptance state of the remote cockpit is updated to the available state, and the cloud dispatch system is notified to wait for the order dispatch from the cloud dispatch system; When the remote cockpit receives a dispatch, it asks the driver whether to accept the dispatched task; If the driver accepts the dispatched task, the remote cockpit is controlled to establish a remote communication connection with the vehicle, and the vehicle is remotely driven through the remote cockpit.

2. The remote cockpit control method according to claim 1, characterized in that: The performing of image detection on a preset detection position in the remote cockpit and determining whether the pilot is in position according to the result of the image detection includes: acquiring, within a preset detection period, a video stream of a seating area in the remote cockpit using a wide-angle color camera; Image detection is performed on the video stream. When a facial image or a human body image is detected, and the time for which the front side of the facial image or the human body image remains facing the wide-angle color camera exceeds a preset in-place detection time threshold, it is determined that the driver is in place.

3. The remote cockpit control method according to claim 2, characterized in that: The image detection includes: Extracting a face target or a human body target from the video stream; Based on the multi-task cascade convolutional neural network algorithm, face detection and positioning are performed on the extracted face target or human target, and it is identified whether the detected face remains facing the front side of the wide-angle color camera.

4. The remote cockpit control method according to claim 2, characterized in that: Also includes: When the order acceptance status of the remote cockpit is in an available state, if the time for which no face image or body image is detected at the detection position exceeds a preset non-positioned time threshold, the order acceptance status of the remote cockpit is updated to an unavailable state and the cloud scheduling system is notified.

5. The remote cockpit control method according to claim 1, characterized in that: Also includes: When the remote cockpit detects the parking instruction issued by the driver, asking the driver whether to end the dispatch task; If the driver confirms to end the dispatch task, the remote cockpit is controlled to disconnect the remote communication connection with the vehicle.

6. The remote cockpit control method according to claim 5, characterized in that: Also includes: When the driver accepts the dispatched task, the order acceptance status of the remote cockpit is updated to an unavailable state, and image detection of the detection location is suspended; When the driver confirms the completion of the dispatching task, the order acceptance status of the remote cockpit is updated to an available status, and image detection of the detection position is restarted.

7. The remote cockpit control method according to claim 6, characterized in that: Also includes: At least one of the operations of the driver confirming whether to start the order acceptance status, confirming whether to accept the task of the dispatched order, and confirming whether to end the task of the dispatched order is performed through voice instructions.

8. A remote cockpit control system, characterized in that: It includes an image detection module, an order control module and a remote control module located in the remote cockpit; The image detection module is used to perform image detection on a preset detection position in the remote cockpit and determine whether the driver is in position based on the result of the image detection; The order acceptance control module is used to communicate with the cloud-based dispatch system and, when it is determined that the driver is in position, inquire the driver whether to enable the order acceptance status; if the driver enables the order acceptance status, the order acceptance status of the remote cockpit is updated to an available state, and the cloud-based dispatch system is notified to wait for the order dispatch from the cloud-based dispatch system; when the remote cockpit receives the dispatch order, it inquires the driver whether to accept the dispatch order; The remote control module is used to control the remote cockpit to establish a remote communication connection with the vehicle after the driver accepts the dispatch task, and remotely drive the vehicle through the remote cockpit.

9. The remote cockpit control system according to claim 8, characterized in that: The image detection module includes a wide-angle color camera arranged in front of the remote cockpit seat, and the wide-angle color camera is used to obtain a video stream of the seat area in the remote cockpit.

10. The remote cockpit control system according to claim 8, characterized in that: It also includes a voice input device arranged in front of the remote cockpit seat, and the voice input device is used to receive the driver's voice operation instructions during the order acceptance process.

Images