Man-machine co-driving system combined with multi-task convolutional neural network cloud

Abstract

The invention discloses a man-machine co-driving system combined with a multi-task convolutional neural network cloud, which comprises a vehicle-mounted terminal, a cloud processing platform and a remote operation terminal, and is characterized in that the vehicle-mounted terminal, the cloud processing platform and the remote operation terminal establish bidirectional data interaction connection through a communication module; according to the system, through a built-in traffic scene recognition module of the cloud processing platform, a knowledge graph technology and a graph convolutional neural network (GCNNs) are adopted to construct a recognition algorithm, compared with a traditional single recognition model, semantic association of various entities (road signs, traffic participants and the like) in a traffic scene can be fully mined, and the traffic scene recognition efficiency is improved. According to the method, the vehicle surrounding environment information can be quickly and accurately recognized, the traffic scene type, the danger level and key elements can be output in real time, the problems of recognition lag and missed judgment and misjudgment in a complex traffic scene are effectively solved, reliable basic support is provided for man-machine co-driving decision, and the driving risk caused by untimely scene recognition is reduced.

Description

Technical Field

[0001] This invention relates to the field of human-machine co-driving technology, specifically a human-machine co-driving system that combines a multi-task convolutional neural network with a cloud-based system. Background Technology

[0002] With the development of autonomous driving technology, human-machine co-driving is increasingly widely used in fields such as remote driving. Its core lies in achieving efficient linkage between vehicle environmental perception, operator status recognition, and cloud-based collaborative decision-making. Current systems suffer from two major pain points in this linkage: First, insufficient linkage between traffic scene recognition and cloud-based collaboration. Existing systems mostly adopt a local processing architecture on the vehicle terminal, relying on a single algorithm for scene recognition. This fails to uncover the semantic relationships between key scene elements, leading to missed or incorrect judgments. Furthermore, a robust cloud-based hybrid enhancement mechanism has not been built, preventing deep integration of onboard information and cloud knowledge, resulting in delayed collaborative decision-making. Second, the linkage between operator status recognition and decision-making efficiency is lacking. Existing systems have insufficient accuracy in recognizing operator attention and intentions, lack high-precision feature extraction algorithms, and the recognition results cannot be effectively synchronized to the cloud-based decision-making module. This leads to a mismatch between decision commands and operator status, further exacerbating driving safety hazards. In addition, MTCNN, GCNNs, and knowledge graph technologies are applied in a fragmented manner and have not been deeply integrated with the cloud to address these linkage pain points. Therefore, developing a human-machine co-driving system that can optimize the linkage of core modules has become an urgent technical challenge. Therefore, those skilled in the art have provided a human-machine co-driving system that combines a multi-task convolutional neural network in the cloud to solve the problems mentioned in the background art. Summary of the Invention

[0003] The purpose of this invention is to address the core pain points of insufficient traffic scene recognition and cloud-based collaborative linkage in existing technologies, as well as the lack of linkage between operator status recognition and decision-making efficiency. By deeply integrating MTCNN, GCNNs, and knowledge graph technologies into a cloud processing platform, this invention provides a human-machine co-driving system that achieves accurate traffic scene recognition, efficient operator status judgment, optimized cloud-terminal collaborative decision-making, and the construction of a closed-loop optimization mechanism. This enhances the safety, collaboration, and decision-making efficiency of human-machine co-driving, compensates for the shortcomings of existing systems, and meets the practical application needs of scenarios such as remote driving.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A human-machine co-driving system that integrates with a multi-task convolutional neural network in the cloud includes an in-vehicle terminal, a cloud processing platform, and a remote operation terminal. The in-vehicle terminal, the cloud processing platform, and the remote operation terminal establish a two-way data interaction connection through a communication module. Vehicle-mounted terminal: Used to collect vehicle status information and surrounding environment information and upload them to the cloud processing platform, while receiving decision control instructions and operation instructions from the cloud processing platform and remote operation terminal. Cloud processing platform: used to process, analyze and store various types of information collected, realize traffic scene recognition, operator attention and intent recognition, and generate human-machine co-driving decision control strategies based on the analysis results; Remote operation terminal: Used for operators to perform remote operations, while feeding back the operator's operation status information to the cloud processing platform, and receiving various scenario and vehicle-related information pushed by the cloud processing platform to assist operators in making accurate operation decisions.

[0005] Furthermore, the cloud processing platform has a built-in traffic scene recognition module. This module uses knowledge graph technology and graph convolutional neural networks (GCNNs) to construct a traffic scene recognition algorithm. This algorithm is used to identify the vehicle's surrounding environment information uploaded by the vehicle terminal in real time, and output the traffic scene type, scene hazard level, and key scene element information. This improves the efficiency of traffic scene recognition and provides basic support for subsequent human-machine co-driving decisions. Key scene elements include road signs, traffic participants, road conditions, and environmental weather information.

[0006] Furthermore, the cloud processing platform has a built-in operator status recognition module. The operator status recognition module is based on a multi-task convolutional neural network (MTCNN) to build a face detection model. The face detection model is used to collect operator face image information uploaded by the remote operation terminal. By detecting key facial feature points and analyzing facial expressions and gaze direction, it can recognize the operator's attention state and operation intention, improve the remote operator's awareness of the situation, and ensure the safety of remote driving operation.

[0007] Furthermore, the Multi-Task Convolutional Neural Network (MTCNN) adopts a three-level cascaded architecture, which includes P-Net, R-Net and O-Net in sequence. P-Net is used to initially screen candidate regions of operator faces, R-Net is used to further filter false positive samples and refine face bounding boxes, and O-Net is used to finally determine the face position and locate key facial feature points, including the eyes, the tip of the nose and the corners of the mouth.

[0008] Furthermore, the cloud processing platform has a built-in cloud-based hybrid enhanced human-machine co-driving module. This module receives vehicle status information and surrounding environment information uploaded by the vehicle terminal, aggregates and processes the two types of information, maps them to a preset spatial map, and then constructs a traffic scene knowledge graph (KG) based on real driving scenario data.

[0009] Furthermore, the traffic scene knowledge graph (KG) covers a variety of typical driving scenarios and emergency scenarios. Through the semantic association and reasoning capabilities of the knowledge graph, it assists remote operators in making more accurate and efficient decisions and controls, realizing cloud-based hybrid enhanced human-machine co-driving.

[0010] Furthermore, the vehicle-mounted terminal includes a perception and acquisition unit and an instruction execution unit. The perception and acquisition unit includes a camera, radar, GPS positioning module, and vehicle status sensor, which are used to collect image information around the vehicle, distance information of surrounding objects, vehicle position information, and vehicle's own operating status information, respectively. The instruction execution unit is used to receive decision control instructions issued by the cloud processing platform and operation instructions from the remote operation terminal, and drive the vehicle to complete corresponding operations such as steering, braking, and acceleration.

[0011] Furthermore, the remote operation terminal includes an operation unit and a display feedback unit. The operation unit is used for the operator to input remote driving operation commands, and the display feedback unit is used to display in real time the images of the vehicle's surrounding environment collected by the vehicle terminal, the vehicle's own status information, the traffic scene information identified by the cloud processing platform, and the operator's attention and intention recognition results, while also providing feedback on the execution status of the operation commands.

[0012] Furthermore, the operator's attention state includes a focused state, a distracted state, and a fatigued state, and the operation intention includes a steering intention, a braking intention, an acceleration intention, and a parking intention. The operator state recognition module provides an operator state reference for the cloud-based hybrid enhanced human-machine co-driving module by outputting the attention state level and operation intention.

[0013] By adopting the above technical solution Compared with the prior art, the beneficial effects of the present invention are: 1. This system utilizes the traffic scene recognition module built into the cloud processing platform and employs knowledge graph technology and graph convolutional neural networks (GCNNs) to construct recognition algorithms. Compared to traditional single recognition models, it can fully explore the semantic relationships of various entities (road signs, traffic participants, etc.) in traffic scenes, enabling rapid and accurate recognition of information about the vehicle's surrounding environment. It can output traffic scene type, hazard level, and key elements in real time, effectively solving the problems of delayed recognition, missed judgments, and misjudgments in complex traffic scenes. It provides a reliable foundation for human-machine co-driving decision-making and reduces driving risks caused by untimely scene recognition. 2. The operator status recognition module is based on a face detection model built with a multi-task convolutional neural network (MTCNN). Relying on the three-level cascaded architecture of P-Net, R-Net, and O-Net, it can accurately locate key facial feature points of the operator. By analyzing facial expressions and gaze direction, it can accurately determine the operator's attention status (focused, distracted, fatigued) and operating intention (steering, braking, etc.) and output the attention level and operating intention. It can promptly remind the operator to avoid problems such as fatigued driving and distracted driving, significantly improve the remote operator's awareness of the situation, avoid operational errors caused by abnormal operator status, and further ensure the safety and reliability of remote driving. 3. The cloud-based hybrid enhanced human-machine co-driving module aggregates vehicle status and surrounding environment information, maps it to a spatial map, and constructs a traffic scene knowledge graph (KG). This knowledge graph covers typical and sudden driving scenarios. With its semantic association and reasoning capabilities, it can help remote operators quickly grasp the core information of complex driving scenarios, making up for the shortcomings of limited operator vision and incomplete scene perception in remote driving. This helps operators make more accurate and efficient decisions and controls, achieves efficient collaboration between the cloud and the terminal, and improves the overall performance and adaptability of the human-machine co-driving system. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a human-machine co-driving system that integrates with a multi-task convolutional neural network in the cloud. Detailed Implementation

[0015] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0016] Please see Figure 1 The present invention provides an embodiment of a human-machine co-driving system that is combined with a multi-task convolutional neural network in the cloud, including an in-vehicle terminal, a cloud processing platform and a remote operation terminal, wherein the in-vehicle terminal, the cloud processing platform and the remote operation terminal establish a two-way data interaction connection through a communication module; Vehicle-mounted terminal: Used to collect vehicle status information and surrounding environment information and upload them to the cloud processing platform, while receiving decision control instructions and operation instructions from the cloud processing platform and remote operation terminal. Cloud processing platform: used to process, analyze and store various types of information collected, realize traffic scene recognition, operator attention and intent recognition, and generate human-machine co-driving decision control strategies based on the analysis results; Remote operation terminal: Used for operators to perform remote operations, while feeding back the operator's operation status information to the cloud processing platform, and receiving various scenario and vehicle-related information pushed by the cloud processing platform to assist operators in making accurate operation decisions.

[0017] Furthermore, the cloud processing platform has a built-in traffic scene recognition module. This module uses knowledge graph technology and graph convolutional neural networks (GCNNs) to construct a traffic scene recognition algorithm. This algorithm is used to identify the vehicle's surrounding environment information uploaded by the vehicle terminal in real time, and output the traffic scene type, scene hazard level, and key scene element information. This improves the efficiency of traffic scene recognition and provides basic support for subsequent human-machine co-driving decisions. Key scene elements include road signs, traffic participants, road conditions, and environmental weather information.

[0018] Furthermore, the cloud processing platform has a built-in operator status recognition module. The operator status recognition module is based on a multi-task convolutional neural network (MTCNN) to build a face detection model. The face detection model is used to collect operator face image information uploaded by the remote operation terminal. By detecting key facial feature points and analyzing facial expressions and gaze direction, it can recognize the operator's attention state and operation intention, improve the remote operator's awareness of the situation, and ensure the safety of remote driving operation.

[0019] Furthermore, the Multi-Task Convolutional Neural Network (MTCNN) adopts a three-level cascaded architecture, which includes P-Net, R-Net and O-Net in sequence. P-Net is used to initially screen candidate regions of operator faces, R-Net is used to further filter false positive samples and refine face bounding boxes, and O-Net is used to finally determine the face position and locate key facial feature points, including the eyes, the tip of the nose and the corners of the mouth.

[0020] Furthermore, the cloud processing platform has a built-in cloud-based hybrid enhanced human-machine co-driving module. This module receives vehicle status information and surrounding environment information uploaded by the vehicle terminal, aggregates and processes the two types of information, maps them to a preset spatial map, and then constructs a traffic scene knowledge graph (KG) based on real driving scenario data.

[0021] Furthermore, the traffic scene knowledge graph (KG) covers a variety of typical driving scenarios and emergency scenarios. Through the semantic association and reasoning capabilities of the knowledge graph, it assists remote operators in making more accurate and efficient decisions and controls, realizing cloud-based hybrid enhanced human-machine co-driving.

[0022] Furthermore, the vehicle-mounted terminal includes a perception and acquisition unit and an instruction execution unit. The perception and acquisition unit includes a camera, radar, GPS positioning module, and vehicle status sensor, which are used to collect image information around the vehicle, distance information of surrounding objects, vehicle position information, and vehicle's own operating status information, respectively. The instruction execution unit is used to receive decision control instructions issued by the cloud processing platform and operation instructions from the remote operation terminal, and drive the vehicle to complete corresponding operations such as steering, braking, and acceleration.

[0023] Furthermore, the remote operation terminal includes an operation unit and a display feedback unit. The operation unit is used for the operator to input remote driving operation commands, and the display feedback unit is used to display in real time the images of the vehicle's surrounding environment collected by the vehicle terminal, the vehicle's own status information, the traffic scene information identified by the cloud processing platform, and the operator's attention and intention recognition results, while also providing feedback on the execution status of the operation commands.

[0024] Furthermore, the operator's attention state includes a focused state, a distracted state, and a fatigued state, and the operation intention includes a steering intention, a braking intention, an acceleration intention, and a parking intention. The operator state recognition module provides an operator state reference for the cloud-based hybrid enhanced human-machine co-driving module by outputting the attention state level and operation intention.

[0025] In this embodiment, the system follows a closed-loop logic of "information collection - cloud processing - decision generation - command execution - feedback optimization". It relies on the two-way data interaction between the in-vehicle terminal, the cloud processing platform, and the remote operation terminal, and combines the core technologies of each module to achieve human-machine collaborative driving. The specific steps are as follows: The first step is information collection and uploading: After the system starts, the sensing and collection unit of the vehicle terminal enters the working state. The camera collects real-time image information of the vehicle's surroundings, the radar collects distance information between the vehicle and surrounding objects, the GPS positioning module collects the vehicle's current location information, and the vehicle status sensor collects the vehicle's own operating status (such as vehicle speed, steering angle, braking status, etc.). At the same time, the remote operation terminal starts the face capture function to capture the operator's face image information in real time and synchronously collects the initial operation commands (if any) input by the operator through the operation unit. The vehicle-related information collected by the vehicle terminal and the operator-related information collected by the remote operation terminal are all uploaded to the cloud processing platform in real time through the communication module to complete the initial information aggregation and uploading.

[0026] The second step involves multi-module collaborative processing in the cloud: After receiving various uploaded information, the cloud processing platform simultaneously activates the built-in traffic scene recognition module, operator status recognition module, and cloud-based hybrid enhanced human-machine co-driving module for collaborative processing. 1. Traffic Scene Recognition and Processing: The traffic scene recognition module calls a recognition algorithm based on knowledge graph technology and graph convolutional neural networks (GCNNs) to analyze the vehicle's surrounding environment information uploaded by the vehicle terminal in real time. It mines the semantic associations of key scene elements such as road signs, traffic participants, road conditions, and environmental weather, quickly identifies the current traffic scene type (such as urban roads, highways, intersections, etc.) and scene hazard level, and outputs scene recognition results to provide a basis for subsequent decision-making. 2. Operator Status Recognition and Processing: The operator status recognition module starts a face detection model based on a multi-task convolutional neural network (MTCNN). P-Net initially filters candidate regions for the operator's face, R-Net filters false positives and refines the face bounding box, and O-Net finally determines the face location and locates key facial feature points such as the eyes, nose tip, and corners of the mouth. Based on these key feature points, the module analyzes the operator's facial expression and gaze direction, and combines this with preset judgment criteria to identify the operator's current attention state (focused, distracted, fatigued) and operational intent (steering, braking, accelerating, stopping, etc.). The module outputs the attention state level and operational intent, and simultaneously feeds this information back to the cloud-based hybrid augmented human-machine co-driving module. 3. Cloud-based hybrid enhancement processing: The cloud-based hybrid enhancement human-machine co-driving module receives vehicle status information and surrounding environment information uploaded by the vehicle terminal, as well as scene recognition results output by the traffic scene recognition module. After aggregating and processing the two types of vehicle-related information, it maps them to a preset spatial map. Combined with real driving scenario data, it calls a preset traffic scene knowledge graph (KG) – which covers a variety of typical driving scenarios and emergency scenarios. With the help of its semantic association and reasoning capabilities, it integrates the operator status recognition results to comprehensively analyze the core needs and potential risks of the current driving scenario.

[0027] The third step is the generation and issuance of decision control instructions: Based on the processing results of the above three modules and combined with the preset logic of human-machine co-driving, the cloud processing platform comprehensively judges the complexity of the current driving scenario and the rationality of the operator's state, generates an appropriate human-machine co-driving decision control strategy, and simultaneously generates specific decision control instructions; the decision control instructions are issued bidirectionally through the communication module, on the one hand to the vehicle terminal to provide the vehicle terminal with basic control basis, and on the other hand to the remote operation terminal to assist the operator in adjusting operating behavior and making accurate decisions.

[0028] The fourth step is instruction execution and real-time feedback: The vehicle-mounted terminal receives decision control instructions from the cloud processing platform and, combined with operator operation instructions (if any) from the remote operation terminal, drives the vehicle to complete corresponding operations such as steering, braking, and acceleration through the instruction execution unit, ensuring that the vehicle operates according to the decision intent. At the same time, the display feedback unit of the remote operation terminal presents in real-time images of the vehicle's surrounding environment collected by the vehicle-mounted terminal, the vehicle's own status information, the traffic scene recognition results output by the cloud processing platform, and the operator's attention and intent recognition results, synchronously feeding back the execution status of the operation instructions, allowing the operator to grasp the vehicle's operating status and their own status in real time, facilitating timely adjustments to the operation. The vehicle-mounted terminal feeds back the vehicle operation execution status and the remote operation terminal's operator operation adjustment status to the cloud processing platform in real time through the communication module.

[0029] The fifth step is closed-loop optimization: The cloud processing platform receives feedback information from the vehicle terminal and remote operation terminal, compares the preset standards with the actual operation, analyzes the adaptability of the decision control commands and the accuracy of the recognition results of each module. If there are deviations (such as scene recognition errors, operator status misjudgments, etc.), the recognition parameters, decision logic and traffic scene knowledge graph (KG) data of each module are adjusted in real time to optimize the decision control commands. The optimized commands are then sent back to the vehicle terminal and remote operation terminal, forming a closed-loop workflow of "collection-processing-decision-execution-feedback-optimization", continuously improving the accuracy, reliability and coordination of system operation, and ensuring the safety and efficiency of remote human-machine co-driving.

[0030] Throughout the entire process, the vehicle terminal, cloud processing platform, and remote operation terminal maintain real-time two-way data interaction through the communication module. The modules work together to fully leverage the technical advantages of knowledge graphs, graph convolutional neural networks, and multi-task convolutional neural networks to achieve efficient identification of traffic scenarios and operator status, as well as cloud-based hybrid enhanced decision-making, ensuring the safety, smoothness, and efficiency of the human-machine co-driving process.

[0031] This system utilizes a traffic scene recognition module built into the cloud processing platform. Employing knowledge graph technology and graph convolutional neural networks (GCNNs) to construct recognition algorithms, compared to traditional single recognition models, it can fully exploit the semantic relationships of various entities (road signs, traffic participants, etc.) in traffic scenes, achieving rapid and accurate identification of information about the vehicle's surrounding environment. It can output traffic scene type, hazard level, and key elements in real time, effectively solving the problems of recognition lag, missed judgments, and false judgments in complex traffic scenes. This provides a reliable foundation for human-machine collaborative driving decision-making and reduces driving risks caused by untimely scene recognition. The operator status recognition module, based on a multi-task convolutional neural network (MTCNN), constructs a face detection model. Relying on a three-level cascaded architecture of P-Net, R-Net, and O-Net, it can accurately locate key facial feature points of the operator. By analyzing facial expressions and gaze direction, it accurately determines the operator's status. The system monitors the operator's attention status (focus, distraction, fatigue) and operational intentions (steering, braking, etc.), outputting attention levels and operational intentions. This allows for timely reminders to the operator to avoid issues such as fatigued driving and distraction, significantly improving the remote operator's awareness of the situation and preventing operational errors caused by abnormal operator states, thus further ensuring the safety and reliability of remote driving. The cloud-based hybrid enhanced human-machine co-driving module aggregates information about the vehicle's own status and the surrounding environment, maps it to a spatial map, and constructs a traffic scene knowledge graph (KG). This knowledge graph covers typical and sudden driving scenarios. With its semantic association and reasoning capabilities, it can assist remote operators in quickly grasping the core information of complex driving scenarios, compensating for the operator's limited field of vision and incomplete scene perception in remote driving. This helps operators make more accurate and efficient decisions and controls, achieving efficient collaboration between the cloud and the terminal, and improving the overall performance and adaptability of the human-machine co-driving system.

[0032] This specification describes the embodiments, but not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A human-machine co-driving system integrated with a multi-task convolutional neural network in the cloud, comprising an in-vehicle terminal, a cloud processing platform, and a remote operation terminal, characterized in that, The vehicle-mounted terminal, cloud processing platform, and remote operation terminal establish a two-way data interaction connection through a communication module. Vehicle-mounted terminal: Used to collect vehicle status information and surrounding environment information and upload them to the cloud processing platform, while receiving decision control instructions and operation instructions from the cloud processing platform and remote operation terminal. Cloud processing platform: used to process, analyze and store various types of information collected, realize traffic scene recognition, operator attention and intent recognition, and generate human-machine co-driving decision control strategies based on the analysis results; Remote operation terminal: Used for operators to perform remote operations, while feeding back the operator's operation status information to the cloud processing platform, and receiving various scenario and vehicle-related information pushed by the cloud processing platform to assist operators in making accurate operation decisions.

2. The human-machine co-driving system combined with a multi-task convolutional neural network in the cloud as described in claim 1, characterized in that, The cloud processing platform has a built-in traffic scene recognition module. The traffic scene recognition module uses knowledge graph technology and graph convolutional neural networks (GCNNs) to construct a traffic scene recognition algorithm, which is used to identify the vehicle's surrounding environment information uploaded by the vehicle terminal in real time, and output the traffic scene type, scene danger level and key scene element information, thereby improving the efficiency of traffic scene recognition and providing basic support for subsequent human-machine co-driving decisions. Key scene elements include road signs, traffic participants, road conditions and environmental weather information.

3. The human-machine co-driving system combined with a multi-task convolutional neural network in the cloud as described in claim 1, characterized in that, The cloud processing platform has a built-in operator status recognition module. The operator status recognition module is based on a multi-task convolutional neural network (MTCNN) to build a face detection model. The face detection model is used to collect operator face image information uploaded by the remote operation terminal. By detecting key facial feature points and analyzing facial expressions and gaze direction, it can recognize the operator's attention status and operation intention, improve the remote operator's awareness of the situation, and ensure the safety of remote driving operation.

4. The human-machine co-driving system combined with a multi-task convolutional neural network in the cloud as described in claim 3, characterized in that, The Multi-Task Convolutional Neural Network (MTCNN) adopts a three-level cascaded architecture, which includes P-Net, R-Net and O-Net in sequence. P-Net is used to initially screen candidate regions of operator faces, R-Net is used to further filter false positive samples and refine face bounding boxes, and O-Net is used to finally determine the face position and locate key facial feature points, including the eyes, the tip of the nose and the corners of the mouth.

5. The human-machine co-driving system combined with a multi-task convolutional neural network in the cloud as described in claim 1, characterized in that, The cloud processing platform has a built-in cloud-based hybrid enhanced human-machine co-driving module. The cloud-based hybrid enhanced human-machine co-driving module is used to receive vehicle status information and vehicle surrounding environment information uploaded by the vehicle terminal, aggregate and process the two types of information and map them to a preset spatial map, and then construct a traffic scene knowledge graph (KG) based on real driving scene data.

6. A human-machine co-driving system combined with a multi-task convolutional neural network in the cloud, as described in claim 5, is characterized in that... The traffic scenario knowledge graph (KG) covers a variety of typical driving scenarios and emergency scenarios. Through the semantic association and reasoning capabilities of the knowledge graph, it assists remote operators in making more accurate and efficient decisions and controls, realizing cloud-based hybrid enhanced human-machine co-driving.

7. The human-machine co-driving system combined with a multi-task convolutional neural network in the cloud as described in claim 1, characterized in that, The vehicle-mounted terminal includes a perception and acquisition unit and an instruction execution unit. The perception and acquisition unit includes a camera, radar, GPS positioning module, and vehicle status sensor, which are used to collect image information around the vehicle, distance information of surrounding objects, vehicle position information, and vehicle's own operating status information, respectively. The instruction execution unit is used to receive decision control instructions issued by the cloud processing platform and operation instructions from the remote operation terminal, and drive the vehicle to complete corresponding operations such as steering, braking, and acceleration.

8. The human-machine co-driving system combined with a multi-task convolutional neural network in the cloud according to claim 1, characterized in that, The remote operation terminal includes an operation unit and a display feedback unit. The operation unit is used for the operator to input remote driving operation commands, and the display feedback unit is used to display in real time the images of the vehicle's surrounding environment collected by the vehicle terminal, the vehicle's own status information, the traffic scene information identified by the cloud processing platform, and the operator's attention and intention recognition results, while also providing feedback on the execution status of the operation commands.

9. A human-machine co-driving system combined with a multi-task convolutional neural network in the cloud, as described in claim 8, is characterized in that... The operator's attention state includes focused state, distracted state, and fatigued state; the operation intention includes steering intention, braking intention, acceleration intention, and stopping intention. The operator state recognition module provides operator state reference for the cloud-based hybrid enhanced human-machine co-driving module by outputting the attention state level and operation intention.

Images