Methods, systems, and computing devices for classifying the degree of human intervention in ground-based unmanned equipment

Abstract

This invention discloses a method, system, and computing device for grading the degree of human intervention in ground-based unmanned equipment. It describes the degree of human intervention using four indicators: head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction. The advantages of this invention are: it improves the objectivity of the evaluation of human intervention in ground-based unmanned equipment; and it improves the efficiency of testing and evaluation based on deep learning intelligent processing technology for video images. By integrating video data and detecting, recognizing, and tracking personnel posture and actions, as well as analyzing human-computer interaction behavior, it achieves a quantitative assessment of the degree of human intervention in ground-based unmanned equipment. This overcomes the limitations of traditional evaluations, which heavily rely on subjective qualitative evaluations by experts and lack sufficient quantitative expression capabilities. It constructs an evaluation system for the degree of human intervention in ground-based unmanned equipment with accurate perception and comprehensive quantitative analysis of video scenes, providing a certain reference for the quantitative assessment of the autonomy of ground-based unmanned equipment.

Description

Technical Field

[0001] This invention relates to the field of ground unmanned equipment technology, specifically to a method, system, and computing device for classifying the degree of human intervention in ground unmanned equipment. Background Technology

[0002] Human-computer interaction (HCI) refers to the communication and interaction between users and systems. The HCI interface typically refers to the user-visible part, such as buttons on devices, car dashboards, and robot control terminals—typical examples of HCI interfaces. With the development of artificial intelligence (AI) technology, the combination of HCI and AI has greatly improved interaction technology. Future HCI technology is developing towards human-machine integration, intelligent human-machine interaction, and natural human-machine interaction. Future military operations will increasingly require collaboration between unmanned systems and personnel. Achieving seamless human-machine integration will reduce the workload of human operators, enhance the autonomous intelligent decision-making capabilities of machines, and improve the efficiency of human-machine collaborative decision-making. Currently, HCI for ground-based unmanned equipment primarily relies on physical interaction methods such as joysticks, buttons, and touchscreens. Advanced human-machine collaborative interaction systems are crucial for improving the autonomy of unmanned equipment and enhancing its combat capabilities.

[0003] To study the relationships between humans and machines, systems, and platforms, and to scientifically and rigorously determine evaluation indicators for the degree of human intervention, in order to improve the performance of unmanned ground equipment, reduce operating and design costs, enhance the adaptability of existing equipment to new environments, and accelerate its deployment, an accurate and reasonable method for classifying the degree of human intervention in unmanned ground equipment is urgently needed. Summary of the Invention

[0004] This invention proposes a method, system, computing device, and storage medium for classifying the degree of human intervention in ground-based unmanned equipment, solving the problem that existing technologies cannot accurately classify the degree of human intervention in ground-based unmanned equipment.

[0005] According to one aspect of the present invention, a method for classifying the degree of human intervention in ground-based unmanned equipment is provided, characterized in that it includes:

[0006] An evaluation system for the degree of human intervention in ground-based unmanned equipment is constructed. The evaluation system is characterized by evaluation indicators such as head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction.

[0007] Calculate evaluation scores for head posture focus, facial expression focus, hand operation time percentage, and hand operation interaction frequency;

[0008] Based on the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency, the degree of human intervention in ground unmanned equipment is determined into five levels: remote control level, intelligent assistance level, human guidance level, human assistance level, and autonomous level.

[0009] Furthermore, the evaluation score for head posture focus is calculated, including:

[0010] The operator's head video was acquired, and a deep learning method was used to calculate three angles of the operator's head: pitch angle, yaw angle, and roll angle.

[0011] Images with pitch, yaw, or roll angles of the operator's head less than a certain threshold are selected as images of the operating interface to be focused on, and the number of frames of the images of the operating interface to be focused on is determined.

[0012] Divide the number of frames in the focus interface image by the total number of frames in the video, and then multiply by 10 to obtain the score for head posture focus.

[0013] Furthermore, the evaluation score for facial expression focus is calculated, including:

[0014] The operator's head video is acquired, and the operator's facial expression concentration is identified through deep transfer learning. The recognition result of each frame is marked as focused or unfocused.

[0015] The number of frames that indicate focus is divided by the total number of frames in the video, and then multiplied by 10, is used as the score for facial expression focus.

[0016] Furthermore, the evaluation score for the percentage of time spent on hand operations is calculated, including:

[0017] Acquire video of operator's hand movements, use deep learning methods to detect hands and screen in the video, and calculate the intersection-union ratio (IOU) between the two;

[0018] The number of image frames in which the hand is performing an operation is determined by comparing the IOU with a set threshold. If the IOU is greater than or equal to the threshold, it is considered that an operation is being performed; if it is less than the threshold, it is considered that no operation is being performed.

[0019] Divide the number of image frames in which the hand is performing the operation by the total number of frames in the video, and then multiply by 10 to obtain the score value for the percentage of time the hand is engaged.

[0020] Further, the evaluation score for the frequency of hand operation interactions is calculated, including:

[0021] Acquire video of the operator's hand movements and calculate the number of interactions between hand movements and non-hand movements in the video;

[0022] The total number of seconds and the number of interactions in this video are counted. The number of interactions is multiplied by the fastest interaction time, then divided by the total number of seconds in the video, and then multiplied by 10 to obtain the score value for the frequency of hand operation interaction.

[0023] Furthermore, deep learning methods are employed, including: using a multi-angle regression loss design, combined with gradient optimization and adaptive methods, and using a convolutional neural network to estimate 3D head pose from image intensity.

[0024] Furthermore, deep transfer learning methods are used to identify the operator's facial expression focus, including: using the DAN-main network for facial expression focus detection. During testing, the face is first cropped, and the head rotation angle is obtained through OpenCV-Python. Face images with a rotation angle greater than a set threshold are classified as unfocused. Face images within the set threshold range are then input into the DAN-main network for facial expression focus detection.

[0025] According to another aspect of the present invention, a grading system for the degree of human intervention in ground-based unmanned equipment is provided, characterized in that it includes:

[0026] The construction unit is used to build an evaluation system for the degree of human intervention in ground unmanned equipment. The evaluation system is characterized by evaluation indicators such as head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction.

[0027] The processing unit is used to calculate evaluation scores for head posture focus, facial expression focus, hand operation time percentage, and hand operation interaction frequency.

[0028] The determination unit is used to determine the degree of human intervention in ground unmanned equipment into five levels based on the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency: remote control level, intelligent assistance level, human guidance level, human assistance level, and autonomous level.

[0029] According to another aspect of the present invention, a computer storage medium is provided, characterized in that the storage medium stores at least one executable instruction, the executable instruction causing a processor to perform an operation corresponding to the ground unmanned equipment human intervention degree classification method.

[0030] According to another aspect of the present invention, a computing device is provided, characterized in that the device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method for classifying the degree of human intervention in ground unmanned equipment.

[0031] As can be seen from the technical inventions provided by this invention, the method and system for grading the degree of human intervention in unmanned ground equipment provided by this invention describe the degree of human intervention in unmanned ground equipment through four indicators: head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency. This is beneficial to improving the collaboration between humans and equipment. At the same time, improving the communication interface and enhancing the usability and reliability of the application can reduce the number of system operators required and lower the cost of designing different system displays or redesigning unmanned ground equipment in the absence of human-machine interaction support. Based on video data acquisition and integration, as well as the detection, recognition, and tracking of human posture and action behavior, and the analysis of human-machine interaction behavior, a quantitative assessment of the degree of human intervention in unmanned ground equipment is achieved. This overcomes the limitations of traditional evaluations that heavily rely on subjective qualitative evaluations by experts and lack quantitative expression capabilities. It constructs a comprehensive quantitative analysis system for evaluating the degree of human intervention in unmanned ground equipment with accurate perception of video scenes, providing a certain reference for the quantitative assessment of the autonomy of unmanned ground equipment.

[0032] The above description is merely an overview of the technical invention of this invention. In order to better understand the technical means of this invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this invention more apparent and understandable, specific embodiments of this invention are described below. Attached Figure Description

[0033] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0034] Figure 1 This is a flowchart of a method for classifying the degree of human intervention in ground-based unmanned equipment according to the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the convolutional neural network of the present invention;

[0036] Figure 3 This is a schematic diagram of the head posture of the present invention;

[0037] Figure 4 This is a schematic diagram of a grading system for the degree of human intervention in ground-based unmanned equipment according to the present invention;

[0038] Figure 5 This is a structural diagram of an exemplary hardware architecture for a computing device that grades the degree of human intervention in ground-based unmanned equipment, as described in an embodiment of the present invention. Detailed Implementation

[0039] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0040] The following provides a detailed description of a method and system for classifying the degree of human intervention in ground-based unmanned equipment, as provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0041] See Figure 1 The present invention provides a method for classifying the degree of human intervention in ground-based unmanned equipment, comprising:

[0042] S101, Construct an evaluation system for the degree of human intervention in ground unmanned equipment. The evaluation system is characterized by evaluation indicators such as head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction.

[0043] S102, calculate the evaluation scores for head posture focus, facial expression focus, hand operation time percentage, and hand operation interaction frequency;

[0044] S103. Based on the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency, the degree of human intervention in ground unmanned equipment is determined into five levels: remote control level, intelligent assistance level, human guidance level, human assistance level, and autonomous level.

[0045] Specifically, in this invention, the degree of human intervention in ground-based unmanned equipment represents the degree of dependence on external operators. A higher degree of human intervention indicates a greater dependence on external operators and a lower degree of autonomy for the ground-based unmanned equipment. This invention designs four indicators to describe the degree of human intervention in ground-based unmanned equipment: head posture focus, facial expression focus, percentage of time spent on hand operations, and frequency of hand operation interactions.

[0046] Specifically, head posture attention: using deep learning methods, three angles of the operator's head can be calculated: pitch, yaw, and roll. Here, pitch and yaw are selected as the calculation basis. In each frame, if the operator's pitch or yaw angle is less than a certain threshold (e.g., 15 degrees), the operator is considered to be paying attention to the interface. The final test index for head posture attention is the number of frames in which the operator is paying attention to the interface divided by the total number of frames in the video, and then multiplied by 10, which is used as the score of head posture attention (the value is between 0 and 10).

[0047] Specifically, common head pose estimation methods are based on the detection of facial landmarks. Landmark-based facial analysis recovers the 3D head pose by establishing a correspondence between landmarks and a 3D head model and performing alignment. This method, which uses convolutional neural networks to extract 2D facial landmarks, offers good flexibility. However, it doesn't utilize all facial information, and 3D pose estimation is impossible if landmarks are not detected. To overcome this deficiency, this invention employs a landmark-free pose estimation method. It utilizes a multi-angle regression loss design, combined with gradient optimization and adaptive methods, to improve the accuracy of head pose assessment. Compared to landmark-based methods, this direct, holistic approach using convolutional neural networks to estimate 3D head pose from image intensity provides higher accuracy. Building upon existing literature, this invention optimizes the gradient descent method and uses the deep residual network RestNet 101, which has a deeper network structure than RestNet 50, achieving better accuracy and performance on the AFLW2000 and 300W-LP datasets.

[0048] Figure 2 The diagram shows a CNN-based image classification network model. The network structure includes two convolutional layers, two pooling layers, and three fully connected layers. The specific process is as follows: The input image is first processed by three convolutional kernels. The output of convolutional layer 1 is three feature maps of the original image. These are then processed using a non-linear activation function and input into pooling layer 1 for pooling, resulting in smaller feature maps. Next, after processing by convolutional layer 2 and pooling layer 2, the resulting feature maps are vectorized and input into the fully connected layers. The fully connected layers are responsible for feature fusion. Finally, the resulting feature vectors are input into a SoftMax classifier for image classification, yielding the classification result.

[0049] In convolutional neural networks (CNNs), the loss function represents the deviation between the predicted and actual values ​​of a sample. It serves as the optimization objective in the deep learning network, and backpropagation is used to optimize the training of the CNN parameters. Backpropagation acts as negative feedback for the network, iteratively optimizing the network's parameters. Classification and regression are two fundamental tasks in supervised learning.

[0050] The following is a specific representation of the loss function in classification and regression tasks. In head pose estimation, the essence is also to use classification and regression methods to predict pose. The method used in this paper can also be said to be a transformation and combination of the two.

[0051] Loss functions for classification tasks: Classification problems deal with discrete values, classifying samples into categories based on data labels, or predicting the distribution of discrete variables. The cross-entropy loss function is frequently used in classification tasks; it represents the error in the probability distribution between sample values ​​and true values. This function transforms the output into a probability distribution, and then the softmax function is used to output the probability score from the forward propagation of the neural network. This is a commonly used loss function for convolutional neural networks in classification tasks, and the formula is as follows:

[0052]

[0053] The loss function for regression tasks: Each dimension of a regression task is a real value, not represented by 0 or 1. It predicts continuous variables, allowing the regression task to handle continuous values ​​and obtain quantitative output results. Regression tasks require the use of Mean Squared Error (MSE), as shown in the equation, to quantitatively represent the error between the actual and predicted values:

[0054]

[0055] Where y represents the actual value and 'y represents the predicted value.

[0056] Specifically, this invention uses deep learning methods, employs a multi-angle regression loss design, and combines gradient optimization and adaptive methods to improve the accuracy of head pose assessment. Compared with keypoint-based methods, the direct holistic method of estimating 3D head pose from image intensity using convolutional neural networks provides higher accuracy. Based on the literature, this invention optimizes the gradient descent method and uses the deep residual network RestNet 101, which has a deeper network structure than RestNet 50, achieving better accuracy and performance on the AFLW2000 and 300W-LP datasets.

[0057] Common head pose estimation methods are based on the detection of facial landmarks. Landmark-based facial analysis reconstructs the 3D head pose by establishing a correspondence between landmarks and a 3D head model and performing alignment. This method, which uses convolutional neural networks to extract 2D facial landmarks, offers good flexibility. However, it does not utilize all facial information, and 3D pose estimation is impossible if landmarks are not detected. To overcome this deficiency, this invention employs a landmark-free pose estimation method.

[0058] Specifically, Euler angles are used to evaluate human head pose. Human head images captured by a monocular camera can be represented by Euler angles, specifically yaw, pitch, and roll, to understand the spatial pose of the human head. A direct, holistic approach using convolutional neural networks to estimate 3D head pose from image intensity provides higher accuracy. A deeper residual network is used, with 198 fully connected layers. The lower layers are for coarse classification, while the upper layers are primarily used for fine classification. Gradient optimization is incorporated into the network training, using a softmax classifier. The cross-entropy loss of each layer's output is calculated, and the mean squared error at the three angles is obtained through final expectation calculation. This, combined with the loss outputs from other layers, is used to calculate the total loss. A gradient optimization strategy is incorporated during network model training to reduce the impact of gradient degradation in deep networks. The expectation value is calculated in the upper classification layers, while only the cross-entropy loss is calculated in the fully connected lower classification layers. SoftMax regression, derived from logistic regression, is a supervised learning method primarily designed to solve multi-class classification problems. By adding a SoftMax function to the last layer of the network and combining it with deep learning techniques, we can achieve classification of the input head pose angle categories. There are three angle categories for head pose evaluation, and each angle undergoes the same processing. After SoftMax regression, the cross-entropy calculation is applied. The cross-entropy function can be obtained using the MAE formula: MAE (Mean Absolute Error) represents the average absolute error, which is used to represent the average absolute error between the predicted and true values, and can reflect the prediction error situation well. The formula is shown below:

[0059]

[0060] y and y' represent the actual value and the predicted value, respectively. It represents the average absolute error between the predicted value and the observed value.

[0061] The loss at each angle is expressed by the following formula:

[0062]

[0063] In the formula, L and MSE represent the cross-entropy loss and mean squared error loss functions, respectively, and n represents the number of categories.

[0064] This invention uses deep transfer learning to identify the operator's facial expression focus level. It not only solves the problems of insufficient sample data leading to training difficulties, data labeling challenges, and algorithm training issues, but also improves machine learning efficiency and reduces learning costs. The recognition result for each frame is either focused or unfocused; the final facial expression focus index is calculated by dividing the number of frames with a focused result by the total number of frames in the video, then multiplying by 10, and serving as the facial expression focus score (with a value between 0 and 10).

[0065] Specifically, the DAN-main network is used for facial expression attention detection. During testing, faces are first cropped, and the head rotation angle is obtained using OpenCV-Python. Images with a rotation angle greater than a set threshold indicate excessive head rotation, classifying them as inattentive. Faces within the threshold range are then input into the DAN-main network for facial expression attention detection. Transfer learning is employed. First, the DAN-main network, primarily used in facial expression recognition, is initialized with its pre-trained weights using transfer learning. This network is then trained on a custom dataset (both inattentive and attentive datasets). The updated network and weights are then used to classify new images. Transfer learning addresses the issue of limited sample data. Pre-trained weights are downloaded from the DAN-main website and input into the dataset for training, allowing for better model convergence. Running `train.py` requires specifying the following parameters: `--aff_pth`: specifies the location of the dataset; `--batch-size`: specifies a batch size of 256 based on available computing resources, meaning 256 images are fed into the model at a time for training; `--lr`: specifies the learning rate. --epochs: Sets the iteration duration to 100 epochs. Before training, training parameters need to be set in the network framework, including the number of input images in a batch, the learning rate, and the size of the convolutional kernels. In the training process of deep learning, the settings of parameters such as the learning rate directly affect the model's performance. When the learning rate is set too high or the stride is too long, it can easily lead to large gradients, causing the model to miss the extreme point and fail to converge. Conversely, when the learning rate is set too low or the stride is too short, it can lead to too many iterations and stall. Therefore, a balance needs to be found to converge to an optimal extreme point. During testing, since the focus is mainly analyzed through facial expressions, the head transformation angle is first limited. The head deflection angle is detected using DILB to obtain the head deflection angles pitch_degree, yaw_degree, and roll_degree. The video to be detected is then fed into the prediction model for prediction, and the class of each frame in the video containing the target is output. The parameters that need to be specified include: --txt_path: Specifies the path to the txt file that records whether each frame shows focus. --test_path: Specifies the video path to be tested. --pose_txt_path: Specifies the path to the txt file that stores the head pose information for each frame. --checkpoint: Sets the weight path. The system will obtain three head deflection angles. First, it checks if the three values ​​are within a certain threshold range. If they are outside the set range, i.e., the head deflection is too large, it is directly judged as unfocused.If the three angle values ​​are within the set range, then use OpenCV-Python to detect faces. If no face is detected, the label is set to "null", indicating that no face was detected. If a face is detected, then it is input into the trained network for detection.

[0066] Specifically, the hand operation time percentage is calculated as follows: Hand and screen are detected in the video using deep learning methods, and their Intersection over Union (IOU) is calculated. The IOU is compared to a set threshold; an IOU greater than or equal to the threshold indicates an operation is in progress, while an IOU less than the threshold indicates no operation is in progress. The final hand operation time percentage is calculated by dividing the number of hand operation frames by the total video length and then multiplying by 10 to obtain the hand operation time percentage score (the value ranges from 0 to 10).

[0067] Specifically, hand operation detection employs transfer learning on YOLOv5s, initializing weights pre-trained on the COCO dataset and then fine-tuning them on a custom-made dataset. The YOLOv5s network is the shallowest and narrowest in the YOLOv5 series. Since the hand target recognition and detection in this invention involves relatively simple features and a relatively uncomplicated environment, YOLOv5s is used as the detection model for training, ensuring both training accuracy and speed. First, a hand operation scenario is simulated using a 1920*1080 resolution camera. Two hand operation videos are recorded, one for training and one for testing. The training video is sampled at 30fps intervals. The most important part of the training dataset is the labeled samples. Targets in the images are labeled in YOLO format using LabelImg. The dataset is then divided into training and validation sets, with 1815 images in the training set and 202 images in the validation set. After obtaining the corresponding model, this hand dataset is used for both training and detection. Calculate the total number of frames in the video and store it as a variable `frames_all`. The total number of frames is the sum of three parts: operation frames (`run_frames`), non-operation frames (`not_run_frames`), and frames where no hand or screen action was detected (`not_detect`). The percentage of time spent on hand operations is calculated by dividing the number of operation frames (`run_frames`) by the total number of frames (`frames_all`).

[0068] Specifically, the hand operation interaction frequency is calculated as follows: First, the number of hand operations and non-operations in each evaluation sample video is calculated. The total number of seconds and the number of interactions in the video are then counted. The number of interactions is multiplied by the fastest interaction time, divided by the total number of seconds in the video, and then multiplied by 10 to obtain the hand operation interaction frequency score (the value is between 0 and 10). First, the total number of frames in the video is calculated and stored as the variable `frames_all`. The number of hand operation interactions is calculated and stored as the variable `run_times`. The fastest time for switching between hand operations and non-operations is set to 2 seconds (this can be adjusted). Then, the hand operation interaction frequency is calculated as the number of interactions `run_times` multiplied by the fastest interaction time divided by the total number of frames `frames_all`.

[0069] Please refer to Figure 4 The present invention provides a grading system for the degree of human intervention in ground-based unmanned equipment, comprising:

[0070] Construction unit 101 is used to construct an evaluation system for the degree of human intervention in ground unmanned equipment. The evaluation system is characterized by evaluation indicators such as head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction.

[0071] The processing unit 102 is used to calculate the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency.

[0072] The determination unit 103 is used to determine the degree of human intervention of the ground unmanned equipment into five levels based on the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency: remote control level, intelligent assistance level, human guidance level, human assistance level, and autonomous level.

[0073] Optionally, ground-based unmanned vehicles are categorized into five levels based on the degree of human intervention during operation: Level 1 (Remote Control): The unmanned equipment neither undertakes self-decision-making nor operates autonomously; all perception, understanding, analysis, planning, and decision-making are performed by humans. The performance of the unmanned equipment is significantly influenced by humans. Level 2 (Intelligent Assistance): The unmanned equipment perceives its surrounding environment and reports to the operator. All analysis, planning, and decision-making are performed by humans, with most perception handled by humans. The operator controls the unmanned equipment's actions according to its reference standards. Level 3 (Human Guidance): The unmanned equipment perceives its environment and reports to the operator. Most analysis, planning, and decision-making are performed by humans, with perception and execution shared by the operator and the unmanned equipment. Level 4 (Human Assistance): The unmanned equipment perceives its environment and reports to the operator. Analysis, planning, and decision-making are shared by the operator and the unmanned equipment, with most perception and execution performed by the unmanned equipment. Level 5 (Autonomy): Within the functional and limiting scope of the unmanned equipment system, the unmanned equipment operates with the greatest degree of independence and autonomy. The performance of the ground-based unmanned equipment is minimally affected by the operator and is not under human control. Ground-based unmanned equipment senses its environment and reports to the operator. All sensing and execution actions are carried out by unmanned equipment, and most of the analysis, planning and decision-making are done by unmanned equipment. Collaboration may need to be done by humans.

[0074] Therefore, this invention divides the degree of human intervention into five levels: extremely low, low, medium, high, and extremely high, i.e., assessment levels 1 to 5. The corresponding index values ​​for each level are shown in Table 1 below.

[0075] Table 1

[0076]

[0077] Figure 5 This is a structural diagram of an exemplary hardware architecture of a computing device for classifying the degree of human intervention in ground-based unmanned equipment, as described in an embodiment of the present invention. The device 900 for classifying the degree of human intervention in ground-based unmanned equipment includes an input device 901, an input interface 902, a central processing unit 903, a memory 904, an output interface 905, and an output device 906. The input interface 902, central processing unit 903, memory 904, and output interface 905 are interconnected via a bus 910. The input device 901 and output device 906 are connected to the bus 910 via the input interface 902 and output interface 905, respectively, and are subsequently connected to other components of the device 900 for classifying the degree of human intervention in ground-based unmanned equipment.

[0078] Specifically, input device 901 receives input information from the outside and transmits the input information to central processing unit 903 through input interface 902; central processing unit 903 processes the input information based on computer-executable instructions stored in memory 904 to generate output information, temporarily or permanently stores the output information in memory 904, and then transmits the output information to output device 906 through output interface 905; output device 906 outputs the output information to the outside of ground unmanned equipment human intervention level classification device 900 for user use.

[0079] This invention also provides a computer storage medium storing computer program instructions, which, when executed by a processor, implement the user behavior recognition method of this invention.

[0080] In an exemplary embodiment, the computing device 900 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0081] It is understood that the memory 904 in this embodiment can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0082] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 904 storing a computer program, which can be executed by the central processing unit 903 of the computing device 900 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0083] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0084] The functional blocks shown in the above structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0085] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

Claims

1. A method for classifying the degree of human intervention in ground-based unmanned equipment, characterized in that, include: An evaluation system for the degree of human intervention in ground-based unmanned equipment is constructed. The evaluation system is characterized by evaluation indicators such as head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction. Calculate evaluation scores for head posture focus, facial expression focus, hand operation time percentage, and hand operation interaction frequency; Based on the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency, the degree of human intervention in ground unmanned equipment is determined into five levels: remote control level, intelligent assistance level, human guidance level, human assistance level, and autonomous level. The evaluation score for head posture focus is calculated, including: The operator's head video was acquired, and a deep learning method was used to calculate three angles of the operator's head: pitch angle, yaw angle, and roll angle. Images with pitch, yaw, or roll angles of the operator's head less than a certain threshold are selected as images of the operating interface to be focused on, and the number of frames of the images of the operating interface to be focused on is determined. Divide the number of frames of the focus operation interface image by the total number of frames of the video, and then multiply by 10 to obtain the score value of head posture focus. The evaluation score for facial expression focus includes: The operator's head video is acquired, and the operator's facial expression concentration is identified through deep transfer learning. The recognition result of each frame is marked as focused or unfocused. Divide the number of frames that indicate focus by the total number of frames in the video, and then multiply by 10 to get the score for facial focus. The evaluation score for the percentage of time spent on hand operations includes: Acquire video of operator's hand movements, use deep learning methods to detect hands and screen in the video, and calculate the intersection-union ratio (IOU) between the two; The number of image frames in which the hand is performing an operation is determined by comparing the IOU with a set threshold. If the IOU is greater than or equal to the threshold, it is considered that an operation is being performed; if it is less than the threshold, it is considered that no operation is being performed. Divide the number of image frames in which the hand is performing the operation by the total number of frames in the video, and then multiply by 10 to obtain the score value for the percentage of time the hand is performing the operation. The evaluation score for the frequency of hand gesture interaction is calculated, including: Acquire video of the operator's hand movements and calculate the number of interactions between hand movements and non-hand movements in the video; The total number of seconds and the number of interactions in this video are counted. The number of interactions is multiplied by the fastest interaction time, then divided by the total number of seconds in the video, and then multiplied by 10 to obtain the score value for the frequency of hand operation interaction.

2. The method for classifying the degree of human intervention in ground-based unmanned equipment according to claim 1, characterized in that, The deep learning approach includes: employing a multi-angle regression loss design, combining gradient optimization and adaptive methods, and using a convolutional neural network to estimate 3D head pose from image intensity.

3. The method for classifying the degree of human intervention in ground-based unmanned equipment according to claim 2, characterized in that, The facial expression focus of operators is identified using deep transfer learning methods, including: using the DAN-main network for facial expression focus detection. During testing, the face is first cropped, and the head rotation angle is obtained using OpenCV-Python. Face images with a rotation angle greater than a set threshold are classified as unfocused. Face images within the set threshold range are then input into the DAN-main network for facial expression focus detection.

4. A system for grading the degree of human intervention using the method described in claim 1, characterized in that, include: The construction unit is used to build an evaluation system for the degree of human intervention in ground unmanned equipment. The evaluation system is characterized by evaluation indicators such as head posture focus, facial expression focus, proportion of hand operation time, and frequency of hand operation interaction. The processing unit is used to calculate evaluation scores for head posture focus, facial expression focus, hand operation time percentage, and hand operation interaction frequency. The determination unit is used to determine the degree of human intervention in ground unmanned equipment into five levels based on the evaluation scores of head posture focus, facial expression focus, hand operation time ratio, and hand operation interaction frequency: remote control level, intelligent assistance level, human guidance level, human assistance level, and autonomous level.

5. A computing device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method for classifying the degree of human intervention in ground unmanned equipment as described in any one of claims 1-3.

6. A computer storage medium, characterized in that, The storage medium stores at least one executable instruction, which causes the processor to perform the operation corresponding to the method for classifying the degree of human intervention in ground unmanned equipment as described in any one of claims 1-3.

Images