Method, system, electronic device and medium for skill assessment of drone operator
By collecting and analyzing multi-source data from drone operators, especially eye-tracking and control command characteristics, and using a skills assessment model for real-time evaluation and diagnosis, the problems of inaccurate evaluation and high training costs in existing technologies are solved, enabling objective and quantitative evaluation and personalized training of agricultural drone operator skills.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEILONGJIANG HUIDA TECHNOLOGY CO LTD
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, the assessment of agricultural drone operator skills relies on the instructor's subjective observation, which leads to inaccurate assessments, high costs, difficulty in diagnosing skill deficiencies, and an inability to provide real-time adjustments to personalized training content.
By collecting multi-source data from drone operators, eye movement features and control command features are extracted, and a skill assessment model is used for objective, quantitative, and real-time evaluation, including eye scanning path, pupil and blink features. Combined with dynamic regions of interest and control command smoothness, a multi-dimensional skill score and defect diagnosis are output.
It enables objective, accurate, quantitative, and real-time assessment of drone operator skills, accurately diagnoses skill gaps, and optimizes training content through adaptive training strategies, thereby shortening the training cycle and reducing costs.
Smart Images
Figure CN122155493A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of drone operator skill assessment technology, and in particular to a method, system, electronic device and medium for assessing the skills of drone operators. Background Technology
[0002] Currently, the skills assessment and training of agricultural drone operators mainly rely on the subjective observation and experience judgment of instructors, lacking objective and quantitative assessment standards. Specifically, the following problems arise: (1) Instructors score by observing the operator's control actions and the drone's flight trajectory, which is easily affected by factors such as personal experience and fatigue, leading to inconsistent and subjective assessment results. (2) Existing assessment methods cannot capture the operator's micro-cognitive state in complex tasks, making it difficult to diagnose skill deficiencies. (3) The training process relies on repeated trial and error and instructor guidance, making it impossible to adjust the training content in a personalized and adaptive manner according to the operator's real-time cognitive load and skill deficiencies, resulting in long training cycles and high costs.
[0003] There are two main existing assessment methods. Method 1: An assessment system based on flight trajectory and task completion. The principle is to record the drone's flight trajectory, speed, and attitude angles using sensors such as GPS and IMU, and then score it based on task completion indicators such as spray coverage and operation time. Its drawback is that it only focuses on the aircraft's external performance, completely ignoring the operator's internal cognitive state. An operator with a smooth trajectory may be distracted, while an operator with frequent trajectory adjustments may be highly focused on processing complex environmental information. This approach cannot distinguish between these two fundamentally different skill states. Method 2: An analysis method based on video recording and post-event review. The principle is to record videos from the operator's first-person or third-person perspective, which are then reviewed and analyzed jointly by the instructor and operator after training. Its drawback is that it heavily relies on manual post-event analysis, cannot provide real-time feedback during training, misses the optimal intervention opportunity, and the analysis process is time-consuming and labor-intensive. Summary of the Invention
[0004] The technical problem this disclosure aims to solve is to overcome the shortcomings of existing technologies that assess the skills of agricultural drone operators through the subjective observation and experience judgment of instructors, which suffer from low assessment accuracy, high cost, and difficulty in diagnosing skill deficiencies. This disclosure provides a method, system, electronic device, and medium for assessing the skills of drone operators.
[0005] This disclosure solves the above-mentioned technical problems through the following technical solution:
[0006] The first aspect of this disclosure provides a method for assessing the skills of a drone operator, the method comprising:
[0007] Collect multi-source data from drone operators;
[0008] Obtain dynamic regions of interest within the task area;
[0009] The eye movement features of the drone operator are extracted from the multi-source data, including eye scanning path features and pupil and blinking features;
[0010] An attention pattern vector is obtained based on the dynamic region of interest, the eye scanning path features, and the pupil and blinking features.
[0011] Obtain the feature sequence of the attention pattern vector within the time window and the corresponding smoothness index of the control command;
[0012] The feature sequence of the attention pattern vector and the smoothness index of the control command are input into the skill assessment model to output a multidimensional skill score vector and a skill defect diagnosis label.
[0013] The skills of the drone operator are assessed based on the multidimensional skill scoring vector and the skill deficiency diagnostic labels.
[0014] Preferably, the step of collecting multi-source data from the drone operator includes:
[0015] Collect raw eye-tracking data from drone operators;
[0016] Acquire drone status data and drone operator control command data;
[0017] The eye-tracking raw data, the drone status data, and the control command data are time-stamp aligned and interpolated to generate the multi-source data.
[0018] Preferably, the step of obtaining the dynamic region of interest in the work area includes:
[0019] Acquire the drone's expected flight path area, obstacle projection area, current crop ridge area to be sprayed, and the area displaying the drone's health status on the drone operator's remote control screen;
[0020] Based on the expected flight path area of the drone at the next moment, the obstacle projection area, the crop row area that should be sprayed at present, and the drone health status area displayed on the drone operator's remote control screen, the dynamic area of interest in the operation area is obtained.
[0021] Preferably, the UAV status data includes the UAV's three-dimensional position, attitude angles, and flight mode; the skill assessment method further includes:
[0022] Based on the three-dimensional position of the UAV, the attitude angle, and the polygon of the work area, the expected projection area of the work area on the UAV operator's remote control screen is obtained.
[0023] Preferably, the skills assessment method further includes:
[0024] The number of fixation points, average fixation duration, and eye scan path length of the drone operator's eyes per unit time are obtained to obtain eye scan path characteristics.
[0025] And / or,
[0026] The skills assessment methods also include:
[0027] The pupil diameter change rate and blink count per unit time of the drone operator are obtained to obtain pupil and blink characteristics;
[0028] And / or,
[0029] The eye-tracking features also include attention allocation features, and the skill assessment method further includes:
[0030] The proportion of the gaze point falling into the dynamic region of interest within the time window and the response time of the first gaze into the obstacle projection area are obtained to obtain attention allocation characteristics.
[0031] Preferably, the skills assessment method further includes:
[0032] In response to the detection of the skill deficiency diagnostic label as insufficient obstacle attention, the number and frequency of virtual obstacles are dynamically increased in the next training cycle of the drone operator, and the task objective is set to the first preset mode;
[0033] And / or,
[0034] The skills assessment methods also include:
[0035] In response to the detection that the skill deficiency diagnostic tag is a status monitoring omission, in the next training cycle of the drone operator, the update frequency of the interface status information of the drone operator's remote controller screen is dynamically adjusted, and the task objective is set to the second preset mode.
[0036] And / or,
[0037] The skills assessment methods also include:
[0038] In response to the detection that the skill deficiency diagnostic label is overcompensated, the sensitivity of the flight control model is adjusted in the next training cycle of the UAV operator, and the mission objective is set to the third preset mode.
[0039] A second aspect of this disclosure provides a skills assessment system for drone operators, the skills assessment system comprising:
[0040] The data acquisition module is used to collect multi-source data from drone operators;
[0041] The first acquisition module is used to acquire dynamic regions of interest in the work area;
[0042] The extraction module is used to extract the eye movement features of the drone operator from the multi-source data. The eye movement features include eye scanning path features and pupil and blinking features.
[0043] The second acquisition module is used to acquire an attention pattern vector based on the dynamic region of interest, the eye scanning path features, and the pupil and blinking features.
[0044] The third acquisition module is used to acquire the feature sequence of the attention pattern vector within the time window and the corresponding smoothness index of the control command.
[0045] The fourth acquisition module is used to input the feature sequence of the attention pattern vector and the smoothness index of the control command into the skill assessment model to output a multi-dimensional skill score vector and a skill defect diagnosis label.
[0046] An assessment module is used to assess the skills of the drone operator based on the multidimensional skill rating vector and the skill defect diagnostic labels.
[0047] Preferably, the acquisition module includes:
[0048] The acquisition unit is used to collect raw eye-tracking data from the drone operator;
[0049] The first acquisition unit is used to acquire UAV status data and UAV operator control command data;
[0050] The generation unit is used to perform timestamp alignment and interpolation processing on the raw eye-tracking data, the UAV status data, and the control command data to generate the multi-source data.
[0051] Preferably, the first acquisition module includes:
[0052] The second acquisition unit is used to acquire the expected flight path area of the drone at the next moment, the obstacle projection area, the crop ridge area that should be sprayed at present, and the area where the drone's health status is displayed on the drone operator's remote control screen.
[0053] The third acquisition unit is used to acquire the dynamic region of interest in the operation area based on the expected flight path area of the UAV at the next moment, the obstacle projection area, the crop row area to be sprayed at present, and the health status area of the UAV displayed on the screen of the UAV operator's remote controller.
[0054] Preferably, the UAV status data includes the UAV's three-dimensional position, attitude angles, and flight mode; the skill assessment system further includes:
[0055] The fifth acquisition module is used to acquire the expected projection area of the working area on the remote control screen of the drone operator based on the three-dimensional position of the drone, the attitude angle, and the polygon of the working area.
[0056] Preferably, the skills assessment system further includes:
[0057] The sixth acquisition module is used to acquire the number of fixation points, average fixation duration, and eye scan path length of the drone operator's eyes per unit time, so as to obtain eye scan path features.
[0058] And / or,
[0059] The skills assessment system also includes:
[0060] The seventh acquisition module is used to acquire the pupil diameter change rate and the number of blinks per unit time of the drone operator in order to obtain pupil and blink characteristics;
[0061] And / or,
[0062] The eye-tracking features also include attention allocation features, and the skill assessment system further includes:
[0063] The eighth acquisition module is used to acquire the proportion of the gaze point falling into the dynamic interest region within the time window and the response time of the first gaze into the obstacle projection region, so as to obtain attention allocation characteristics.
[0064] Preferably, the skills assessment system further includes:
[0065] The first setting module is used to dynamically increase the number and frequency of virtual obstacles in the next training cycle of the drone operator in response to the detection that the skill deficiency diagnostic label is insufficient attention to obstacles, and set the task objective to the first preset mode.
[0066] And / or,
[0067] The skills assessment system also includes:
[0068] The second setting module is used to respond to the detection that the skill defect diagnosis tag is a status monitoring omission, and to dynamically adjust the update frequency of the interface status information of the drone operator's remote controller screen in the next training cycle of the drone operator, and set the task target to the second preset mode.
[0069] And / or,
[0070] The skills assessment system also includes:
[0071] The third setting module is used to adjust the sensitivity of the flight control model and set the mission objective to the third preset mode in the next training cycle of the UAV operator in response to the detection that the skill defect diagnostic label is over-compensated.
[0072] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the skill assessment method for drone operators described in the first aspect.
[0073] The fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the skill assessment method for drone operators described in the first aspect.
[0074] The fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the skill assessment method for a drone operator as described in the first aspect.
[0075] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.
[0076] The positive and progressive effects of this disclosure are as follows:
[0077] This disclosure integrates the eye-tracking characteristics of drone operators with the smoothness index of drone control commands, and uses a skills assessment model to achieve an objective, accurate, quantitative, and real-time assessment of drone operator skills, which can accurately diagnose the skill deficiencies of drone operators. Attached Figure Description
[0078] Figure 1 A flowchart of a skill assessment method for drone operators provided in Embodiment 1 of this disclosure.
[0079] Figure 2 This is a schematic diagram illustrating the division of dynamic regions of interest provided in embodiments 1 and 2 of this disclosure.
[0080] Figure 3 A schematic diagram of a remote control screen interface for real-time feedback assessment of drone operator skills provided in embodiments 1 and 2 of this disclosure.
[0081] Figure 4 This is a schematic diagram of the modules of the drone operator skills assessment system provided in Embodiment 2 of this disclosure.
[0082] Figure 5 This is a schematic diagram of the electronic device for implementing the skill assessment method for drone operators according to Embodiment 3 of this disclosure. Detailed Implementation
[0083] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.
[0084] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.
[0085] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good morals.
[0086] Example 1
[0087] Figure 1 A flowchart of a skill assessment method for a drone operator provided in Embodiment 1 of this disclosure is shown below. Figure 1 As shown, the skills assessment method includes:
[0088] S1. Collect multi-source data from drone operators;
[0089] In an optional implementation, S1 includes:
[0090] Collect raw eye-tracking data from drone operators;
[0091] In this embodiment, raw eye movement data of the drone operator during training is collected using an eye tracker. This raw eye movement data includes: fixation sequence G. t =(x t ,y t Pupil diameter P t Blink frequency B t Where t is the timestamp, x t y t The normalized coordinates of the gaze point in the remote control screen coordinate system.
[0092] Acquire drone status data and drone operator control command data;
[0093] In this embodiment, the agricultural drone's status data is acquired in real time through the drone flight control system. The agricultural drone's status data includes: three-dimensional position Pos.t =(lat t ,lon t ,alt t ), attitude angle Att t =(roll t pitch t ,yaw t Flight Mode FM t Among them, lat t lon t alt t These represent the latitude, longitude, and altitude of the agricultural drone's flight, respectively. t pitch t yaw t These are the roll angle, pitch angle, and yaw angle for agricultural drones.
[0094] In this embodiment, the drone operator's control command data (Ctrl) is obtained through the remote controller channel. t =(l-stick t ,r-stick t S-action t ), where l-stick t For left analog stick motion, r-stick t S-action is the amount of motion input for the right analog stick. t The spray switch is in operation.
[0095] The eye-tracking raw data, drone status data, and control command data are timestamped and interpolated to generate multi-source data.
[0096] In this implementation, a unified time base is established, and timestamp alignment and interpolation are performed on all input data streams to generate synchronized multi-source data MSD. t ={G t ,P t B t Pos t Att t FM t ,Ctrl t}
[0097] S2. Obtain the dynamic region of interest within the task area;
[0098] In an optional implementation, S2 includes:
[0099] Acquire the drone's expected flight path area, obstacle projection area, current crop ridge area to be sprayed, and the area displaying the drone's health status on the drone operator's remote control screen;
[0100] The dynamic region of interest in the operation area is obtained based on the expected flight path area of the drone at the next moment, the obstacle projection area, the crop row area that should be sprayed at present, and the area where the drone's health status is displayed on the drone operator's remote control screen.
[0101] In this embodiment, such as Figure 2 This is a schematic diagram illustrating the division of dynamic interest regions, such as... Figure 3 A schematic diagram of the remote controller screen interface for real-time feedback of drone operator skill assessment is shown. The expression of the dynamic area of interest is shown in formula (1):
[0102] ROI i =(ROI_Navigation,ROI_Obstacle,ROI_SprayTarget,ROI_StatusPanel)(1)
[0103] Among them, ROI i The ROI_Navigation represents the expected flight path of the drone at the next moment, the ROI_Obstacle represents the real-time detected obstacle projection area, the ROI_SprayTarget represents the crop row area that should be sprayed at the moment, and the ROI_StatusPanel represents the health status area of the drone displayed on the remote controller screen, which shows virtual instruments such as the drone's battery level, pesticide dosage, and signal strength.
[0104] It should be noted that each ROI_Navigation, ROI_Obstacle, ROI_SprayTarget, and ROI_StatusPanel uses a polygon Polygon_ROI=(p1,p2,...,p n ) expression, where p i Represents the vertices of a polygon representing a dynamic region of interest.
[0105] S3. Extract the eye movement features of the drone operator from multi-source data. The eye movement features include eye scanning path features and pupil and blinking features.
[0106] In an optional implementation, the skill assessment method further includes: obtaining the number of fixation points, average fixation duration, and eye scan path length of the drone operator's eyes per unit time to obtain eye scan path characteristics;
[0107] In this embodiment, the number N of fixation points per unit time is calculated. fixation Mean fixation duration D avg_fix The eyeball scanning path features are obtained by measuring the scanning path length L.
[0108] In an optional implementation, the skill assessment method further includes: obtaining the pupil diameter change rate and the number of blinks per unit time of the drone operator to obtain pupil and blink characteristics;
[0109] In this embodiment, the rate of change of pupil diameter ΔP is calculated. t And the number of blinks per unit time B rate Obtain pupil and blink characteristics.
[0110] S4. Obtain attention pattern vectors based on dynamic regions of interest, eye scanning path features, and pupil and blinking features;
[0111] In this embodiment, attention pattern vector AP is defined. t The expression is shown in formula (2):
[0112] AP t =[P_attention(ROI_Navigation),P_attention(ROI_Obstacle),P_attention(ROI_SprayTarget),P_attention(ROI_StatusPanel),N fixation D avg_fix ,L,ΔP t (2)
[0113] S5. Obtain the feature sequence of the attention pattern vector within the time window and the corresponding smoothness index of the control command.
[0114] In this embodiment, the expression for the smoothness index of the control command is shown in formula (3):
[0115] Smoothness t =1 / (1+α*Var(Δ²Ctrl t )) (3)
[0116] Among them, Var(Δ²Ctrl) t ) represents the variance of the second difference, and α represents the adjustment parameter that controls the degree of influence of the variance on the fraction; Δ²Ctrl t [i] = ΔCtrl t [i+1] - ΔCtrl t [i], i = 0, 1, ..., W-2; ΔCtrl t [i] = Ctrl t {i+1} - Ctrl t {i}, i = 0, 1, ..., W-1.
[0117] S6. Input the feature sequence of the attention pattern vector and the smoothness index of the control command into the skill assessment model to output a multi-dimensional skill score vector and a skill defect diagnosis label.
[0118] In this embodiment, a gradient boosting decision tree model is used as the skill assessment model. This skill assessment model uses the feature sequence {AP} within a time window W. t-W ,...,AP t} and the corresponding control command smoothness index Smoothness t As input, the skills assessment model outputs a multi-dimensional skills score vector, Score. t And skill deficiency diagnostic label Defect t .
[0119] Furthermore, the expression for the multidimensional skill rating vector is shown in formula (4):
[0120] Score t =[S situation ,S decision ,S control ,S workload (4)
[0121] Among them, Score t S represents a multidimensional skill rating vector. situation The situational awareness score is represented by the formula shown in formula (5):
[0122] S situation =β1*f(P_attention(ROI_Navigation))+β2*g(T first_response )+β3*h(P_attention(ROI_Obstacle)) (5), where β1, β2, and β3 all represent coefficients, β1+β2+β3=1, and f, g, and h all represent functions.
[0123] S decision The decision efficiency score is represented by the formula shown in formula (6):
[0124] S decision =γ1*Correlation(Attention_shift,Control_change)+γ2*(1-Decision_latency)+γ3*(1-L normalized(6), where γ1, γ2, and γ3 all represent coefficients, γ1+γ2+γ3=1, Attention_shift represents the attention parameter, Control_chang represents the control command parameter, Decision_latency represents the decision parameter, and L normalized This represents the path length planning index.
[0125] S control The control precision score is represented by formula (7).
[0126] S control =δ1*Smoothness t +δ2*(1-Tracking_error)+δ3*(1-Overshoot_ratio) (7), where δ1, δ2, and δ3 all represent coefficients, δ1+δ2+δ3=1, Smoothness t This represents the smoothness index of control commands, Tracking_error represents the spray tracking index, and Overshoot_ratio represents the overshoot index.
[0127] S workload The cognitive load score is represented by the formula shown in formula (8):
[0128] S workload =ε1*(1-ΔP t_normalized )+ε2*(1-B rate_normalized )+ε3*(1-N fixation_normalized (8),
[0129] Where ε1, ε2, and ε3 all represent coefficients, ε1 + ε2 + ε3 = 1, ΔP t_normalized B represents the planning index for the rate of change of pupil diameter. rate_normalized N represents the blink planning indicator. fixation_normalized This represents the fixation point planning index per unit of time.
[0130] S7. Assess drone operator skills based on multidimensional skill rating vectors and skill defect diagnostic labels.
[0131] In this embodiment, the multidimensional skill score vector Score is used. t And skill deficiency diagnostic label Defect t Real-time visual feedback is provided to the operator's remote control interface and the instructor's monitoring interface.
[0132] The training system is based on Score t and Defect t The historical sequence is used to call the adaptive training strategy module.
[0133] It should be noted that the skill deficiency diagnostic labels include at least one of the following: insufficient attention to obstacles, omission of status monitoring, and overcompensation of manipulation.
[0134] This implementation method integrates the eye-tracking characteristics of the drone operator with the smoothness index of the drone's control commands. Through a skills assessment model, it achieves an objective, accurate, quantitative, and real-time assessment of the drone operator's skills, and can accurately diagnose the drone operator's skill deficiencies.
[0135] In one optional implementation, the UAV status data includes the UAV's three-dimensional position, attitude angles, and flight mode; the skill assessment method further includes:
[0136] The expected projection area of the work area on the drone operator's remote controller screen is obtained based on the drone's three-dimensional position and attitude angles, as well as the polygon of the work area.
[0137] In this embodiment, based on the three-dimensional position Pos t and attitude angle Att t By combining the work area polygon Polygon_task in the loaded farmland digital map, perspective projection transformation is used to calculate the expected projection area of the work area on the drone operator's remote control screen from the perspective of the drone's onboard camera.
[0138] In an optional implementation, the eye-tracking features further include attention allocation features, and the skill assessment method further includes:
[0139] The proportion of the gaze point falling into the dynamic region of interest within the time window and the response time of the first gaze into the obstacle projection region are obtained to obtain attention allocation characteristics.
[0140] In this implementation, for each dynamic region of interest (ROI) i Calculate the number of times the gaze point falls within the dynamic region of interest (ROI) within a time window W. i The proportion within P_attention(ROI) i =Count(G t ∈ROI i ) / Count(all G t Simultaneously, the response time T for the first gaze of the obstacle projection region ROI_Obstacle is calculated. first_response .
[0141] In an optional implementation, the skills assessment method further includes:
[0142] In response to the detection of a skill deficiency diagnostic label as insufficient obstacle attention, the number and frequency of virtual obstacles are dynamically increased in the next training cycle of the drone operator, and the task objective is set to the first preset mode;
[0143] In this embodiment, the first preset mode is to successfully avoid all obstacles while ensuring spray coverage;
[0144] Specifically, if a skill deficiency diagnostic label "Defect" is detected... t If the drone operator is not paying enough attention to obstacles, the number and frequency of virtual obstacles will be dynamically increased in the next training cycle, and the task objective will be to successfully avoid all obstacles while ensuring spray coverage.
[0145] In an optional implementation, the skills assessment method further includes:
[0146] In response to the detection of a skill deficiency diagnostic tag indicating a status monitoring omission, the update frequency of the drone operator's remote controller screen interface status information is dynamically adjusted in the next training cycle, and the task objective is set to the second preset mode.
[0147] In this embodiment, the second preset mode is to accurately report changes in system status while completing the spraying operation;
[0148] Specifically, if a skill deficiency diagnostic label "Defect" is detected... t If there is a gap in status monitoring, the update frequency of the remote controller interface status information will be dynamically adjusted in the next training cycle of the drone operator, and the task objective will be set to accurately report system status changes while completing the spraying operation.
[0149] In an optional implementation, the skills assessment method further includes:
[0150] In response to the detection of a skill deficiency diagnostic label indicating overcompensation, the sensitivity of the flight control model is adjusted in the next training cycle for the drone operator, and the mission objective is set to the third preset mode.
[0151] In this embodiment, the third preset mode is to maintain a smooth and stable flight trajectory under disturbance conditions, and the rate of change of control commands is lower than a set threshold.
[0152] Specifically, if a skill deficiency diagnostic label "Defect" is detected... t To compensate for overcompensation, the sensitivity of the flight control model is adjusted specifically in the next training cycle for the drone operator, and the mission objective is set to maintain a smooth and stable flight trajectory under disturbance conditions, and to keep the rate of change of control commands below a set threshold.
[0153] This implementation method transforms subjective experience judgments into objective data scores through the fusion analysis of quantitative eye-tracking features, UAV status data, and control command data. It can accurately identify skill deficiencies that cannot be detected by other methods, providing a more comprehensive evaluation dimension and high consistency of results. Furthermore, this implementation method can run online in real time with a latency of less than 100ms. It can provide immediate diagnosis and analysis at the moment of a skill error by the UAV operator or within a short period of 2 seconds, and can provide real-time visual or auditory prompts through the system interface. This enables simultaneous operation, evaluation, and correction, greatly shortening the feedback loop.
[0154] Example 2
[0155] Corresponding to the aforementioned embodiment of a skill assessment method for drone operators, this disclosure also provides an embodiment of a skill assessment system for drone operators.
[0156] Figure 4 This is a schematic diagram of a skill assessment system for drone operators provided in Embodiment 2 of this disclosure, as shown below. Figure 4 As shown, the skills assessment system includes:
[0157] Acquisition module 21 is used to collect multi-source data from the drone operator;
[0158] In an optional implementation, the acquisition module 21 includes:
[0159] The acquisition unit is used to collect raw eye-tracking data from the drone operator;
[0160] In this embodiment, raw eye movement data of the drone operator during training is collected using an eye tracker. This raw eye movement data includes: fixation sequence G. t =(x t ,y t Pupil diameter P t Blink frequency B t Where t is the timestamp, x t y t The normalized coordinates of the gaze point in the remote control screen coordinate system.
[0161] The first acquisition unit is used to acquire UAV status data and UAV operator control command data;
[0162] In this embodiment, the agricultural drone's status data is acquired in real time through the drone flight control system. The agricultural drone's status data includes: three-dimensional position Pos. t =(lat t ,lon t ,alt t ), attitude angle Att t=(roll t pitch t ,yaw t Flight Mode FM t Among them, lat t lon t alt t These represent the latitude, longitude, and altitude of the agricultural drone's flight, respectively. t pitch t yaw t These are the roll angle, pitch angle, and yaw angle for agricultural drones.
[0163] In this embodiment, the drone operator's control command data (Ctrl) is obtained through the remote controller channel. t =(l-stick t ,r-stick t S-action t ), where l-stick t For left analog stick motion, r-stick t S-action is the amount of motion input for the right analog stick. t The spray switch is in operation.
[0164] The generation unit is used to perform timestamp alignment and interpolation on raw eye-tracking data, UAV status data, and control command data to generate multi-source data.
[0165] In this implementation, a unified time base is established, and timestamp alignment and interpolation are performed on all input data streams to generate synchronized multi-source data MSD. t ={G t ,P t B t Pos t Att t FM t ,Ctrl t}
[0166] The first acquisition module 22 is used to acquire dynamic regions of interest in the work area;
[0167] In an optional implementation, the first acquisition module 22 includes:
[0168] The second acquisition unit is used to acquire the expected flight path area of the drone at the next moment, the obstacle projection area, the crop ridge area that should be sprayed at present, and the area where the drone's health status is displayed on the drone operator's remote control screen.
[0169] The third acquisition unit is used to acquire the dynamic region of interest in the operation area based on the expected flight path area of the UAV at the next moment, the obstacle projection area, the crop row area that should be sprayed at present, and the area where the health status of the UAV is displayed on the screen of the UAV operator's remote controller.
[0170] In this embodiment, such as Figure 2 This is a schematic diagram illustrating the division of dynamic interest regions, such as... Figure 3 A schematic diagram of the remote control screen interface for real-time feedback of drone operator skill assessment is provided. The expression of the dynamic interest area is shown in formula (1) in Example 1.
[0171] Extraction module 23 is used to extract the eye movement features of the drone operator from multi-source data. The eye movement features include eye scanning path features and pupil and blinking features.
[0172] In an optional implementation, the skills assessment system further includes:
[0173] The sixth acquisition module is used to acquire the number of fixation points, average fixation duration, and eye scan path length of the drone operator's eyes per unit time, so as to obtain eye scan path features.
[0174] In this embodiment, the number N of fixation points per unit time is calculated. fixation Mean fixation duration D avg_fix The eyeball scanning path features are obtained by measuring the scanning path length L.
[0175] In an optional implementation, the skills assessment system further includes:
[0176] The seventh acquisition module is used to acquire the pupil diameter change rate and the number of blinks per unit time of the drone operator in order to obtain pupil and blink characteristics;
[0177] In this embodiment, the rate of change of pupil diameter ΔP is calculated. t And the number of blinks per unit time B rate Obtain pupil and blink characteristics.
[0178] The second acquisition module 24 is used to acquire attention pattern vectors based on dynamic interest regions, eye scanning path features and pupil and blinking features;
[0179] In this embodiment, attention pattern vector AP is defined. t The expression is shown in formula (2) in Example 1.
[0180] The third acquisition module 25 is used to acquire the feature sequence of the attention pattern vector within the time window and the corresponding smoothness index of the control command.
[0181] In this embodiment, the expression for the smoothness index of the control command is as shown in formula (3) in Example 1.
[0182] The fourth acquisition module 26 is used to input the feature sequence of the attention pattern vector and the smoothness index of the control command into the skill assessment model to output a multi-dimensional skill score vector and a skill defect diagnosis label.
[0183] In this embodiment, a gradient boosting decision tree model is used as the skill assessment model. This skill assessment model uses the feature sequence {AP} within a time window W. t-W ,...,AP t} and the corresponding control command smoothness index Smoothness t As input, the skills assessment model outputs a multi-dimensional skills score vector, Score. t And skill deficiency diagnostic label Defect t .
[0184] Furthermore, the expression for the multidimensional skill rating vector is shown in formula (4) in Example 1.
[0185] Among them, Score t S represents a multidimensional skill rating vector. situation S represents the situational awareness score, which is calculated as shown in formula (5) in Example 1. decision S represents the decision efficiency score, which is calculated as shown in formula (6) in Example 1. control The control accuracy score is represented by the formula (7) shown in Example 1. workload The cognitive load score is represented by the formula (8) in Example 1.
[0186] Assessment module 27 is used to assess the skills of drone operators based on multidimensional skill rating vectors and skill defect diagnostic labels.
[0187] In this embodiment, the multidimensional skill score vector Score is used. t And skill deficiency diagnostic label Defect t Real-time visual feedback is provided to the operator's remote control interface and the instructor's monitoring interface.
[0188] The training system is based on Score t and Defect t The historical sequence is used to call the adaptive training strategy module.
[0189] It should be noted that the skill deficiency diagnostic labels include at least one of the following: insufficient attention to obstacles, omission of status monitoring, and overcompensation of manipulation.
[0190] This implementation method integrates the eye-tracking characteristics of the drone operator with the smoothness index of the drone's control commands. Through a skills assessment model, it achieves an objective, accurate, quantitative, and real-time assessment of the drone operator's skills, and can accurately diagnose the drone operator's skill deficiencies.
[0191] In one optional implementation, the UAV status data includes the UAV's three-dimensional position, attitude angles, and flight mode; the skill assessment system also includes:
[0192] The fifth acquisition module is used to acquire the expected projection area of the working area on the drone operator's remote controller screen based on the three-dimensional position and attitude angle of the drone and the polygon of the working area.
[0193] In this embodiment, based on the three-dimensional position Pos t and attitude angle Att t By combining the work area polygon Polygon_task in the loaded farmland digital map, perspective projection transformation is used to calculate the expected projection area of the work area on the drone operator's remote control screen from the perspective of the drone's onboard camera.
[0194] In an optional implementation, the eye-tracking features further include attention allocation features, and the skill assessment system further includes:
[0195] The eighth acquisition module is used to acquire the proportion of the gaze point falling into the dynamic region of interest within the time window and the response time of the first gaze into the obstacle projection area, so as to obtain attention allocation characteristics.
[0196] In this implementation, for each dynamic region of interest (ROI) i Calculate the number of times the gaze point falls within the dynamic region of interest (ROI) within a time window W. i The proportion within P_attention(ROI) i =Count(G t ∈ROI i ) / Count(all G t Simultaneously, the response time T for the first gaze of the obstacle projection region ROI_Obstacle is calculated. first_response .
[0197] In an optional implementation, the skills assessment system further includes:
[0198] The first setting module is used to dynamically increase the number and frequency of virtual obstacles in the next training cycle of the drone operator in response to the detection of a skill deficiency diagnostic label as insufficient attention to obstacles, and set the task objective to the first preset mode.
[0199] In this embodiment, the first preset mode is to successfully avoid all obstacles while ensuring spray coverage;
[0200] Specifically, if a skill deficiency diagnostic label "Defect" is detected... t If the drone operator is not paying enough attention to obstacles, the number and frequency of virtual obstacles will be dynamically increased in the next training cycle, and the task objective will be to successfully avoid all obstacles while ensuring spray coverage.
[0201] In an optional implementation, the skills assessment system further includes:
[0202] The second setting module is used to respond to the detection of a skill defect diagnostic tag as a status monitoring omission, and to dynamically adjust the update frequency of the interface status information of the drone operator's remote controller screen in the next training cycle of the drone operator, and set the task target to the second preset mode.
[0203] In this embodiment, the second preset mode is to accurately report changes in system status while completing the spraying operation;
[0204] Specifically, if a skill deficiency diagnostic label "Defect" is detected... t If there is a gap in status monitoring, the update frequency of the remote controller interface status information will be dynamically adjusted in the next training cycle of the drone operator, and the task objective will be set to accurately report system status changes while completing the spraying operation.
[0205] In an optional implementation, the skills assessment system further includes:
[0206] The third setting module is used to adjust the sensitivity of the flight control model and set the mission objective to the third preset mode in the next training cycle of the UAV operator in response to the detection of a skill deficiency diagnostic label as over-compensation.
[0207] In this embodiment, the third preset mode is to maintain a smooth and stable flight trajectory under disturbance conditions, and the rate of change of control commands is lower than a set threshold.
[0208] Specifically, if a skill deficiency diagnostic label "Defect" is detected... t To compensate for overcompensation, the sensitivity of the flight control model is adjusted specifically in the next training cycle for the drone operator, and the mission objective is set to maintain a smooth and stable flight trajectory under disturbance conditions, and to keep the rate of change of control commands below a set threshold.
[0209] This implementation method transforms subjective experience judgments into objective data scores through the fusion analysis of quantitative eye-tracking features, UAV status data, and control command data. It can accurately identify skill deficiencies that cannot be detected by other methods, providing a more comprehensive evaluation dimension and high consistency of results. Furthermore, this implementation method can run online in real time with a latency of less than 100ms. It can provide immediate diagnosis and analysis at the moment of a skill error by the UAV operator or within a short period of 2 seconds, and can provide real-time visual or auditory prompts through the system interface. This enables simultaneous operation, evaluation, and correction, greatly shortening the feedback loop.
[0210] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.
[0211] Example 3
[0212] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the skill assessment method for drone operators described in any of the above embodiments. Figure 5 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0213] like Figure 5 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).
[0214] Bus 93 includes a data bus, an address bus, and a control bus.
[0215] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.
[0216] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0217] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the skill assessment method for drone operators provided in any of the above embodiments.
[0218] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 96. Figure 5 As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.
[0219] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.
[0220] Example 4
[0221] Embodiment 4 of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the skill assessment method for drone operators provided in any of the above embodiments.
[0222] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.
[0223] Example 5
[0224] Embodiment 5 of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the skill assessment method for drone operators described in any of the above embodiments.
[0225] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.
[0226] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.
[0227] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.
Claims
1. A method for assessing the skills of a drone operator, characterized in that, The skills assessment methods include: Collect multi-source data from drone operators; Obtain dynamic regions of interest within the task area; The eye movement features of the drone operator are extracted from the multi-source data, including eye scanning path features and pupil and blinking features; An attention pattern vector is obtained based on the dynamic region of interest, the eye scanning path features, and the pupil and blinking features. Obtain the feature sequence of the attention pattern vector within the time window and the corresponding smoothness index of the control command; The feature sequence of the attention pattern vector and the smoothness index of the control command are input into the skill assessment model to output a multidimensional skill score vector and a skill defect diagnosis label. The skills of the drone operator are assessed based on the multidimensional skill scoring vector and the skill deficiency diagnostic labels.
2. The skill assessment method for drone operators as described in claim 1, characterized in that, The steps for collecting multi-source data from drone operators include: Collect raw eye-tracking data from drone operators; Acquire drone status data and drone operator control command data; The eye-tracking raw data, the drone status data, and the control command data are time-stamp aligned and interpolated to generate the multi-source data.
3. The skill assessment method for drone operators as described in claim 1, characterized in that, The step of obtaining the dynamic region of interest in the task area includes: Acquire the drone's expected flight path area, obstacle projection area, current crop ridge area to be sprayed, and the area displaying the drone's health status on the drone operator's remote control screen; Based on the expected flight path area of the drone at the next moment, the obstacle projection area, the crop row area that should be sprayed at present, and the drone health status area displayed on the drone operator's remote control screen, the dynamic area of interest in the operation area is obtained.
4. The skill assessment method for drone operators as described in claim 2, characterized in that, The UAV status data includes the UAV's three-dimensional position, attitude angles, and flight mode; the skill assessment method also includes: Based on the three-dimensional position of the UAV, the attitude angle, and the polygon of the work area, the expected projection area of the work area on the UAV operator's remote control screen is obtained.
5. The skill assessment method for drone operators as described in claim 1, characterized in that, The skills assessment methods also include: The number of fixation points, average fixation duration, and eye scan path length of the drone operator's eyes per unit time are obtained to obtain eye scan path characteristics. And / or, The skills assessment methods also include: The pupil diameter change rate and blink count per unit time of the drone operator are obtained to obtain pupil and blink characteristics; And / or, The eye-tracking features also include attention allocation features, and the skill assessment method further includes: The proportion of the gaze point falling into the dynamic region of interest within the time window and the response time of the first gaze into the obstacle projection area are obtained to obtain attention allocation characteristics.
6. The skill assessment method for drone operators as described in claim 1, characterized in that, The skills assessment methods also include: In response to the detection of the skill deficiency diagnostic label as insufficient obstacle attention, the number and frequency of virtual obstacles are dynamically increased in the next training cycle of the drone operator, and the task objective is set to the first preset mode; And / or, The skills assessment methods also include: In response to the detection that the skill deficiency diagnostic tag is a status monitoring omission, in the next training cycle of the drone operator, the update frequency of the interface status information of the drone operator's remote controller screen is dynamically adjusted, and the task objective is set to the second preset mode. And / or, The skills assessment methods also include: In response to the detection that the skill deficiency diagnostic label is overcompensated, the sensitivity of the flight control model is adjusted in the next training cycle of the UAV operator, and the mission objective is set to the third preset mode.
7. A skill assessment system for drone operators, characterized in that, The skills assessment system includes: The data acquisition module is used to collect multi-source data from drone operators; The first acquisition module is used to acquire dynamic regions of interest in the work area; The extraction module is used to extract the eye movement features of the drone operator from the multi-source data. The eye movement features include eye scanning path features and pupil and blinking features. The second acquisition module is used to acquire an attention pattern vector based on the dynamic region of interest, the eye scanning path features, and the pupil and blinking features. The third acquisition module is used to acquire the feature sequence of the attention pattern vector within the time window and the corresponding smoothness index of the control command. The fourth acquisition module is used to input the feature sequence of the attention pattern vector and the smoothness index of the control command into the skill assessment model to output a multi-dimensional skill score vector and a skill defect diagnosis label. An assessment module is used to assess the skills of the drone operator based on the multidimensional skill rating vector and the skill defect diagnostic labels.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the skill assessment method for the drone operator as described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the skill assessment method for the drone operator as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the skill assessment method for drone operators as described in any one of claims 1 to 6.