Agricultural machine driver operation state sensing system based on digital twinning

Abstract

The invention discloses an agricultural machinery driver operation state sensing system based on digital twinning, and belongs to the technical field of agricultural machinery intelligence and digital twinning. The invention aims to solve the problems of incomplete operation state sensing, inaccurate risk early warning and low man-machine ring cooperation efficiency caused by single sensing dimension, isolated data mode, static risk assessment and lack of closed-loop control in the existing agricultural machine state monitoring technology. According to the method, three types of state vectors of machinery, a driver and an environment are synchronously collected through a multi-source sensor, and after preprocessing, a fusion feature vector is generated by adopting a weighted multi-mode fusion model. Based on the vector, performing state classification by using a multilayer perceptron neural network, and describing the interaction between a machine and a driver state in combination with a coupling constraint dynamic model; meanwhile, a fatigue index and an environment risk index are calculated, and real-time accurate recognition of the operation state is achieved through a comprehensive discrimination function. And a discrimination result is fed back to the digital twin virtual model in real time, and online parameter optimization and closed-loop control are driven.

Description

Technical Field

[0001] This invention relates to the field of intelligent agricultural machinery and digital twin technology, specifically to a digital twin-based agricultural machinery driver operation status perception system, which falls under the category of agricultural machinery operation monitoring, driver status detection, and agricultural operation safety control technology. Background Technology

[0002] Agricultural mechanization is the core driving force of modern agricultural development, and intelligent agricultural machinery such as large tractors, combine harvesters, and precision seeders have become the mainstay of modern agricultural production. However, with the extension of agricultural machinery operation time and the increasing complexity of the operating environment, the safety and efficiency issues of agricultural machinery operation have become increasingly prominent, becoming one of the key bottlenecks restricting the development of agricultural modernization.

[0003] Currently, agricultural machinery condition monitoring technology mainly focuses on the collection and analysis of mechanical parameters. By monitoring mechanical parameters such as engine speed, oil pressure, and water temperature, early warning of malfunctions can be achieved. While such systems can ensure machinery safety to a certain extent, their perspective is limited to the "machine" itself, neglecting the core element of the "human-machine-environment" system: the driver. In long-term, high-intensity, and monotonous repetitive field work, drivers are prone to fatigue, distraction, and other physiological changes. These changes directly affect operational accuracy and reaction speed, and are the main human factors causing operational accidents.

[0004] In driver condition monitoring, existing technologies mostly employ single-modal detection methods. For example, vision-based fatigue detection uses cameras to capture facial features of the driver (such as the frequency of eye closure and the number of yawns) for assessment. However, the drastic changes in lighting and severe dust interference in field operations, coupled with the driver wearing protective gear such as sun hats and masks, significantly impact the reliability and accuracy of visual detection. Furthermore, visual detection alone cannot quantify the driver's physiological load (such as heart rate variability and skin conductance), making it difficult to comprehensively assess their fatigue level and stress state.

[0005] Furthermore, the efficiency and safety of agricultural machinery operations are highly dependent on the complex field environment. Environmental factors such as terrain slope, surface flatness, and soil moisture not only affect the stability and quality of agricultural machinery operation but are also closely related to the driver's workload and fatigue accumulation. For example, when operating on rugged or steep terrain, the driver needs to frequently adjust the steering wheel to maintain the direction of travel, significantly increasing their mental and physical burden and accelerating fatigue. Current technical solutions mostly treat the environment as an isolated external condition, failing to conduct dynamic correlation analysis and integrated modeling with the machine's state and the driver's physiological behavior, resulting in one-sided state perception and inaccurate risk assessment.

[0006] Digital twin technology, as a cutting-edge means to achieve real-time interaction and integration between the physical world and cyberspace, has demonstrated enormous potential in fields such as industrial manufacturing and smart cities. This technology provides a new paradigm for the state perception, prediction, and control of complex systems by constructing virtual models that closely match physical entities and enabling bidirectional data-driven iteration and optimization between the two. However, in the field of agricultural machinery, existing digital twin applications mostly focus on performance simulation and fault diagnosis of the machinery itself, and have not yet formed a comprehensive operational state perception system that covers all elements of "machinery-driver-environment," integrates multimodal real-time data, and possesses dynamic risk assessment and closed-loop control capabilities.

[0007] Therefore, the existing technology has the following obvious shortcomings:

[0008] 1. Single perception dimension: It may only focus on the machine or only on the driver, lacking an integrated collaborative perception architecture that integrates "human-machine-environment".

[0009] 2. Data modal isolation: Visual, sensor, and environmental data are isolated from each other, and effective multimodal information fusion has not been achieved, resulting in state recognition being susceptible to interference and lacking accuracy.

[0010] 3. Static Risk Assessment: Assessments of fatigue and environmental risks are mostly based on static thresholds or data from a single moment, failing to reflect the dynamic evolution process of multiple factors coupled together.

[0011] 4. Lack of closed-loop system: The status perception, risk assessment and operation control links are disconnected, and a real-time feedback and optimization closed loop based on digital twin has not been formed.

[0012] In summary, there is an urgent need to develop an innovative technical solution that can deeply integrate multi-source heterogeneous data on machine status, driver physiological behavior, and the working environment, and leverage digital twin technology to construct a real-time, accurate, and robust global operational status perception and dynamic risk assessment system, fundamentally improving the safety boundaries and overall efficiency of agricultural machinery operations. This invention is proposed against this backdrop. Summary of the Invention

[0013] To address the problems of incomplete operational status perception, inaccurate risk warning, and low efficiency of human-machine-environment collaboration caused by existing agricultural machinery condition monitoring technologies, which suffer from single perception dimensions, isolated data modalities, static risk assessment, and lack of closed-loop control, this invention provides an agricultural machinery driver operational status perception system based on digital twins.

[0014] The present invention discloses a digital twin-based agricultural machinery driver operation status perception system, comprising:

[0015] The data acquisition module is used to acquire mechanical state vectors. Driver physiological behavior vector and the working environment state vector ;

[0016] The data processing module is used to perform data normalization and time synchronization alignment on the collected vectors.

[0017] The feature fusion module is used to fuse the processed multimodal data using a weighted multimodal fusion model to obtain a fused feature vector. ;

[0018] The job status modeling module uses a multilayer perceptron neural network to process the fused feature vector. Classify and output a set of job status categories. The module also establishes a dynamic model to describe the coupling constraint relationship between mechanical state and driver's physiological state.

[0019] The risk assessment module is used to assess the risk based on the driver's physiological behavior vector. Calculate the fatigue index And according to the work environment state vector Calculate the environmental risk index ;

[0020] The work status comprehensive judgment module is used to determine the work status based on the probability distribution and the fatigue index. and the environmental risk index The final judgment job status is calculated and output through the discriminant function. ;

[0021] The system feedback and digital twin model update module is used to update the final judgment operation status. Feedback is fed into the digital twin virtual model, and online learning algorithms are used to dynamically adjust the model parameters.

[0022] Preferably, the mechanical state vector for:

[0023]

[0024] In the formula, Engine speed, For driving speed, Hydraulic pressure, For steering wheel angle, For throttle opening, Braking pressure, For the height of agricultural machinery, This refers to the rotational speed of the power output shaft.

[0025] Preferably, the driver's physiological behavior vector for:

[0026]

[0027] In the formula, The aspect ratio of the eyes, The length-to-width ratio of the mouth The percentage of eyelid closure. The head tilt angle, The yaw angle of the nose. For head roll angle, Heart rate, This is the skin conductance response. Body temperature.

[0028] Preferably, the working environment state vector for:

[0029]

[0030] In the formula, Light intensity, For ambient temperature, For ambient humidity, For terrain slope, For surface roughness, This refers to the duration of continuous operation.

[0031] Preferably, in the feature fusion module, the fused feature vector Obtained through the following weighted multimodal fusion model:

[0032]

[0033] In the formula, For the mechanical state fusion weight vector, The driver's physiological behavior fusion weight vector, The weight vector for fusing the operational environment state satisfies: ;

[0034] This is the normalized and aligned mechanical state vector. This is the normalized and aligned vector of the driver's physiological behavior. This is the normalized and aligned working environment state vector. This indicates vector concatenation.

[0035] Preferably, in the job status modeling module, the probability distribution Obtained through the following model:

[0036]

[0037] In the formula, For activation function, For the first The weights of the layer network, For the first Bias of the layer network;

[0038] Probability distribution of each category in the output job status category set , Corresponding to categories The probability distribution, This represents the number of categories.

[0039] Preferably, in the operation state modeling module, the dynamic model is:

[0040]

[0041]

[0042] For mechanical state dynamics functions, This refers to mechanical state-related noise or random disturbance terms.

[0043] Let be a function representing the changes in the driver's physiological state. This refers to physiological state-related noise or random disturbance terms.

[0044] Preferably, in the risk assessment module, the fatigue index Calculated using the following model:

[0045]

[0046] In the formula, , , , , An indicator of the driver's physiological behavior vector;

[0047] These are the weighting coefficients for each indicator in the driver's physiological behavior vector. .

[0048] Preferably, in the risk assessment module, the environmental risk index Calculated using the following model:

[0049]

[0050] In the formula, , , , , These are the indicators of the operational environment state vector;

[0051] This represents the maximum value of the light intensity index. This represents the maximum value of the ambient temperature index. This represents the maximum value of the terrain slope index;

[0052] The weight coefficients of each indicator in the business environment state vector are: .

[0053] Preferably, in the comprehensive operation status discrimination module, the discrimination function is:

[0054]

[0055] In the formula, The state-related risk correction factor function is obtained by the following formula:

[0056]

[0057] In the formula, The state sensitivity coefficient, As the fatigue weighting factor, As an environmental weighting factor, it satisfies: .

[0058] The beneficial effects of this invention are:

[0059] 1. Full-element perception: Simultaneously monitor data from machinery, driver, and environment to achieve global operational status modeling, overcoming the limitations of existing systems that only focus on a single dimension.

[0060] 2. Multimodal fusion: By using a weighted multimodal fusion model, sensor data, visual features and environmental information are organically combined, which improves the accuracy and robustness of state recognition.

[0061] 3. Enhanced physical constraints: The introduction of a coupled dynamic model of mechanical state and driver's physiological state makes the state perception more consistent with actual physical laws, improving the interpretability and reliability of the model.

[0062] 4. Risk assessment closed loop: Innovatively introduce a dual assessment system of fatigue risk index and environmental risk index, and ensure the reasonable weight of risk factors in the final state judgment through risk correction factor function.

[0063] 5. Real-time interaction with digital twins: The judgment results are synchronized to the digital twin virtual model in real time, realizing the integration of virtual and real and online parameter optimization, enabling the system to have self-learning and self-adaptive capabilities.

[0064] 6. Proactive safety intervention: When a high-risk condition is detected (such as severe fatigue, mechanical failure, or dangerous environment), the system can issue an audible and visual warning, or link the agricultural machinery through the control interface to perform proactive intervention actions such as deceleration and stopping, which significantly improves operational safety.

[0065] 7. Improved operational efficiency: Through precise perception of operational status and optimization suggestions, it helps drivers adjust their operational strategies, reduce ineffective operational time, and improve overall operational efficiency. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of a digital twin-based agricultural machinery driver operation status perception system as described in this invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0069] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0070] Specific Implementation Method 1: The following is combined with... Figure 1 This embodiment describes a digital twin-based agricultural machinery driver operation status perception system, comprising:

[0071] The data acquisition module is used to acquire mechanical state vectors. Driver physiological behavior vector and the working environment state vector ;

[0072] The data processing module is used to perform data normalization and time synchronization alignment on the collected vectors.

[0073] The feature fusion module is used to fuse the processed multimodal data using a weighted multimodal fusion model to obtain a fused feature vector. ;

[0074] The job status modeling module uses a multilayer perceptron neural network to process the fused feature vector. Classify and output a set of job status categories. The module also establishes a dynamic model to describe the coupling constraint relationship between mechanical state and driver's physiological state.

[0075] The risk assessment module is used to assess the risk based on the driver's physiological behavior vector. Calculate the fatigue index And according to the work environment state vector Calculate the environmental risk index ;

[0076] The work status comprehensive judgment module is used to determine the work status based on the probability distribution and the fatigue index. and the environmental risk index The final judgment job status is calculated and output through the discriminant function. ;

[0077] The system feedback and digital twin model update module is used to update the final judgment operation status. Feedback is fed into the digital twin virtual model, and online learning algorithms are used to dynamically adjust the model parameters.

[0078] The present invention will now be described using a specific embodiment.

[0079] I. System Hardware Configuration

[0080] The hardware components of this system include:

[0081] 1. Mechanical status sensor group: installed in key parts of agricultural machinery, including engine speed sensor, vehicle speed sensor, hydraulic pressure sensor, steering wheel angle sensor, throttle opening sensor, brake pressure sensor, implement height sensor, PTO (power take-off shaft) speed sensor, etc.

[0082] 2. Driver monitoring equipment:

[0083] Camera: Used to capture facial images of the driver and calculate... , , and head posture angle.

[0084] Physiological sensors: including heart rate monitors (for measuring...) ), Skin conductance sensor (measurement) ), body temperature sensor (measurement) ).

[0085] 3. Environmental sensing sensors:

[0086] Light sensor: measures ambient light intensity .

[0087] Temperature and humidity sensor: measures ambient temperature and humidity .

[0088] Tilt sensor: measures the tilt angle of agricultural machinery and calculates the terrain slope by combining it with GPS elevation data. .

[0089] Vibration sensor: Estimates surface roughness by analyzing the vibration spectrum characteristics of the vehicle body. .

[0090] Timer: Records the duration of continuous operation. .

[0091] 4. Data Processing Unit: Employs an embedded industrial computer, responsible for data preprocessing, feature extraction, and fusion calculation.

[0092] 5. Control and execution unit: including audible and visual alarm devices and agricultural machinery control system interface, used to execute early warning and intervention commands.

[0093] II. Software Algorithm Implementation

[0094] Step 1: Data Acquisition and Preprocessing

[0095] All sensor data are collected synchronously at a frequency of 10Hz.

[0096] The mechanical state vector for:

[0097]

[0098] In the formula, Engine speed, For driving speed, Hydraulic pressure, For steering wheel angle, For throttle opening, Braking pressure, For the height of agricultural machinery, This refers to the rotational speed of the power output shaft.

[0099] The driver's physiological behavior vector for:

[0100]

[0101] In the formula, The aspect ratio of the eyes, The length-to-width ratio of the mouth The percentage of eyelid closure. The head tilt angle, The yaw angle of the nose. For head roll angle, Heart rate, This is the skin conductance response. Body temperature.

[0102] The working environment state vector for:

[0103]

[0104] In the formula, Light intensity, For ambient temperature, For ambient humidity, For terrain slope, For surface roughness, This refers to the duration of continuous operation.

[0105] Normalize each indicator, and then uniformly normalize and map all indicators to... The interval is defined, and interpolation is performed based on the timestamp to ensure that all vectors are aligned at the same time point t:

[0106]

[0107]

[0108]

[0109] This is the normalized and aligned mechanical state vector. This is the normalized and aligned vector of the driver's physiological behavior. This is the normalized and aligned working environment state vector;

[0110] , These are the theoretical maximum and minimum values ​​of the mechanical state vector;

[0111] , These represent the theoretical maximum and minimum values ​​of the driver's physiological behavior vector;

[0112] , These represent the theoretical maximum and minimum values ​​of the working environment state vector.

[0113] Step 2, Feature Fusion

[0114] The fused feature vector Obtained through the following weighted multimodal fusion model:

[0115]

[0116] In the formula, For the mechanical state fusion weight vector, The driver's physiological behavior fusion weight vector, The weight vector for fusing the operational environment state satisfies: ;

[0117] This is the normalized and aligned mechanical state vector. This is the normalized and aligned vector of the driver's physiological behavior. This is the normalized and aligned working environment state vector. This indicates vector concatenation.

[0118] Step 3: Calculate the probability distribution

[0119] The probability distribution Obtained through the following model:

[0120]

[0121] In the formula, For activation function, For the first The weights of the layer network, For the first Bias of the layer network;

[0122] Probability distribution of each category in the output job status category set , Corresponding to categories The probability distribution, This represents the number of categories.

[0123] Step 4: State Classification

[0124] State classification is performed using a multilayer perceptron neural network (MLP), with the input layer having dimensions and... (Similar to the previous one), it also includes hidden layers and output layers; activation function Use ReLU.

[0125] use The function converts the output into a probability distribution of the job status category set. .

[0126] Define job status category set Each category represents a different meaning, for example This is a normal work category. It falls under the category of fatigued driving. This is a category of abnormal mechanical condition. This is a category for operations in hazardous environments.

[0127] Step 5: Risk Assessment

[0128] The fatigue index Calculated using the following model:

[0129]

[0130] In the formula, , , , , An indicator of the driver's physiological behavior vector;

[0131] These are the weighting coefficients for each indicator in the driver's physiological behavior vector. .

[0132] The environmental risk index Calculated using the following model:

[0133]

[0134] In the formula, , , , , These are the indicators of the operational environment state vector;

[0135] This represents the maximum value of the light intensity index. This represents the maximum value of the ambient temperature index. This represents the maximum value of the terrain slope index;

[0136] The weight coefficients of each indicator in the business environment state vector are: .

[0137] Step 6: Comprehensive Judgment

[0138] Calculate the overall score for each state and select the state with the highest score as the final state. The discriminant function is:

[0139]

[0140] In the formula, The state-related risk correction factor function is obtained by the following formula:

[0141]

[0142] In the formula, The state sensitivity coefficient, As the fatigue weighting factor, As an environmental weighting factor, it satisfies: .

[0143] Step 7, Feedback and Updates

[0144] Real-time feedback on current work status The digital twin virtual model is used to dynamically adjust model parameters based on actual work results and online learning algorithms. .

[0145] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A digital twin-based agricultural machinery driver operation status perception system, characterized in that, include: The data acquisition module is used to acquire mechanical state vectors. Driver physiological behavior vector and the working environment state vector ; The data processing module is used to perform data normalization and time synchronization alignment on the collected vectors. The feature fusion module is used to fuse the processed multimodal data using a weighted multimodal fusion model to obtain a fused feature vector. ; The job status modeling module uses a multilayer perceptron neural network to process the fused feature vector. Classify and output a set of job status categories. The module also establishes a dynamic model to describe the coupling constraint relationship between mechanical state and driver's physiological state. The risk assessment module is used to assess the risk based on the driver's physiological behavior vector. Calculate fatigue index And according to the work environment state vector Calculate the environmental risk index ; The work status comprehensive judgment module is used to determine the work status based on the probability distribution and the fatigue index. and the environmental risk index The final judgment job status is calculated and output through the discriminant function. ; The system feedback and digital twin model update module is used to update the final judgment operation status. Feedback is fed back to the digital twin virtual model, and online learning algorithms are used to dynamically adjust the model parameters.

2. The agricultural machinery driver operation status perception system based on digital twin according to claim 1, characterized in that, The mechanical state vector for: In the formula, Engine speed, For driving speed, Hydraulic pressure, For steering wheel angle, For throttle opening, Braking pressure, For the height of agricultural machinery, This refers to the rotational speed of the power output shaft.

3. The agricultural machinery driver operation status perception system based on digital twin according to claim 2, characterized in that, The driver's physiological behavior vector for: In the formula, The aspect ratio of the eyes, The length-to-width ratio of the mouth The percentage of eyelid closure. The head pitch angle, The yaw angle of the nose. For head roll angle, Heart rate, This is the skin conductance response. Body temperature.

4. The agricultural machinery driver operation status perception system based on digital twin according to claim 3, characterized in that, The working environment state vector for: In the formula, Light intensity, For ambient temperature, For ambient humidity, For terrain slope, For surface roughness, This refers to the duration of continuous operation.

5. The agricultural machinery driver operation status perception system based on digital twin according to claim 4, characterized in that, In the feature fusion module, the fused feature vector Obtained through the following weighted multimodal fusion model: In the formula, For the mechanical state fusion weight vector, The driver's physiological behavior fusion weight vector, The weight vector for fusing the operational environment state satisfies: ; This is the normalized and aligned mechanical state vector. This is the normalized and aligned vector of the driver's physiological behavior. This is the normalized and aligned working environment state vector. This indicates vector concatenation.

6. The agricultural machinery driver operation status perception system based on digital twin according to claim 5, characterized in that, In the job status modeling module, the probability distribution Obtained through the following model: In the formula, For activation function, For the first The weights of the layer network, For the first The bias of the layer network; Probability distribution of each category in the output job status category set , Corresponding to categories The probability distribution, This represents the number of categories.

7. The agricultural machinery driver operation status perception system based on digital twin according to claim 6, characterized in that, In the operation state modeling module, the dynamic model is: For mechanical state dynamics functions, This refers to mechanical state-related noise or random disturbance terms. Let be a function representing the changes in the driver's physiological state. This refers to physiological state-related noise or random disturbance terms.

8. The agricultural machinery driver operation status perception system based on digital twin according to claim 7, characterized in that, In the risk assessment module, the fatigue index Calculated using the following model: In the formula, , , , , An indicator of the driver's physiological behavior vector; These are the weighting coefficients for each indicator in the driver's physiological behavior vector. .

9. The agricultural machinery driver operation status perception system based on digital twin according to claim 8, characterized in that, In the risk assessment module, the environmental risk index Calculated using the following model: In the formula, , , , , These are the indicators of the operational environment state vector; This represents the maximum value of the light intensity index. This represents the maximum value of the ambient temperature index. This represents the maximum value of the terrain slope index. The weight coefficients of each indicator in the business environment state vector are: .

10. The agricultural machinery driver operation status perception system based on digital twin according to claim 9, characterized in that, In the comprehensive operation status discrimination module, the discrimination function is: In the formula, The state-related risk correction factor function is obtained by the following formula: In the formula, The state sensitivity coefficient, As the fatigue weighting factor, As an environmental weighting factor, it satisfies: .

Images