Multi-modal interaction intelligent HMI system for agricultural machinery based on LVGL

Abstract

The application relates to the field of intelligent control of agricultural machines, and discloses an agricultural machine multi-modal interaction intelligent HMI system based on LVGL. The system collects agricultural machine operation parameters, agricultural implement state parameters, cab environment parameters, operator state parameters, and touch, voice and gesture input data, identifies a current work scene, determines the modal reliability of each interaction mode, and determines a modal access set and a confirmation strategy according to the risk level of a control object, modal restriction rules and execution conditions. On this basis, the target control instruction is subjected to execution condition checking, and hierarchical display and refreshing scheduling are carried out on a safety critical layer, a task critical layer and an auxiliary information layer based on LVGL, and the scene label, the modal access set and the interface display content are updated in combination with execution feedback. The system can improve the man-machine interaction accuracy, control safety and interface response efficiency in an agricultural scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent control of agricultural machinery, and more specifically to an agricultural machinery multimodal interactive intelligent HMI system based on LVGL. Background Technology

[0002] During the intelligent operation of agricultural machinery, the human-machine interface in the cab gradually assumes functions such as setting operating parameters, displaying the status of agricultural implements, controlling the actuators, and providing alarm information. Especially on agricultural machinery equipped with power take-off shafts, hydraulic lifting mechanisms, and unfolding and folding mechanisms, operators need to frequently complete query, adjustment, and control operations through the HMI terminal.

[0003] Existing agricultural machinery HMIs mostly use touch interfaces combined with a few physical buttons for interaction, with some devices introducing voice or gesture input to improve ease of operation. These solutions typically recognize and respond to input types separately, or only control page switching and refresh based on interface display requirements, failing to adequately consider the collaborative relationships between the machinery's operating status, implement status, cab environment status, and operator status.

[0004] Because agricultural machinery operations generally involve high noise, strong vibration, significant changes in lighting, continuous occupation of the operator's hands, and high risks associated with the movement of agricultural implements, existing interactive structures may still use similar interactive access methods and interface display methods in different operating scenarios. This can lead to a mismatch between the applicability of the interactive modality and the current working conditions, resulting in high-risk controlled objects being prone to false triggering, false confirmation, or untimely display of key information under complex working conditions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an LVGL-based multimodal interactive intelligent HMI system for agricultural machinery, thereby resolving the technical problems existing in the prior art.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: The LVGL-based multimodal interactive intelligent HMI method for agricultural machinery includes the following steps: S1: Collect agricultural machinery operating parameters, implement status parameters, cab environment parameters, operator status parameters, and multimodal interactive input data, including touch input data, voice input data, and gesture input data; S2: Based on the agricultural machinery operating parameters, implement status parameters, cab environment parameters, and operator status parameters, identify the current operating scenario of the agricultural machinery and generate a scenario label corresponding to the current operating scenario; S3: Based on scene labels, cab environment parameters, and operator status parameters, determine the modal reliability of touch interaction mode, voice interaction mode, and gesture interaction mode in the current working scenario, and parse the target control command based on multimodal interaction input data; S4: Based on the type of the controlled object corresponding to the target control instruction and the current work scenario, determine the risk level corresponding to the target control instruction, and based on the modal reliability, risk level, and modal restriction rules corresponding to the current work scenario, determine the modal access set and confirmation strategy corresponding to the target control instruction; S5: When the interaction mode corresponding to the target control command belongs to the modality admission set, the execution condition of the target control command is verified; after the execution condition verification is passed, the target control command is confirmed according to the confirmation strategy and the control command is output to the corresponding agricultural machinery actuator; if the execution condition verification fails, the execution of the target control command is blocked and an alarm message is output. S6: Based on the LVGL interface display framework, according to scene tags, risk levels and current interface rendering load, the display priority allocation and refresh scheduling of the safety critical layer, task critical layer and auxiliary information layer in the HMI interface are performed. The safety critical layer and task critical layer are refreshed first, and the auxiliary information layer is delayed, partially refreshed or hidden. S7: Obtain the execution feedback results of the control commands, and update the scene labels, the modal admission set corresponding to the target control commands, and the content displayed on the HMI interface based on the execution feedback results.

[0007] Preferably, the scene tags include field straight-line operation scene, field turning scene, reversing attachment scene, farm implement unfolding scene, farm implement folding scene, power take-off shaft engagement scene, hydraulic lifting scene, obstacle avoidance scene, and parking maintenance scene.

[0008] Preferably, the agricultural machinery operating parameters include vehicle speed, gear, steering status, braking status, and obstacle warning status; The agricultural implement status parameters include power output shaft status, hydraulic actuation status, implement locking status, implement unfolding / folding status, and implement lifting / lowering status; The cab environmental parameters include noise intensity, vibration intensity, and screen glare intensity; The operator's status parameters include head orientation, gaze area, hand occupancy status, and operator fatigue status.

[0009] Preferably, determining the modal reliability of each interaction mode in step S3 specifically includes: Modal reliability of the voice interaction modality is determined based on noise intensity and head orientation; The modal reliability of the touch interaction mode is determined based on vibration intensity, screen glare intensity, and hand occupancy status. The modal reliability of gesture interaction modality is determined based on the area where the gaze rests, the state of hand occupancy, and the state of operational fatigue. The candidate interaction modes corresponding to the target control command are sorted according to the modal reliability of each interaction mode, and the candidate interaction modes with modal reliability lower than the preset reliability threshold are excluded from the modal admission set.

[0010] Preferably, in step S4, the risk level is divided into low risk level, medium risk level and high risk level according to the degree of impact of the controlled object on the safety of agricultural machinery operation and work. Page switching control and information query control correspond to low risk level, operation parameter adjustment control corresponds to medium risk level, and power output shaft engagement control, hydraulic actuation control, implement deployment control, implement folding control, and implement lifting control correspond to high risk level.

[0011] Preferably, determining the modal admission set and confirmation strategy corresponding to the target control command in step S4 specifically includes: For low-risk controlled objects, execution can be triggered directly by a single interaction modality; For objects under medium-risk control, execution is allowed to be triggered by a single interaction modality, and a cancellation window with a preset duration is opened after execution; For high-risk controlled objects, the first interaction mode is required to output the action intention, and the action can only be executed after confirmation by a second interaction mode different from the first interaction mode or by a physical confirmation document. When the current work scenario is identified as a turning scenario, a reversing device scenario, or an obstacle avoidance scenario, the triggering functions of the voice interaction mode and gesture interaction mode corresponding to the high-risk level control object are frozen, and only the confirmation function corresponding to the touch interaction mode or physical confirmation device is retained.

[0012] Preferably, the execution condition verification in step S5 includes verifying the vehicle speed, gear position, power output shaft status, hydraulic actuation status, implement locking status, implement unfolding / folding status, implement lifting status, and obstacle warning status. Control commands are output to the corresponding agricultural machinery actuators only when the results of each verification meet the execution conditions corresponding to the target control command; If any verification result does not meet the execution conditions corresponding to the target control command, the current state of the actuator remains unchanged, and the corresponding blocking reason information is output in the HMI interface.

[0013] Preferably, in step S6, the content displayed in the HMI interface includes a safety-critical layer, a task-critical layer, and an auxiliary information layer; The safety critical layer is used to display alarm information, confirmation information, and execution blocking information; The task key layer is used to display the current work scenario, current task parameters, and current farm implement status; The auxiliary information layer is used to display historical curves, statistical reports, and non-critical configuration pages; When the risk level is high or the current interface rendering load is determined to be high, the dynamic animation in the auxiliary information layer is frozen, the historical curve is refreshed with a delay, and only the alarm area, confirmation area and current task parameter area are partially refreshed.

[0014] Preferably, before parsing the target control command in step S3, the local voice command set corresponding to the current work scenario and the current attached farm implement type is invoked according to the scenario tag corresponding to the current work scenario and the current attached farm implement type; For voice input data that does not belong to the current local voice command set, no executable target control command is generated, and guidance prompt information is output; The guidance prompts are used to indicate the currently executable voice command type or to prompt switching to the touch interaction mode to perform the corresponding operation.

[0015] Preferably, in step S7, after obtaining the execution feedback result of the control command, for the medium-risk level control object, the pre-execution status information, post-execution status information, and cancellation control are displayed within a preset time period; For high-risk control targets, display the status information before execution, the status information after execution, and the execution condition verification results. Based on the execution feedback results and execution condition verification results, the scenario labels corresponding to the current operation scenario, the modal access set related to the controlled object, and the corresponding HMI interface display content are updated in a coordinated manner to ensure that the confirmation strategy of subsequent control commands matches the current agricultural machinery status.

[0016] In summary, the present invention has the following main beneficial effects: This application collects agricultural machinery operating parameters, implement status parameters, cab environmental parameters, and operator status parameters collaboratively. It also evaluates the reliability of touch interaction mode, voice interaction mode, and gesture interaction mode separately in conjunction with the current working scenario. This allows the system to no longer directly execute control based on a single input result, but to dynamically determine a more suitable interaction mode under different working conditions such as straight-line operation in the field, turning at the edge of the field, reversing to connect implements, and unfolding and folding implements. This reduces the impact of complex agricultural machinery scenarios such as high noise, strong vibration, strong glare, and operator hand occupation on the accuracy of interaction, reduces the probability of misidentification and false triggering, and improves the stability and applicability of human-machine interaction.

[0017] By classifying risk levels according to the degree of impact of the controlled object on the safety of agricultural machinery operation and work, and setting up modal access sets, dual confirmation mechanisms, and execution condition verification logic for high-risk controlled objects, high-risk actions such as power output shaft engagement control, hydraulic execution control, implement deployment control, implement folding control, and implement lifting control can only be executed when conditions such as vehicle speed, gear position, implement locking state, and obstacle warning state are met. This achieves the effect of transforming the general multimodal interaction scheme into a constrained control scheme for safe agricultural machinery operation, avoiding the direct triggering of dangerous actions under unsuitable working conditions, and improving the overall safety and control reliability of the machine operation.

[0018] By constructing a safety-critical layer, a task-critical layer, and an auxiliary information layer based on the LVGL interface display framework, and prioritizing and scheduling the refresh of different display content based on risk level and interface rendering load, the system prioritizes the display of alarm information, confirmation information, execution blocking information, current task parameters, and farm implement status during high-risk control processes or under high load conditions. At the same time, historical curves, statistical reports, and non-critical configuration pages are refreshed with delay or in a partial manner. This achieves the effects of ensuring the priority presentation of key interactive information, improving the response efficiency of embedded HMI terminals under complex agricultural machinery operating conditions, and enabling timely linkage and updates of execution feedback, scene switching, and subsequent control strategies. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1 refer to Figure 1 The LVGL-based multimodal interactive intelligent HMI method for agricultural machinery includes the following steps: S1: Collect agricultural machinery operating parameters, implement status parameters, cab environment parameters, operator status parameters, and multimodal interactive input data, including touch input data, voice input data, and gesture input data; S2: Based on the agricultural machinery operating parameters, implement status parameters, cab environment parameters, and operator status parameters, identify the current operating scenario of the agricultural machinery and generate a scenario label corresponding to the current operating scenario; S3: Based on scene labels, cab environment parameters, and operator status parameters, determine the modal reliability of touch interaction mode, voice interaction mode, and gesture interaction mode in the current working scenario, and parse the target control command based on multimodal interaction input data; S4: Based on the type of the controlled object corresponding to the target control instruction and the current work scenario, determine the risk level corresponding to the target control instruction, and based on the modal reliability, risk level, and modal restriction rules corresponding to the current work scenario, determine the modal access set and confirmation strategy corresponding to the target control instruction; S5: When the interaction mode corresponding to the target control command belongs to the modality admission set, the execution condition of the target control command is verified; after the execution condition verification is passed, the target control command is confirmed according to the confirmation strategy and the control command is output to the corresponding agricultural machinery actuator; if the execution condition verification fails, the execution of the target control command is blocked and an alarm message is output. S6: Based on the LVGL interface display framework, according to scene tags, risk levels and current interface rendering load, the display priority allocation and refresh scheduling of the safety critical layer, task critical layer and auxiliary information layer in the HMI interface are performed. The safety critical layer and task critical layer are refreshed first, and the auxiliary information layer is delayed, partially refreshed or hidden. S7: Obtain the execution feedback results of the control commands, and update the scene labels, the modal admission set corresponding to the target control commands, and the content displayed on the HMI interface based on the execution feedback results.

[0022] This method is applied to agricultural machinery with a power take-off shaft, hydraulic lifting mechanism, implement unfolding and folding mechanism, and cab display terminal. It is suitable for scenarios where tractors work in conjunction with implements such as seeders, fertilizer applicators, sprayers, and rotary tillers. The method is executed collaboratively by an HMI terminal, a vehicle controller, and an implement control unit. The HMI terminal is used to collect multimodal interactive input data, parse target control commands, complete interface display, and interactive confirmation. The vehicle controller is used to collect agricultural machinery operating parameters and complete execution condition verification. The implement control unit is used to collect implement status parameters and receive control output signals.

[0023] In this embodiment, the HMI terminal uses an embedded display device, which includes a touch screen, a voice acquisition unit, a gesture acquisition unit, an environmental perception unit, and a graphics processing unit. The graphics processing unit constructs the HMI interface based on the LVGL interface display framework. The vehicle controller communicates with the HMI terminal, the implement control unit, and each status acquisition unit via the vehicle communication bus. The implement control unit can be an independent controller or a functional module within the vehicle controller specifically for handling implement operation logic.

[0024] In this embodiment, the multimodal interactive input data includes touch input data, voice input data, and gesture input data. Touch input data is collected by a touchscreen display and includes at least control trigger information, touch location, and touch action type. Voice input data is collected by a microphone array installed in the driver's cab and includes at least voice segments and timestamp information. Gesture input data is collected by an image acquisition device or depth acquisition device installed in the driver's cab and includes at least gesture category information, gesture trigger time, and gesture location information.

[0025] In this embodiment, the agricultural machinery operating parameters include vehicle speed, gear position, steering status, braking status, and obstacle warning status; the implement status parameters include power take-off shaft status, hydraulic actuation status, implement locking status, implement unfolding / folding status, and implement lifting / lowering status; the cab environmental parameters include noise intensity, vibration intensity, and screen glare intensity; and the operator status parameters include head orientation, line-of-sight area, hand occupancy status, and operator fatigue status. All of the above parameters have clearly defined data collection sources.

[0026] The vehicle speed, gear position, steering status, and braking status are provided by the vehicle chassis control module, transmission control module, and braking detection module; the obstacle warning status is provided by the obstacle detection module, which can be implemented using any one or a combination of cameras, millimeter-wave radar, and ultrasonic radar. The power output shaft status is determined by the power output shaft switch status and power output shaft speed feedback; the hydraulic actuation status is determined by the hydraulic valve drive status and hydraulic circuit feedback status; the implement locking status is determined by the locking limit switch or locking position sensor; the implement unfolding / folding status and implement lifting / lowering status are determined by position feedback sensors, angle sensors, displacement sensors, or limit switches. Noise intensity is calculated from the ambient sound in the cab collected by the microphone array; vibration intensity is obtained by the inertial measurement unit installed in the cab or HMI terminal; screen glare intensity is determined by the ambient light sensor detection results combined with the screen surface reflection status. Head orientation, gaze area, hand occupancy status, and operator fatigue status are identified by the image acquisition device in the cab.

[0027] To facilitate accurate understanding by those skilled in the art, this embodiment provides the following explanations for some key terms. The "action intent" refers to the target action information to be confirmed, obtained from the first interaction modality parsing, but not yet entering the final control output stage. The "head orientation matching degree" refers to the degree of consistency between the operator's head orientation and the current effective voice acquisition direction. The "eye gaze dwell effectiveness" refers to the percentage of time the operator's gaze remains within the gesture recognition area or display interaction area within a preset statistical time window. The "hand occupancy degree" refers to the degree to which the operator's hands are in a steering wheel operation, joystick operation, or other continuously occupied state within a preset statistical time window. The "blocking reason information" refers to the prompt information generated by the unmet specific condition item when the execution condition verification fails. The "current task parameter area" refers to the display area in the task key layer used to display the main operation parameters in the current operation scenario.

[0028] The method in this embodiment includes the following steps.

[0029] Step S1 collects agricultural machinery operating parameters, implement status parameters, cab environment parameters, operator status parameters, and multimodal interactive input data.

[0030] Specifically, the vehicle controller and implement control unit periodically collect agricultural machinery operating parameters and implement status parameters according to a unified time base and send them to the HMI terminal. Cab environment parameters and operator status parameters are collected and cached locally by the HMI terminal. When the HMI terminal receives new multimodal interactive input data, it retrieves the agricultural machinery operating parameters, implement status parameters, cab environment parameters, and operator status parameters from the most recent valid time window, aligning the various types of input data in time. This ensures that the data used for subsequent scene recognition, modal reliability calculation, and execution condition verification belong to the same valid state period, avoiding errors caused by the cross-use of data from different sampling times.

[0031] Step S2 identifies the current operating scenario of the agricultural machinery based on the agricultural machinery operating parameters, implement status parameters, cab environment parameters, and operator status parameters, and generates a scenario label corresponding to the current operating scenario.

[0032] In this embodiment, the scene labels include field straight-line operation scenes, field turning scenes, reversing attachment scenes, implement unfolding scenes, implement folding scenes, power take-off shaft engagement scenes, hydraulic lifting scenes, obstacle avoidance scenes, and parking maintenance scenes. Scene labels are not simply used for interface display, but also establish correspondences with modal constraint rules, control condition tables, and local voice command sets.

[0033] In specific identification processes, rule fusion should be prioritized.

[0034] When the vehicle speed is continuously moving forward, the steering change is less than the preset steering judgment range, and the agricultural implement is in continuous operation, it is identified as a straight-line operation scenario in the field.

[0035] When a vehicle approaches the boundary of a plot of land, its turning state changes continuously, and the lifting or preparation state of agricultural implements switches, it is identified as a turning scene at the edge of the field.

[0036] When the gear is in reverse or the vehicle is in reverse, and the implement is not locked or the attachment operation is in progress, it is identified as a reversing attachment scenario.

[0037] When a control intention to unfold farm implements is received, and the folded state of the farm implements changes from the folded position to the unfolded position, it is identified as a farm implement unfolding scenario.

[0038] When a control intention to fold farm implements is received, and the farm implement changes from the unfolded position to the folded position, it is identified as a farm implement folding scenario.

[0039] When a power output shaft engagement control intention is received or the power output shaft state is switched from disconnected to engaged, it is identified as a power output shaft engagement scenario.

[0040] When a hydraulic lifting control intention is received or the lifting status of agricultural implements changes, it is identified as a hydraulic lifting scenario.

[0041] When the obstacle warning status is in an alarm state or a temporary avoidance action is required, it is identified as an obstacle avoidance scenario.

[0042] When the vehicle is parked, the power output shaft is disconnected, the hydraulic actuator is inactive, and the operator enters the maintenance interface or performs an inspection operation, it is identified as a parking maintenance scenario.

[0043] In this embodiment, the scene labels and modal restriction rules are stored in a rule table using a one-to-one or one-to-many mapping relationship. The rule table can be pre-installed in the HMI terminal or vehicle controller and configured according to different agricultural implement operation processes during the prototype debugging phase.

[0044] Step S3 determines the modal reliability of touch interaction mode, voice interaction mode and gesture interaction mode in the current working scenario based on the scene label, cab environment parameters and operator state parameters, and parses the target control command based on the multimodal interaction input data.

[0045] To provide a clear basis for modal reliability assessment, this embodiment establishes scoring models for voice interaction, touch interaction, and gesture interaction modalities, respectively. The calculation formulas are as follows: ; ; ; In the formula, Indicates the modal reliability of the voice interaction mode; Indicates the modal reliability of touch interaction modes; Indicates the modal reliability of gesture interaction modes; This represents the normalized noise intensity. This represents the normalized head orientation matching degree; This represents the normalized vibration intensity; This represents the normalized screen glare intensity. This indicates the normalized degree of hand occupancy. This indicates the normalized effectiveness of gaze persistence. This indicates the normalized level of operational fatigue. , , , , , , , These are the weighting coefficients, and the sum of the weighting coefficients in the same formula is 1.

[0046] The upper and lower limits of each normalized quantity are derived from real test data that can be collected during prototype debugging and system integration testing. Specifically, the accuracy rates of voice recognition, touch false triggering, and gesture recognition can be tested under different noise levels, vibration levels, screen glare levels, hand occupancy states, and fatigue states. The effective upper and lower limits of each normalized quantity are then determined based on the test results. The weighting coefficients are determined by the degree of influence of each factor in the above test results on the interaction success rate and false triggering rate, and are stored as calibration parameters in the HMI terminal or vehicle controller. Therefore, the modal reliability in this embodiment is not arbitrarily set, but is generated based on the real logic of system integration testing and prototype testing.

[0047] After obtaining the modal reliability of each interaction mode, this embodiment further calculates the modal admission score based on the current work scenario and risk level, using the following formula: ; In the formula, Indicates the first Modal admission scores corresponding to each interaction modality; Indicates the first Modal reliability corresponding to each interaction mode; Indicates the first [number]th [unit] in the current work scenario Scene access factors for each interaction modality; Indicates the current risk level. Risk adaptation factor for each interaction modality; , , These are the weighting coefficients, and the sum of the three is 1.

[0048] Among them, the scene admission factor The scene admission factor is determined by looking up the modal restriction rules corresponding to the current work scenario. For interactive modalities that are allowed to be used directly, the scene admission factor takes a first preset value; for interactive modalities that are only allowed for confirmation and not for triggering, the scene admission factor takes a second preset value; for interactive modalities that are prohibited from being triggered in the current scenario, the scene admission factor takes zero.

[0049] The risk adaptation factor The risk adaptation factor is determined by looking up the table based on the risk level corresponding to the target control instruction. For low-risk control objects that allow direct execution via a single interaction mode, the risk adaptation factor is set to a high value; for medium-risk control objects that allow execution via a single interaction mode but retain an undo window, the risk adaptation factor is set to a medium value; for high-risk control objects that require double confirmation, the risk adaptation factor corresponding to direct execution via a single interaction mode is set to a low value or zero.

[0050] In this embodiment, both the preset reliability threshold and the modal admission threshold are derived from multi-scenario interaction test results collected during the prototype integration phase. These multi-scenario interaction test results include at least the interaction recognition accuracy, false trigger rate, and missed recognition rate under different noise intensities, vibration intensities, screen glare intensities, and hand occupancy states. Based on these test results, the controller determines the minimum acceptable reliability threshold for each interaction modality, ensuring that the interaction modalities included in the modal admission set meet the preset recognition accuracy requirements and that the false trigger rate is controlled within an allowable range. Candidate interaction modalities with a reliability threshold below the preset reliability threshold are directly excluded from the modal admission set; similarly, candidate interaction modalities with a modal admission score below the modal admission threshold are not included in the modal admission set corresponding to the current target control command.

[0051] After completing modal reliability calculations and modal admission set selection, the HMI terminal parses the target control commands based on multimodal interactive input data. For touch input data, the type of controlled object and the direction of the target action are determined based on the page control identifier, control position, and control trigger action. For voice input data, speech-to-text transcription is performed first, followed by semantic mapping based on the current work scenario and the type of currently attached implement. For gesture input data, gesture category recognition is completed first, and the recognition result is mapped to a trigger action, confirmation action, or cancellation action. The parsing results include at least the type of controlled object, the direction of the target action, the source of the interaction modality, and timestamp information.

[0052] Step S4 determines the risk level corresponding to the target control instruction based on the control object type corresponding to the target control instruction and the current operation scenario, and determines the modal access set and confirmation strategy corresponding to the target control instruction based on the modal reliability of each interaction modality, the risk level, and the modal restriction rules corresponding to the current operation scenario.

[0053] In this embodiment, the risk level is divided into low-risk, medium-risk, and high-risk levels according to the degree of impact of the controlled object on the operational safety of agricultural machinery. Page switching control and information query control correspond to the low-risk level, operation parameter adjustment control corresponds to the medium-risk level, and power output shaft engagement control, hydraulic actuation control, implement deployment control, implement folding control, and implement lifting control correspond to the high-risk level.

[0054] When determining the confirmation strategy, for low-risk control objects, a single interaction modality is allowed to directly trigger execution. For medium-risk control objects, a single interaction modality is allowed to trigger execution, and a cancellation window of a preset duration will be opened after execution. For high-risk control objects, the first interaction modality is required to output the action intent first, and then it must be confirmed by a second interaction modality different from the first interaction modality, or by a physical confirmation device, before entering the control output process.

[0055] The preset duration is determined based on the execution completion time of the corresponding controlled object, the status feedback stabilization time, and the operator's normal confirmation reaction time. It can be configured during the machine calibration stage according to different implement types and different work objects. In other words, the cancellation window for medium-risk level controlled objects is not arbitrarily set, but is based on the actual execution completion and status stabilization time required, combined with the operator's actual reaction time to complete the confirmation or cancellation operation under normal operating conditions.

[0056] In this embodiment, when the current work scenario is identified as a turning point, a reversing maneuver, or an obstacle avoidance scenario, the triggering functions of the voice and gesture interaction modes corresponding to the high-risk control object are frozen, and only the confirmation function corresponding to the touch interaction mode or physical confirmation device is retained. In other words, under the aforementioned high-safety-pressure scenarios, even if the voice or gesture interaction mode recognizes the action intention corresponding to a high-risk control object, it is not allowed to directly enter the execution process as a trigger mode; it can only be used for prompting or confirmation. In this way, this embodiment enables scene tags not only to participate in display switching but also to directly determine the interaction access boundaries of high-risk control objects in different scenarios.

[0057] Step S5: When the interaction mode corresponding to the target control command belongs to the mode admission set, the execution condition of the target control command is verified; after the execution condition verification is passed, the target control command is confirmed according to the confirmation strategy, and the control command is output to the corresponding agricultural machinery actuator; when the execution condition verification fails, the execution of the target control command is blocked and an alarm message is output.

[0058] In this embodiment, the execution condition verification includes verifying vehicle speed, gear position, power output shaft status, hydraulic actuation status, implement locking status, implement unfolding / folding status, implement lifting / lowering status, and obstacle warning status. Different control objects correspond to different execution condition tables. The execution condition table consists of object type, target action, required status items, and prohibited status items, and is determined during the whole machine calibration stage based on the implement instruction manual, vehicle control requirements, and prototype debugging results.

[0059] For example, for power take-off shaft engagement control, at least it should be verified whether the current gear is in an engagement-allowed state, whether the obstacle warning state is in a non-alarm state, whether the implement locking state is completed, and whether the current state of the power take-off shaft meets the switching conditions.

[0060] For implement deployment control, at least the following should be checked: whether the vehicle speed is within the allowable deployment range, whether the implement locking state meets the pre-deployment conditions, whether the implement unfolding / folding state allows switching from the folded position to the unfolded position, and whether the obstacle warning state is in a non-alarm state.

[0061] For implement folding control, at least the following should be verified: whether the vehicle speed is within the allowable folding range, whether the hydraulic actuation state can be switched, whether the implement unfolding and folding state is within the allowable folding state, and whether the obstacle warning state is in a non-alarm state.

[0062] For implement lifting control, at least verify whether the hydraulic actuator is allowed to operate, whether the implement is in a lifting and lowering state when unfolded or folded, and whether the obstacle warning state is in a non-alarm state.

[0063] When any verification result fails to meet the execution conditions of the corresponding controlled object, the system maintains the current state of the actuator and outputs the blocking reason information in the HMI interface. The blocking reason information directly corresponds to the specific unmet condition, such as the vehicle speed not meeting the deployment condition, the implement locking not being completed, or the presence of an obstacle warning. This blocking reason information serves both as a safety reminder and is used for subsequent execution feedback updates and interface linkage display.

[0064] Step S6, based on the LVGL interface display framework, allocates display priority and schedules refresh for the safety critical layer, task critical layer, and auxiliary information layer in the HMI interface according to the scene label, the risk level, and the current interface rendering load. Prioritizes refreshing the safety critical layer and the task critical layer, and performs delayed refresh, partial refresh, or hiding processing on the auxiliary information layer.

[0065] In this embodiment, the safety-critical layer is used to display alarm information, confirmation information, and execution blocking information, and has the highest priority; the task-critical layer is used to display the current operation scenario, current task parameters, and current farm implement status; and the auxiliary information layer is used to display historical curves, statistical reports, and non-critical configuration pages.

[0066] To determine whether the current UI rendering load is under high load, this embodiment uses the UI rendering load index for evaluation, and the calculation formula is as follows: ; In the formula, This indicates the UI rendering load index; This represents the normalized display-related processor utilization. This indicates the percentage of the interface redrawn area after normalization. This represents the normalized frame refresh latency. , , These are the weighting coefficients, and the sum of the three is 1.

[0067] The display-related processor utilization, interface redraw area ratio, and frame refresh latency can all be obtained by monitoring during HMI terminal operation. The preset load judgment threshold is set based on the HMI terminal's processor performance, display resolution, target refresh cycle, and interface response test results collected during the prototype debugging phase. When the interface rendering load index exceeds the preset load judgment threshold for multiple consecutive refresh cycles, the current interface rendering load is determined to be in a high-load state.

[0068] When the risk level is high or the current interface rendering load is determined to be high, the system freezes the dynamic animation in the auxiliary information layer, delays the refresh of historical curves, and only performs partial refreshes on the alarm area, confirmation area, and current task parameter area. In LVGL implementation, the safety-critical layer, task-critical layer, and auxiliary information layer can be mapped to different display containers, and the safety-critical layer and task-critical layer are prioritized for display using partial redrawing and dirty area refresh mechanisms. In this way, LVGL in this embodiment is no longer a general display framework, but rather undertakes the specific implementation function of prioritizing the fidelity of safety information in high-risk scenarios.

[0069] Step S3 corresponds to the invocation of the local voice command set in claim 9: Before parsing the voice input data, the local voice command set corresponding to the current work scenario and the type of the currently attached farm implement is invoked based on the scenario label and the type of the currently attached farm implement.

[0070] In this embodiment, the type of currently attached agricultural implement is determined by the implement identification code, bus configuration parameters, or manual confirmation. The local voice command set is pre-established and stored in the HMI terminal or vehicle controller according to the implement type and the operation scenario. After identifying the currently attached agricultural implement type and the current operation scenario, the corresponding local voice command set is invoked for semantic mapping.

[0071] For example, when the currently attached agricultural implement type is a seeder and the scene label is a straight-line field operation scene, the local voice command set mainly retains commands for seed quantity query, section status query, and fine-tuning of operation parameters; when the scene label is switched to a field turning scene, the system actively shrinks the local voice command set, retaining only query commands, low-risk confirmation commands, and prompt commands, and does not retain voice commands that directly trigger high-risk control objects.

[0072] For example, when the currently attached agricultural implement is a sprayer, the local voice command set is switched to voice command categories adapted to this agricultural implement, such as spray volume query, spray volume fine-tuning, and working status query.

[0073] For voice input data that does not belong to the current local voice command set, the system does not generate executable target control commands, but instead outputs guidance prompts. These prompts indicate the type of currently executable voice command or suggest switching to the touch interaction mode to execute the corresponding operation. In this way, this embodiment limits the executable semantic space of the voice command to the range allowed by the current tool type and the current work scenario, reducing the risk of mis-identification and incorrect execution from the source.

[0074] Step S7: Obtain the execution feedback result of the control command, and update the scene label, the modal admission set corresponding to the target control command, and the HMI interface display content according to the execution feedback result.

[0075] In this embodiment, after obtaining the execution feedback result of the control command, for medium-risk level control objects, the pre-execution status information, post-execution status information, and cancellation control are displayed within a preset time period. The cancellation control is used to trigger a recovery control command to restore the state to the pre-execution state within the preset time period.

[0076] For high-risk controlled objects, the system displays pre-execution status information, post-execution status information, and execution condition verification result information. The execution condition verification result information includes at least the security conditions that have been met and the specific conditions that were used to block or pass the verification.

[0077] Meanwhile, based on the execution feedback results and execution condition verification results, the system updates the scene label corresponding to the current operation scenario, the modal access set related to the controlled object, and the corresponding HMI interface display content in a coordinated manner, so that the confirmation strategy of subsequent control commands matches the current agricultural machinery status.

[0078] For example, after the farm implement unfolding action is completed, if the unfolded and folded state of the farm implement has stabilized in the unfolded position, the system will switch the scene label from farm implement unfolding scene to field straight-line operation scene or new stable operation scene, and simultaneously close the confirmation interface related to the unfolding action, retaining only the current task parameters and farm implement status display.

[0079] For example, after the power output shaft engagement action is completed, if the power output shaft state has stabilized in the engagement state, the high-risk trigger entry related to the power output shaft engagement action is switched to the disconnection control entry, and the corresponding modal admission set and confirmation strategy are updated.

[0080] When a high-risk control object is blocked due to the failure to meet the execution conditions, the system can keep the corresponding high-risk trigger entry in a restricted state for a short period of time while retaining the display of the blocking reason area, until the relevant status quantity is restored to the allowed state.

[0081] Through the above methods, this embodiment forms a complete closed loop of parameter acquisition, scenario identification, modal reliability calculation, modal admission and confirmation strategy determination, execution condition verification, control output, hierarchical display, execution feedback, and linkage update.

[0082] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal interactive intelligent HMI method for agricultural machinery based on LVGL, characterized in that, Includes the following steps: S1: Collect agricultural machinery operating parameters, implement status parameters, cab environment parameters, operator status parameters, and multimodal interactive input data, including touch input data, voice input data, and gesture input data; S2: Based on the agricultural machinery operating parameters, implement status parameters, cab environment parameters, and operator status parameters, identify the current operating scenario of the agricultural machinery and generate a scenario label corresponding to the current operating scenario; S3: Based on scene labels, cab environment parameters, and operator status parameters, determine the modal reliability of touch interaction mode, voice interaction mode, and gesture interaction mode in the current working scenario, and parse the target control command based on multimodal interaction input data; S4: Based on the type of the controlled object corresponding to the target control instruction and the current work scenario, determine the risk level corresponding to the target control instruction, and based on the modal reliability, risk level, and modal restriction rules corresponding to the current work scenario, determine the modal access set and confirmation strategy corresponding to the target control instruction; S5: When the interaction mode corresponding to the target control command belongs to the modality admission set, the execution condition of the target control command is verified; after the execution condition verification is passed, the target control command is confirmed according to the confirmation strategy and the control command is output to the corresponding agricultural machinery actuator; if the execution condition verification fails, the execution of the target control command is blocked and an alarm message is output. S6: Based on the LVGL interface display framework, according to scene tags, risk levels and current interface rendering load, the display priority allocation and refresh scheduling of the safety critical layer, task critical layer and auxiliary information layer in the HMI interface are performed. The safety critical layer and task critical layer are refreshed first, and the auxiliary information layer is delayed, partially refreshed or hidden. S7: Obtain the execution feedback results of the control commands, and update the scene labels, the modal admission set corresponding to the target control commands, and the content displayed on the HMI interface based on the execution feedback results.

2. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 1, characterized in that, The scene tags include straight-line operation scene in the field, turning scene at the edge of the field, reversing attachment scene, unfolding scene of farm implements, folding scene of farm implements, power take-off shaft engagement scene, hydraulic lifting scene, obstacle avoidance scene, and parking maintenance scene.

3. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 2, characterized in that, The agricultural machinery operating parameters include vehicle speed, gear, steering status, braking status, and obstacle warning status; The agricultural implement status parameters include power output shaft status, hydraulic actuation status, implement locking status, implement unfolding / folding status, and implement lifting / lowering status; The cab environmental parameters include noise intensity, vibration intensity, and screen glare intensity; The operator's status parameters include head orientation, gaze area, hand occupancy status, and operator fatigue status.

4. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 3, characterized in that, Determining the modal reliability of each interaction mode in step S3 specifically includes: Modal reliability of the voice interaction modality is determined based on noise intensity and head orientation; The modal reliability of the touch interaction mode is determined based on vibration intensity, screen glare intensity, and hand occupancy status. The modal reliability of gesture interaction modality is determined based on the area where the gaze rests, the state of hand occupancy, and the state of operational fatigue. The candidate interaction modes corresponding to the target control command are sorted according to the modal reliability of each interaction mode, and the candidate interaction modes with modal reliability lower than the preset reliability threshold are excluded from the modal admission set.

5. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 4, characterized in that, In step S4, the risk level is divided into low risk level, medium risk level and high risk level according to the degree of impact of the controlled object on the safety of agricultural machinery operation and work. Page switching control and information query control correspond to low risk level, operation parameter adjustment control corresponds to medium risk level, and power output shaft engagement control, hydraulic actuation control, implement deployment control, implement folding control, and implement lifting control correspond to high risk level.

6. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 5, characterized in that, The step S4, which determines the modal admission set and confirmation strategy corresponding to the target control command, specifically includes: For low-risk controlled objects, execution can be triggered directly by a single interaction modality; For objects under medium-risk control, execution is allowed to be triggered by a single interaction modality, and a cancellation window with a preset duration is opened after execution; For high-risk controlled objects, the first interaction mode is required to output the action intention, and the action can only be executed after confirmation by a second interaction mode different from the first interaction mode or by a physical confirmation document. When the current work scenario is identified as a turning scenario, a reversing device scenario, or an obstacle avoidance scenario, the triggering functions of the voice interaction mode and gesture interaction mode corresponding to the high-risk level control object are frozen, and only the confirmation function corresponding to the touch interaction mode or physical confirmation device is retained.

7. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 6, characterized in that, The execution condition verification in step S5 includes verifying the vehicle speed, gear position, power output shaft status, hydraulic actuation status, implement locking status, implement unfolding / folding status, implement lifting status, and obstacle warning status. Control commands are output to the corresponding agricultural machinery actuators only when the results of each verification meet the execution conditions corresponding to the target control command; If any verification result does not meet the execution conditions corresponding to the target control command, the current state of the actuator remains unchanged, and the corresponding blocking reason information is output in the HMI interface.

8. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 7, characterized in that, In step S6, the content displayed in the HMI interface includes a safety-critical layer, a task-critical layer, and an auxiliary information layer. The safety critical layer is used to display alarm information, confirmation information, and execution blocking information; The task key layer is used to display the current work scenario, current task parameters, and current farm implement status; The auxiliary information layer is used to display historical curves, statistical reports, and non-critical configuration pages; When the risk level is high or the current interface rendering load is determined to be high, the dynamic animation in the auxiliary information layer is frozen, the historical curve is refreshed with a delay, and only the alarm area, confirmation area and current task parameter area are partially refreshed.

9. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 8, characterized in that, Before parsing the target control command in step S3, the local voice command set corresponding to the current operation scenario and the current attached farm implement type is called according to the scenario label corresponding to the current operation scenario and the current attached farm implement type. For voice input data that does not belong to the current local voice command set, no executable target control command is generated, and guidance prompt information is output; The guidance prompts are used to indicate the currently executable voice command type or to prompt switching to the touch interaction mode to perform the corresponding operation.

10. The LVGL-based multimodal interactive intelligent HMI system for agricultural machinery according to claim 9, characterized in that, In step S7, after obtaining the execution feedback result of the control command, for the medium-risk level control object, the pre-execution status information, post-execution status information and cancellation control are displayed within a preset time period. For high-risk control targets, display the status information before execution, the status information after execution, and the execution condition verification results. Based on the execution feedback results and execution condition verification results, the scenario labels corresponding to the current operation scenario, the modal access set related to the controlled object, and the corresponding HMI interface display content are updated in a coordinated manner to ensure that the confirmation strategy of subsequent control commands matches the current agricultural machinery status.

Images