Construction vehicle-mounted intelligent display screen system based on multi-modal interaction and control method

Abstract

This application provides a construction vehicle-mounted intelligent display system and control method based on multimodal interaction, belonging to the field of intelligent control technology for engineering vehicles. The system includes an onboard intelligent controller and connected to it a vehicle bus interface, a driver monitoring camera, a gesture recognition sensor, a touch display screen, a physical operation module, a distributed tactile warning device, and an environmental perception sensor. The vehicle bus interface acquires vehicle status data, the driver monitoring camera collects head posture and gaze direction, the gesture recognition sensor collects gesture operations, and the environmental perception sensor collects surrounding environmental data. The onboard intelligent controller assesses the safety status level in real time based on the above data, dynamically allocates the interaction priority and feedback intensity of each module, and dynamically adjusts the information layout of the display screen. This application improves the safety and operational efficiency of human-computer interaction in construction scenarios through multimodal fusion and dynamic priority allocation.

Description

Technical Field

[0001] This application belongs to the field of intelligent control technology for engineering vehicles, specifically relating to a construction vehicle-mounted intelligent display screen system and control method based on multimodal interaction. Background Technology

[0002] Currently, the driving environment for construction vehicles (such as bulldozers, excavators, and road rollers) is becoming increasingly complex, and in-vehicle information terminals have evolved from simple instrument panels to intelligent displays that integrate navigation, surround view, vehicle information, construction coordination, and other functions. However, existing in-vehicle display systems still have the following shortcomings: First, the interaction methods are simplistic, posing safety hazards. Most in-vehicle systems still rely primarily on touchscreens for interaction. In the harsh environment of construction vehicles, characterized by high vibration and dust, touchscreen operation suffers from low precision and is prone to accidental touches. More importantly, while working, drivers' hands are typically occupied by the steering wheel and joysticks, requiring their eyes to constantly focus on the work surface. Frequent screen touches severely distract attention, creating hand-eye conflict and increasing operational risks. Second, information presentation is rigid, leading to high cognitive load. Existing systems typically display information in a fixed layout, unable to dynamically adjust based on vehicle status (driving, working, parked) and environmental risks (obstacles, blind spots). In complex working conditions, a large amount of irrelevant information is presented simultaneously, requiring drivers to sift through it to extract key information, resulting in a high cognitive load and reduced emergency response speed. Finally, there is a lack of proactive, tiered warning mechanisms. Existing visual or audible warning methods have limitations. Visual warnings require the driver to look at the screen, while audible warnings are easily masked in noisy construction environments with engine roars and mechanical collisions, failing to create a reliable and mandatory alarm channel. There is a lack of mechanisms that can proactively allocate different sensory channels for tiered warnings based on risk levels.

[0003] Therefore, there is an urgent need to design an in-vehicle intelligent system that can proactively adapt to construction scenarios, provide efficient and safe interaction methods, and achieve hierarchical and multi-channel early warning. Summary of the Invention

[0004] In a first aspect, embodiments of this application provide a construction vehicle-mounted intelligent display system based on multimodal interaction, including an on-board intelligent controller; The vehicle-mounted intelligent controller is connected to a vehicle bus interface, a driver monitoring camera, a gesture recognition sensor, a touch screen, a physical operation module, a distributed tactile warning device, and at least one environmental perception sensor. Vehicle bus interface, used to acquire vehicle status data; Driver monitoring cameras are used to collect data on the driver's head posture and line of sight. Gesture recognition sensors are used to collect the driver's gesture operation data; A touch display screen is used to provide a visual interactive interface and receive touch input; The physical operation module is used to receive physical button or knob operations from the driver; A distributed tactile warning device is used to provide tactile vibration feedback to the driver; Environmental perception sensors are used to collect data on the environment surrounding the vehicle. The vehicle-mounted intelligent controller is configured as follows: Based on the vehicle status data, environmental perception data, driver head posture and gaze direction data, and gesture operation data, the safety situation level of the current driving scenario is assessed in real time. Based on the safety situation level, the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device are dynamically allocated, and the information layout and content of the touch screen are dynamically adjusted.

[0005] Furthermore, the vehicle bus interface adopts a CAN bus interface or an Ethernet interface to connect the vehicle chassis controller, engine controller, and working device controller. The vehicle status data includes vehicle speed data, engine speed, engine load data, and working device status data. The vehicle-mounted intelligent controller obtains vehicle speed data from the vehicle chassis controller, engine speed data and engine load data from the engine controller, and working device status data from the working device controller through the vehicle bus interface.

[0006] Furthermore, the environmental perception sensors include 360° surround-view cameras and radar; A 360° surround-view camera is used to capture panoramic images of the vehicle's exterior. The radar uses millimeter-wave radar or lidar to detect the distance and relative speed of obstacles.

[0007] Furthermore, the driver monitoring camera uses an infrared camera or a 3D ToF camera and is installed on the A-pillar of the cab or the front of the roof; The gesture recognition sensor is a ToF camera or an infrared depth sensor, installed on the ceiling or above the dashboard in the driver's cab, to recognize the driver's preset gesture commands in the air.

[0008] Furthermore, the physical operation module includes a multi-function knob and several shortcut keys; The multi-function knob and various shortcut keys are integrated and set on the top of the steering wheel spokes or control lever; A multi-function knob for menu navigation and parameter adjustment; Keyboard shortcuts are used to switch display modes or confirm task status with a single keystroke. The distributed tactile warning device includes a linear resonant motor and several eccentric rotary vibration motors; The linear resonant motor is integrated below the touch screen to provide vibration feedback for operation confirmation; Each eccentric rotary vibration motor is integrated into the steering wheel, seat, and seat belt to generate multi-level tactile warnings based on different vibration intensities, frequencies, and position signals sent by the vehicle's intelligent controller.

[0009] Furthermore, it also includes a communication module; The communication module is a 4G / 5G cellular communication module or a Wi-Fi module, used to receive construction task information from the cloud dispatch center or local area network server; The vehicle-mounted intelligent controller is also configured as follows: The system receives construction task information via a communication module, controls the touchscreen display to show the information, and provides an interactive interface for one-click confirmation of task status.

[0010] Secondly, embodiments of this application also provide a control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction, comprising the following steps: S1. Acquire vehicle status data, environmental perception data, driver head posture and gaze direction data, gesture operation data, and construction task information; S2. Assess the safety status level of the current driving scenario based on vehicle status data, environmental perception data, driver head posture and line of sight data, and gesture operation data; S3. Based on the security status level, dynamically allocate the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device; S4. Based on the assigned priority and feedback intensity, control the touch screen, physical operation module, gesture recognition sensor and distributed tactile early warning device to perform corresponding interactive feedback, and dynamically adjust the information layout and content of the touch screen based on the security status level and construction task information.

[0011] Furthermore, the specific steps of step S2 are as follows: S21. Extract vehicle speed from the acquired vehicle status data. Calculate the vehicle operation risk coefficient based on engine load and working device status. ; S22. Extract the relative distance between the target object and the vehicle from the acquired environmental perception data. and relative velocity The obstacle approach risk coefficient is calculated using the following formula. :

[0012] in, , These are calibration coefficients; S23. Obtain the driver's gaze direction through the driver monitoring camera and determine the continuous duration of the gaze deviating from the work surface. The operational deviation risk coefficient is calculated using the following piecewise function. : ; S24. Risk factor of approaching obstacles Vehicle operation risk coefficient Operational deviation risk coefficient Substitute the values ​​into the following weighted summation formula to calculate the comprehensive security situation index. :

[0013] in, , , The weighting factors are preset and greater than 0, satisfying α+β+γ=1; based on the comprehensive security situation index... The numerical range in which the value is located determines the corresponding security status level.

[0014] Furthermore, the specific steps of step S3 are as follows: S31. A rule table containing the mapping relationship between security status levels and each interaction channel is preset, and the rule table defines at least the following mappings: S311. Output channel priority, wherein the output channels include visual feedback from the touch display screen and haptic feedback from the distributed haptic warning device: When the security situation level is low risk, tactile confirmation feedback takes precedence over visual information display; the tactile confirmation feedback is provided by an operation confirmation vibration from a linear resonant motor below the touch screen; the current display mode is maintained; When the safety situation level is medium risk, visual warnings take precedence over tactile warnings; the visual warnings are partial highlighting on the touch screen or pop-up prompts, and the tactile warnings are a single moderate-intensity vibration from an eccentric rotary vibration motor in the steering wheel or seat; the display mode is automatically switched, and the warning information is highlighted. When the security situation level is high risk, tactile forced warning takes precedence over visual forced warning; the tactile forced warning is that the eccentric rotary vibration motor in the steering wheel and seat continuously exceeds the preset intensity threshold and vibrates at a preset frequency, while the visual forced warning is that the touch screen flashes in full and becomes inoperable, forcibly displaying the risk source and guidance information; and forcibly switching to high-risk warning mode; S312. Input channel response strategy, where the input channel is the physical operation module and the gesture recognition sensor: When the safety situation level is low or medium risk, the physical operation module and gesture recognition sensor respond normally, and the driver can browse the menu, adjust parameters and confirm tasks through the multi-function knob or button. When the security situation level is high risk, non-security-related operations in the entity operation module are temporarily suppressed, only preset security confirmation shortcut keys are allowed to take effect, and the gesture recognition function is turned off. S32. Using the calculated security situation level as an index, query the rule table to obtain the optimal interaction channel allocation strategy for the current moment. The optimal interaction channel allocation strategy includes the priority order of output channels, the feedback intensity coefficient of each output channel, display control parameters, haptic feedback control parameters, and the operation response priority of input channels; wherein, the feedback intensity coefficient is determined according to the security situation level according to a preset mapping table. S33. Send the display control parameters in the optimal interaction channel allocation strategy to the touch screen, send the haptic feedback control parameters to the distributed haptic warning device, and configure the operation response priority to the physical operation module.

[0015] Furthermore, the specific steps of step S4 are as follows: S41. Based on the safety status level, construction task information, and the display control parameters issued in step S3, a target display mode matching the display control parameters is called from the preset display modes. The display modes include driving mode, operation mode, and early warning mode to determine the target display mode. S42. Control the touch screen to switch to the target display mode, dynamically adjust the information layout and content, display construction task information and provide an interactive interface for one-click confirmation of task status; S43. When the driver inputs a command through a physical operation module or a valid gesture, the vibration feedback intensity of the linear resonant motor below the touch screen is triggered to provide a highly responsive operation feedback. S44. When the safety situation level assessed in step S2 increases, interrupt the currently ongoing non-safety-related tactile interaction and call the eccentric rotary vibration motor in the distributed tactile early warning device to generate a multi-point vibration early warning of the corresponding level according to the intensity coefficient in the tactile feedback control parameters.

[0016] As can be seen from the above technical solutions, this application has the following advantages: The construction vehicle-mounted intelligent display system and control method based on multimodal interaction provided in this application solves the problems of single interaction, information overload, and warning failure of traditional engineering vehicle displays by constructing a multimodal interaction system that includes vehicle bus, environmental perception, driver monitoring, and distributed tactile feedback, combined with dynamic resource allocation based on safety status level. It realizes the initiative of human-machine interaction, the three-dimensionality of safety warning, and the intelligence of operation process in construction scenarios, reduces the cognitive load of drivers and the risk of misoperation, and improves the safety of operation and construction management efficiency under complex working conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the construction vehicle-mounted intelligent display system based on multimodal interaction according to the present invention.

[0019] Figure 2 This is a schematic diagram of the overall process of the construction vehicle-mounted intelligent display screen control method based on multimodal interaction according to the present invention.

[0020] Figure 3 This is a schematic diagram illustrating the specific process of the construction vehicle-mounted intelligent display control method based on multimodal interaction according to the present invention. Detailed Implementation

[0021] The following detailed description of the construction vehicle-mounted intelligent display system based on multimodal interaction will provide a more comprehensive overview of various embodiments of this disclosure. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0022] For example, the operating environment of construction vehicles (such as excavators and bulldozers) is becoming increasingly complex, and onboard displays have integrated multiple functions such as navigation, surround view, and construction coordination. However, existing systems have the following problems: First, the interaction method is simplistic and operationally risky. Over-reliance on touchscreens makes accidental touches easy in high-vibration, high-dust environments. Drivers need to operate the machinery with both hands and keep their eyes on the outside while working; frequent screen touches lead to hand-eye conflict and severely distract attention. Second, the information layout is rigid, resulting in a heavy cognitive load. Static screen information cannot be dynamically adjusted according to driving or operational status. Key warnings are mixed with irrelevant information, making it difficult for drivers to quickly grasp key points in complex conditions, leading to slow reaction times. Finally, the warning mechanism is passive and poorly adaptable to the environment. Traditional audible and visual warnings are easily masked or ignored in noisy construction environments, and the lack of multi-channel (such as tactile) mandatory warning mechanisms based on risk levels fails to ensure effective alerts in dangerous situations.

[0023] To address the aforementioned issues, this embodiment provides a construction vehicle-mounted intelligent display system based on multimodal interaction. By integrating multimodal data and dynamically prioritizing data, it resolves hand-eye conflict and achieves safe and efficient intelligent construction interaction.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figure 1 The diagram shown is a schematic of a construction vehicle-mounted intelligent display system based on multimodal interaction in a specific embodiment. The system includes an on-board intelligent controller. The vehicle-mounted intelligent controller is connected to a vehicle bus interface, a driver monitoring camera, a gesture recognition sensor, a touch screen, a physical operation module, a distributed tactile warning device, and at least one environmental perception sensor. It should be noted that the vehicle-mounted intelligent controller, as the core processing unit, integrates perception, decision-making and control functions. By running specific algorithms, it can fuse multi-source heterogeneous data from vehicles, the environment and drivers, which is the logical basis for realizing intelligent dynamic interactive scheduling and connects and coordinates the work of the following modules. Vehicle bus interface, used to acquire vehicle status data; It should be noted that the vehicle bus interface directly connects to the vehicle's CAN / Ethernet, allowing for low-cost acquisition of underlying data without damaging the original vehicle wiring. This ensures real-time monitoring of the vehicle's operating intentions and status, which is a prerequisite for risk assessment. Driver monitoring cameras are used to collect data on the driver's head posture and line of sight. It should be noted that the driver monitoring camera converts the driver's subjective behavior of deviating from the line of sight into a calculable digital signal, which solves the problem that traditional in-vehicle systems only focus on the vehicle and not the driver, enabling the system to determine whether the driver is suitable to receive information. Gesture recognition sensors are used to collect the driver's gesture operation data; A touch display screen is used to provide a visual interactive interface and receive touch input; It should be noted that the touch screen, as the main visual interaction carrier for information output, with high brightness and anti-glare characteristics, ensures that information is visible in strong light construction environments, while retaining interactive capabilities as a touch input terminal. The physical operation module is used to receive physical button or knob operations from the driver; It should be noted that the physical operation module provides clear operation stops and tactile feedback through physical buttons and knobs, making it suitable for operation while wearing gloves. This solves the problems of high accidental touch rates and the need for visual guidance when using touchscreens in bumpy environments, thus ensuring operational safety. A distributed tactile warning device is used to provide tactile vibration feedback to the driver; It should be noted that the distributed tactile warning device is a supplement to the non-visual channel. When the driver's visual load is overloaded (e.g. looking outside the car) or hearing is affected by noise (e.g. engine noise), information is transmitted through skin touch. It is highly concealed, has little interference, and can accurately locate (e.g., a left vibration indicates a car on the left), with extremely high transmission efficiency. Environmental perception sensors are used to collect data on the environment surrounding the vehicle. It should be noted that by collecting panoramic and obstacle data outside the vehicle, quantitative detection of blind spots and potential risks is achieved, expanding the driver's environmental perception range. The vehicle-mounted intelligent controller is configured as follows: Based on the vehicle status data, environmental perception data, driver head posture and gaze direction data, and gesture operation data, the safety situation level of the current driving scenario is assessed in real time. Based on the safety situation level, the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device are dynamically allocated, and the information layout and content of the touch screen are dynamically adjusted.

[0026] This embodiment solves the problems of limited traditional screen interaction and ineffective early warning by introducing physical control and distributed haptic feedback, combined with dynamic safety situation assessment, thereby improving operational safety and efficiency.

[0027] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the specific implementation process in this embodiment, another construction vehicle-mounted intelligent display system based on multimodal interaction is provided, which includes an on-board intelligent controller. The vehicle-mounted intelligent controller is connected to a vehicle bus interface, a driver monitoring camera, a gesture recognition sensor, a touch screen, a physical operation module, a distributed tactile warning device, and at least one environmental perception sensor. The vehicle bus interface is used to acquire vehicle status data; specifically, it can be a CAN bus or an on-board Ethernet interface, connected to the vehicle chassis controller, engine controller and working device controller, to acquire data such as vehicle speed, engine speed and load, and working device (such as blade and arm) status in real time. A driver monitoring camera is used to collect data on the driver's head posture and line of sight; preferably an infrared or 3D ToF camera is installed on the A-pillar of the cab or the front of the roof to obtain data on the driver's head posture and line of sight in a non-contact manner and to determine whether the driver's attention is focused on the work surface. Gesture recognition sensors are used to collect the driver's gesture operation data; preferably a ToF camera or an infrared depth sensor, installed on the ceiling or above the dashboard in the driver's cab, are used to recognize the driver's preset gesture commands in the air (such as waving to turn a page, clenching a fist to confirm), as a supplementary, non-contact interaction method. A touch display screen is used to provide a visual interactive interface and receive touch input; The physical operation module is used to receive physical button or knob operations from the driver; A distributed tactile warning device is used to provide tactile vibration feedback to the driver; Environmental perception sensors are used to collect perception data of the environment around the vehicle. The environmental perception sensors include at least one 360° surround view camera and radar (such as millimeter-wave radar or lidar). The surround view camera is used to provide panoramic images of the outside of the vehicle, and the radar is used to accurately detect the distance and relative speed of surrounding obstacles, together constituting the source of environmental perception data. The vehicle-mounted intelligent controller is configured as follows: Based on the vehicle status data, environmental perception data, driver head posture and gaze direction data, and gesture operation data, the safety situation level of the current driving scenario is assessed in real time. Based on the safety situation level, the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device are dynamically allocated, and the information layout and content of the touch screen are dynamically adjusted. The vehicle bus interface uses a CAN bus interface or an Ethernet interface to connect the vehicle chassis controller, engine controller, and working device controller. The vehicle status data includes vehicle speed data, engine speed, engine load data, and working device status data. The vehicle-mounted intelligent controller obtains vehicle speed data from the vehicle chassis controller, engine speed data and engine load data from the engine controller, and working device status data from the working device controller through the vehicle bus interface. The physical operation module includes a multi-function knob and several shortcut keys; The multi-function knob and various shortcut keys are integrated and set on the top of the steering wheel spokes or control lever; A multi-function knob for menu navigation and parameter adjustment; Keyboard shortcuts are used to switch display modes or confirm task status with a single keystroke. The distributed tactile warning device includes a linear resonant motor and several eccentric rotary vibration motors; The linear resonant motor is integrated below the touch screen to provide vibration feedback for operation confirmation; Each eccentric rotary vibration motor is integrated into the steering wheel, seat and seat belt, and is used to generate multi-level tactile warnings based on different vibration intensities, frequencies and position signals sent by the vehicle intelligent controller; It also includes a communication module; The communication module is a 4G / 5G cellular communication module or a Wi-Fi module, used to communicate with the cloud dispatch center or local area network server to receive construction task information such as construction drawings, task lists, and dispatch instructions. The vehicle-mounted intelligent controller is also configured as follows: The system receives construction task information through a communication module, controls the touch screen to display it, and provides an interactive interface for one-click confirmation of task status. The vehicle-mounted intelligent controller executes a core algorithm that integrates multi-source data from the vehicle, environment, and driver to assess the safety status level of the current driving scenario in real time. Based on this safety status level, it dynamically decides the priority and feedback intensity of each interaction channel (display, physical buttons, gestures, and touch). For example, in low-risk situations, it prioritizes confirmation using knobs and touch, while in high-risk situations, it forces a full-screen warning and suppresses unsafe operations. At the same time, it dynamically adjusts the layout of the display screen (such as switching between driving mode, work mode, and warning mode) according to the safety status and construction tasks.

[0028] like Figure 2 As shown, the following are embodiments of the control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction provided in this disclosure. This method and the construction vehicle-mounted intelligent display screen system based on multimodal interaction in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction, please refer to the embodiments of the construction vehicle-mounted intelligent display screen system based on multimodal interaction described above.

[0029] The method includes the following steps: S1. Acquire vehicle status data, environmental perception data, driver head posture and gaze direction data, gesture operation data, and construction task information; It should be noted that by simultaneously collecting four types of data—vehicles, environment, drivers, and construction tasks—a digital profile of the construction scenario is constructed, which is a prerequisite for achieving accurate situation assessment and ensures that there are no blind spots in decision-making. S2. Assess the safety status level of the current driving scenario based on vehicle status data, environmental perception data, driver head posture and line of sight data, and gesture operation data; It should be noted that by transforming raw data into a simple level indicator through an algorithm, the transformation from data to cognition is achieved, enabling machines to perceive the level of danger in the current environment. S3. Based on the security status level, dynamically allocate the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device; It should be noted that this step dynamically adjusts the parameters of each interaction channel based on the level obtained in step S2, realizing the on-demand allocation of interaction resources. Convenience is prioritized in low-risk situations, and security is prioritized in high-risk situations. This avoids interference from invalid information, ensures the maximum utilization of system resources, and achieves optimized resource allocation. S4. Based on the assigned priority and feedback intensity, control the touch screen, physical operation module, gesture recognition sensor and distributed tactile early warning device to perform corresponding interactive feedback, and dynamically adjust the information layout and content of the touch screen based on the security status level and construction task information. It should be noted that this step translates the decision made in step S3 into a specific image and vibration that the driver can perceive. The dynamic adjustment of the screen layout ensures that key information (such as warnings and blind spot images) is always in the best position in the driver's field of vision, thus achieving the optimization of human-computer interaction.

[0030] This embodiment assesses the safety situation through multi-source data fusion and dynamically adjusts the screen layout and haptic feedback intensity, solving the problems of dispersed interaction and blind spots in early warning for engineering vehicles, and realizing proactive safety protection and efficient human-machine collaboration during the construction process.

[0031] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation process in this embodiment, another method for controlling a construction vehicle-mounted intelligent display screen based on multimodal interaction is provided, such as... Figure 3 As shown, the method includes the following steps: S1. Acquire vehicle status data, environmental perception data, driver head posture and gaze direction data, gesture operation data, and construction task information; The specific steps of step S1 are as follows: S11. Obtain vehicle speed data from the vehicle chassis controller via the vehicle bus interface, obtain engine speed and engine load data from the engine controller, obtain working device status data from the working device controller, and obtain vehicle status data. S12. Collect panoramic images of the outside of the vehicle through a 360° surround view camera, and detect the relative distance and relative speed between obstacles and the vehicle through millimeter-wave radar or lidar to obtain environmental perception data; S13. Obtain driver's head posture, gaze direction, and hand movement data through driver monitoring camera and gesture recognition sensor to obtain driver operation data; S14. Obtain construction task information by synchronizing construction drawings, daily work orders and task status information from the cloud dispatch center or local area network through 4G / 5G cellular communication modules or Wi-Fi devices; S2. Assess the safety status level of the current driving scenario based on vehicle status data, environmental perception data, driver head posture and line of sight data, and gesture operation data; Specifically, the system comprehensively calculates the vehicle operation risk coefficient V (based on vehicle speed, load, etc.), obstacle approach risk coefficient D (based on distance, relative speed), and driver operation deviation risk coefficient O (based on the duration of line-of-sight deviation). A comprehensive safety situation index S is obtained through weighted summation and mapped to low, medium, and high risk levels. The weighting factors α, β, and γ can be calibrated according to different vehicle models and work types, and satisfy the normalization condition (α+β+γ=1). The operation deviation risk coefficient O is calculated using a piecewise function; for example, 0 for line-of-sight on the work surface, 0.3 for scanning the screen (0-1 second), 0.7 for brief deviation (1-2 seconds), and 1.0 for prolonged deviation (>2 seconds). Specific thresholds are adjustable. The specific steps of step S2 are as follows: S21. Extract vehicle speed from the acquired vehicle status data. Calculate the vehicle operation risk coefficient based on engine load and working device status. ;

[0032] in, This refers to the vehicle's maximum design speed. This represents the current engine load percentage (range 0~100%). The risk factor for the working device's condition is determined based on the condition calibration obtained from the working device's controller during normal operation. When there is a mild abnormality When there is a serious abnormality ; , , As a preset weighting factor, and ; S22. Extract the relative distance between the target object and the vehicle from the acquired environmental perception data. and relative velocity The obstacle approach risk coefficient is calculated using the following formula. :

[0033] in, , These are calibration coefficients; S23. Obtain the driver's gaze direction through the driver monitoring camera and determine the continuous duration of the gaze deviating from the work surface. The operational deviation risk coefficient is calculated using the following piecewise function. : ; S24. Risk factor of approaching obstacles Vehicle operation risk coefficient Operational deviation risk coefficient Substitute the values ​​into the following weighted summation formula to calculate the comprehensive security situation index. :

[0034] in, , , The weighting factors are preset and greater than 0, satisfying α+β+γ=1; based on the comprehensive security situation index... The numerical range in which the value is located determines the corresponding security status level; S3. Based on the security status level, dynamically allocate the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device; The specific steps of step S3 are as follows: S31. A rule table containing the mapping relationship between security status levels and each interaction channel is preset, and the rule table defines at least the following mappings: S311. Output channel priority, wherein the output channels include visual feedback from the touch display screen and haptic feedback from the distributed haptic warning device: When the security situation level is low risk, tactile confirmation feedback takes precedence over visual information display; the tactile confirmation feedback is provided by an operation confirmation vibration from a linear resonant motor below the touch screen; the current display mode is maintained; When the safety situation level is medium risk, visual warnings take precedence over tactile warnings; the visual warnings are partial highlighting on the touch screen or pop-up prompts, and the tactile warnings are a single moderate-intensity vibration from an eccentric rotary vibration motor in the steering wheel or seat; the display mode is automatically switched, and the warning information is highlighted. When the security situation level is high risk, tactile forced warning takes precedence over visual forced warning; the tactile forced warning is that the eccentric rotary vibration motor in the steering wheel and seat continuously exceeds the preset intensity threshold and vibrates at a preset frequency; the visual forced warning is that the touch screen flashes in full and becomes inoperable, forcibly displaying the risk source and guidance information; and forcibly switching to high-risk warning mode. S312. Input channel response strategy, where the input channel is the physical operation module and the gesture recognition sensor: When the safety situation level is low or medium risk, the physical operation module and gesture recognition sensor respond normally, and the driver can browse the menu, adjust parameters and confirm tasks through the multi-function knob or button. When the security situation level is high risk, non-security-related operations in the entity operation module are temporarily suppressed, only preset security confirmation shortcut keys are allowed to take effect, and the gesture recognition function is turned off. S32. Using the calculated security situation level as an index, query the rule table to obtain the optimal interaction channel allocation strategy for the current moment. The optimal interaction channel allocation strategy includes the priority order of output channels, the feedback intensity coefficient of each output channel, display control parameters, haptic feedback control parameters, and the operation response priority of input channels; wherein, the feedback intensity coefficient is determined according to the security situation level according to a preset mapping table. S33. Send the display control parameters in the optimal interaction channel allocation strategy to the touch screen, send the haptic feedback control parameters to the distributed haptic warning device, and configure the operation response priority to the physical operation module; S4. Based on the assigned priority and feedback intensity, control the touch screen, physical operation module, gesture recognition sensor and distributed tactile early warning device to perform corresponding interactive feedback, and dynamically adjust the information layout and content of the touch screen based on the security status level and construction task information. The specific steps of step S4 are as follows: S41. Based on the safety status level, construction task information, and the display control parameters issued in step S3, call the target mode that matches the display control parameters from the preset display modes. The display modes include driving mode, operation mode, and early warning mode. S42. Control the touch screen to switch to the target display mode, dynamically adjust the information layout and content, display construction task information and provide an interactive interface for one-click confirmation of task status; S43. When the driver inputs a command through a physical operation module or a valid gesture, the vibration feedback intensity of the linear resonant motor below the touch screen is triggered to provide a highly responsive operation feedback. S44. When the safety situation level assessed in step S2 increases, interrupt the currently ongoing non-safety-related tactile interaction and call the eccentric rotary vibration motor in the distributed tactile early warning device to generate a multi-point vibration early warning of the corresponding level according to the intensity coefficient in the tactile feedback control parameters.

[0035] For example, taking a typical application of an intelligent bulldozer equipped with this system in earthmoving operations, the system operates according to the following steps: Scenario 1: Moving to the work area The driver starts the vehicle, and the system executes step S1. It obtains the current vehicle speed (e.g., 25 km / h) from the vehicle chassis controller via the vehicle bus interface, the engine speed (1500 rpm) and engine load (30%) from the engine controller, and the blade is in the retracted state from the work device controller. It collects surrounding images through a 360° surround view camera and detects no obstacles ahead through millimeter-wave radar. It detects that the driver's line of sight is facing the road ahead through the driver monitoring camera. It synchronizes the day's work tasks (leveling area A, elevation requirement ±2cm) from the dispatch center via the 4G network. The system executes step S2, based on a vehicle speed of 25 km / h (62.5% lower than the maximum design speed of 40 km / h), engine load of 30%, and normal operating device status. =0), calculate the vehicle operation risk coefficient. =0.2; Based on radar data (no target ahead), obstacle approach risk factor. =0; Based on the driver's line of sight on the work surface (the road ahead), the operational deviation risk coefficient is 0. =0; Substitute into the weighted formula (let...) =0.5, =0.3, =0.2), calculate the comprehensive security situation index. =0.06, which is classified as a low-risk level; The system executes step S3, queries the rule table, and under the low-risk level, the output channel priority is tactile confirmation feedback > visual information display. The display control parameter points to the driving mode, the tactile feedback control parameter is set to low-intensity confirmation vibration of the touch screen linear resonant motor, and the operation response priority is normal (i.e., normal response of the physical operation module). The system executes step S4, the touch screen switches to driving mode, the left 1 / 3 area displays simplified instruments (such as vehicle speed, RPM, water temperature), the right 2 / 3 area displays the navigation map, and the bottom of the screen displays the construction task card: Area A leveling - to begin; when the driver turns on the right turn signal to prepare to turn right, the environmental perception sensor detects an obstacle in the right blind spot (e.g., relative distance 3 meters), step S2 recalculates the safety situation level and raises it to medium risk, step S3 immediately adjusts the display control parameters to warning mode, and the tactile feedback control parameters are set to a single medium-intensity vibration of the right-side eccentric rotary vibration motor of the steering wheel (frequency 50Hz, intensity 0.7, belonging to tactile warning), step S4 is executed, the right side of the screen automatically switches to the real-time image of the right blind spot camera and superimposes the steering prediction trajectory line (i.e., visual warning), and at the same time the right-side eccentric rotary vibration motor of the steering wheel vibrates with medium intensity to remind the driver to pay attention to the right side.

[0036] Scenario 2: Receiving Tasks and Starting Work After the vehicle arrives at the work area, the driver presses the task shortcut key (i.e., the physical operation module) on the steering wheel. In step S4, S43 is triggered, and the linear resonant motor provides a short vibration (e.g., 0.1 seconds, intensity 0.8) as operation confirmation. In step S1, the driver's operation data of the task key being pressed is simultaneously acquired. In step S3, the operation response priority causes the instruction to be processed immediately, and the display control parameters are temporarily switched to the "Collaborative Construction" interface in the work mode. The tactile feedback control parameters maintain a low-risk setting. In step S4, the touch screen displays a large digital map of the work area and the leveling elevation requirements for the day, and a task list pops up. The driver uses the multi-function knob on the right to select the sub-task: Leveling Area A, presses the knob to confirm, and the system reports via the 4G network: Task received. After the work begins, the blade contacts the ground, and the working device controller feeds back the blade pressure data. In step S1, the working device status change is acquired (i.e., blade presses down, load increases to 70%). In step S2, the safety situation is recalculated, the vehicle speed is reduced to 5 km / h, the engine load is 70%, and the working device operates normally. =0), =0.35; Radar detected no nearby obstacles ahead. =0; The driver's gaze is focused on the working surface in front of the blade. =0; =0.105, still low risk; step S3 automatically switches the display mode to the "fine operation" interface of the operation mode, and the screen displays information such as blade posture, flatness deviation, and compaction guidance.

[0037] Scenario 3: Dynamic early warning during operation During the operation, the millimeter-wave radar suddenly detected a person rapidly approaching from the left rear of the vehicle (relative distance 2 meters, relative speed 1.5 m / s); step S22 calculates the obstacle approach risk factor. ,set up =1, =0.5, then D=0.5+0.75=1.25; Step S21: Current vehicle speed 3km / h, engine load 65%, working device normal. =0.3; The driver's line of sight is still forward. =0; let =0; =0.5, =0.2, =0.3, then S=0.5×1.25+0.2×0.3+0.3×0=0.685, which is judged as a high-risk level; Step S31: Query the rule table. Under high-risk levels, the output channel priority is tactile forced warning > visual forced warning. The input channel response strategy is: non-safety-related operations in the physical operation module are temporarily suppressed, and only safety confirmation buttons are allowed to function. Step S32: Generate the optimal interaction channel allocation strategy: the tactile feedback control parameters are set to the eccentric rotary vibration motors on the left side of the steering wheel and the left side of the seat vibrating continuously at the highest intensity (e.g., amplitude 1.0, frequency 80Hz) (i.e., tactile forced warning); the display control parameters are forcibly switched to high-risk warning mode, requiring a full-screen display of a magnified view of the left blind spot with a superimposed red flashing border (visual warning). (Forced warning, inoperable); Step S33 sends out parameters; Step S44 immediately interrupts the currently ongoing non-safety-related tactile interaction (such as menu browsing vibration), calls the eccentric rotary vibration motors on the left side of the steering wheel and the left side of the seat to execute high-intensity vibration, and at the same time Step S42 forces the touch screen to switch to the left blind spot magnified view, displaying the warning text "Personnel approaching from the left rear!" in full screen; After the driver feels the forced vibration on the left, he immediately stops working, observes the left side and takes evasive action; At this time, except for the confirmation warning shortcut key, all other multi-function knobs / shortcut keys in the physical operation module are temporarily suppressed to prevent accidental operation.

[0038] Scenario 4: Task Completion Reporting After leveling area A, the driver presses the task shortcut key on the steering wheel again. Step S43 provides operation confirmation vibration, and the screen pops up the current task status menu. The driver uses the multi-function knob to select "complete" and press confirm. Step S14 reports the progress and completion quality data (leveling deviation value) to the dispatch center through the 4G / 5G cellular communication module. After receiving the feedback, the dispatch center updates the task status, and the system synchronously displays that the task has been completed.

[0039] This embodiment demonstrates, through step-by-step explanation of the specific scenarios described above, the closed-loop process from data acquisition, situation assessment, strategy allocation to interactive execution, and verifies that safe, efficient and intelligent interaction can be effectively achieved in complex construction environments.

[0040] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0041] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A construction vehicle-mounted intelligent display system based on multimodal interaction, characterized in that, Including in-vehicle intelligent controllers; The vehicle-mounted intelligent controller is connected to a vehicle bus interface, a driver monitoring camera, a gesture recognition sensor, a touch screen, a physical operation module, a distributed tactile warning device, and at least one environmental perception sensor. Vehicle bus interface, used to acquire vehicle status data; Driver monitoring cameras are used to collect data on the driver's head posture and line of sight. Gesture recognition sensors are used to collect the driver's gesture operation data; A touch display screen is used to provide a visual interactive interface and receive touch input; The physical operation module is used to receive physical button or knob operations from the driver; A distributed tactile warning device is used to provide tactile vibration feedback to the driver; Environmental perception sensors are used to collect data on the environment surrounding the vehicle. The vehicle-mounted intelligent controller is configured as follows: Based on the vehicle status data, environmental perception data, driver head posture and gaze direction data, and gesture operation data, the safety situation level of the current driving scenario is assessed in real time. Based on the safety situation level, the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device are dynamically allocated, and the information layout and content of the touch screen are dynamically adjusted.

2. The construction vehicle-mounted intelligent display system based on multimodal interaction according to claim 1, characterized in that, The vehicle bus interface uses a CAN bus interface or an Ethernet interface to connect the vehicle chassis controller, engine controller, and working device controller. The vehicle status data includes vehicle speed data, engine speed, engine load data, and working device status data. The vehicle-mounted intelligent controller obtains vehicle speed data from the vehicle chassis controller, engine speed data and engine load data from the engine controller, and working device status data from the working device controller through the vehicle bus interface.

3. The construction vehicle-mounted intelligent display system based on multimodal interaction according to claim 1, characterized in that, Environmental sensing sensors include 360° surround-view cameras and radar; A 360° surround-view camera is used to capture panoramic images of the vehicle's exterior. The radar uses millimeter-wave radar or lidar to detect the distance and relative speed of obstacles.

4. The construction vehicle-mounted intelligent display system based on multimodal interaction according to claim 1, characterized in that, The driver monitoring camera uses an infrared camera or a 3D ToF camera and is installed on the A-pillar of the cab or the front of the roof. The gesture recognition sensor is a ToF camera or an infrared depth sensor, installed on the ceiling or above the dashboard in the driver's cab, to recognize the driver's preset gesture commands in the air.

5. The construction vehicle-mounted intelligent display system based on multimodal interaction according to claim 1, characterized in that, The physical operation module includes a multi-function knob and several shortcut keys; The multi-function knob and various shortcut keys are integrated and set on the top of the steering wheel spokes or control lever; A multi-function knob for menu navigation and parameter adjustment; Keyboard shortcuts are used to switch display modes or confirm task status with a single keystroke. The distributed tactile warning device includes a linear resonant motor and several eccentric rotary vibration motors; The linear resonant motor is integrated below the touch screen to provide vibration feedback for operation confirmation; Each eccentric rotary vibration motor is integrated into the steering wheel, seat, and seat belt to generate multi-level tactile warnings based on different vibration intensities, frequencies, and position signals sent by the vehicle's intelligent controller.

6. The construction vehicle-mounted intelligent display system based on multimodal interaction according to claim 1, characterized in that, It also includes a communication module; The communication module is a 4G / 5G cellular communication module or a Wi-Fi module, used to receive construction task information from the cloud dispatch center or local area network server; The vehicle-mounted intelligent controller is also configured as follows: The system receives construction task information via a communication module, controls the touchscreen display to show the information, and provides an interactive interface for one-click confirmation of task status.

7. A control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction, characterized in that, Includes the following steps: S1. Acquire vehicle status data, environmental perception data, driver head posture and gaze direction data, gesture operation data, and construction task information; S2. Assess the safety status level of the current driving scenario based on vehicle status data, environmental perception data, driver head posture and line of sight data, and gesture operation data; S3. Based on the security status level, dynamically allocate the interaction priority and feedback intensity of the touch screen, physical operation module, gesture recognition sensor and distributed tactile warning device; S4. Based on the assigned priority and feedback intensity, control the touch screen, physical operation module, gesture recognition sensor and distributed tactile early warning device to perform corresponding interactive feedback, and dynamically adjust the information layout and content of the touch screen based on the security status level and construction task information.

8. The control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction according to claim 7, characterized in that, The specific steps of step S2 are as follows: S21. Extract vehicle speed from the acquired vehicle status data. Calculate the vehicle operation risk coefficient based on engine load and working device status. ; S22. Extract the relative distance between the target object and the vehicle from the acquired environmental perception data. and relative velocity The obstacle approach risk coefficient is calculated using the following formula. : in, , These are calibration coefficients; S23. Obtain the driver's gaze direction through the driver monitoring camera and determine the continuous duration of the gaze deviating from the work surface. The operational deviation risk coefficient is calculated using the following piecewise function. : ； S24. Risk factor of approaching obstacles Vehicle operation risk coefficient Operational deviation risk coefficient Substitute the values ​​into the following weighted summation formula to calculate the comprehensive security situation index. : in, , , The weighting factors are preset and greater than 0, satisfying α+β+γ=1; based on the comprehensive security situation index... The numerical range in which the value is located determines the corresponding security status level.

9. The control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction according to claim 7, characterized in that, The specific steps of step S3 are as follows: S31. A rule table containing the mapping relationship between security status levels and each interaction channel is preset, and the rule table defines at least the following mappings: S311. Output channel priority, wherein the output channels include visual feedback from the touch display screen and haptic feedback from the distributed haptic warning device: When the security situation level is low risk, tactile confirmation feedback takes precedence over visual information display; the tactile confirmation feedback is provided by an operation confirmation vibration from a linear resonant motor below the touch screen; the current display mode is maintained; When the security situation level is medium risk, visual warnings take precedence over tactile warnings. The visual warning is a partial highlight on the touch screen or a pop-up notification; the tactile warning is a single, moderate-intensity vibration from an eccentric rotary vibration motor in the steering wheel or seat; the display mode is automatically switched, and the warning information is highlighted. When the security situation level is high-risk, tactile mandatory warning takes precedence over visual mandatory warning. The tactile forced warning is that the eccentric rotary vibration motor in the steering wheel and seat continuously exceeds the preset intensity threshold and vibrates at a preset frequency. The visual forced warning is that the touch screen flashes in full screen and becomes inoperable, and the risk source and guidance information are forcibly displayed. Forced switch to high-risk warning mode; S312. Input channel response strategy, where the input channel is the physical operation module and the gesture recognition sensor: When the safety situation level is low or medium risk, the physical operation module and gesture recognition sensor respond normally, and the driver can browse the menu, adjust parameters and confirm tasks through the multi-function knob or button. When the security situation level is high risk, non-security-related operations in the entity operation module are temporarily suppressed, only preset security confirmation shortcut keys are allowed to take effect, and the gesture recognition function is turned off. S32. Using the calculated security situation level as an index, query the rule table to obtain the optimal interaction channel allocation strategy for the current moment. The optimal interaction channel allocation strategy includes the priority order of output channels, the feedback intensity coefficient of each output channel, display control parameters, haptic feedback control parameters, and the operation response priority of input channels; wherein, the feedback intensity coefficient is determined according to the security situation level according to a preset mapping table. S33. Send the display control parameters in the optimal interaction channel allocation strategy to the touch screen, send the haptic feedback control parameters to the distributed haptic warning device, and configure the operation response priority to the physical operation module.

10. The control method for a construction vehicle-mounted intelligent display screen based on multimodal interaction according to claim 9, characterized in that, The specific steps of step S4 are as follows: S41. Based on the safety status level, construction task information, and the display control parameters issued in step S3, a target display mode matching the display control parameters is called from the preset display modes. The display modes include driving mode, operation mode, and early warning mode to determine the target display mode. S42. Control the touch screen to switch to the target display mode, dynamically adjust the information layout and content, display construction task information and provide an interactive interface for one-click confirmation of task status; S43. When the driver inputs a command through a physical operation module or a valid gesture, the vibration feedback intensity of the linear resonant motor below the touch screen is triggered to provide a highly responsive operation feedback. S44. When the safety situation level assessed in step S2 increases, interrupt the currently ongoing non-safety-related tactile interaction and call the eccentric rotary vibration motor in the distributed tactile early warning device to generate a multi-point vibration early warning of the corresponding level according to the intensity coefficient in the tactile feedback control parameters.

Images