Tractor intelligent cab man-machine interaction system and interaction method

Abstract

The invention discloses a tractor intelligent cockpit man-machine interaction system and an interaction method, and relates to the technical field of tractor intelligent cockpits, and the tractor intelligent cockpit man-machine interaction system structurally comprises a multi-modal data acquisition module, an environment intelligent sensing module, an AI big data processing module, a control module, an actuator assembly and an adaptive display module. The data acquisition module is used for acquiring information in the intelligent cockpit. The environment sensing module is used for collecting information around the tractor and information of the tractor. And the data processing module is used for processing and transmitting the data. And the control module is used for outputting a control instruction. And the actuator assembly is used for displaying the completion result. The interaction method comprises the following steps: 1, data acquisition; 2, processing the data; 3, making a control instruction; 4, the actuator assembly completes corresponding actions; and 5, the adaptive display module displays a result. The tractor man-machine interaction system can solve the technical problems that an operation interface of an existing tractor man-machine interaction system is not friendly, complex environment information cannot be efficiently processed, and interaction efficiency is low.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit technology for tractors, specifically to an intelligent cockpit human-machine interaction system and interaction method for tractors. Background Technology

[0002] Tractors, as an important tractor in modern agriculture, can be equipped with different attachments and tools, such as plows, harrows, seeders, and sprayers. They can quickly and efficiently complete various agricultural tasks such as tilling, sowing, fertilizing, and harvesting, and play a very crucial role in promoting the modernization of agricultural production methods, reducing labor intensity, and promoting economic development.

[0003] With the development of smart agriculture, the operational complexity and dependence on operators of traditional tractors have limited the efficiency and widespread adoption of agricultural machinery. Although various autonomous driving technologies have emerged in recent years, existing tractor human-machine interaction systems still suffer from problems such as unfriendly interfaces, inefficient handling of complex environmental information, and low interaction efficiency, which limit the performance of tractor autonomous driving systems. Summary of the Invention

[0004] The purpose of this invention is to provide a human-machine interaction system and method for an intelligent driver's cab of a tractor, which can solve the technical problems of existing human-machine interaction systems of tractors, such as unfriendly operation interface, inability to efficiently process complex environmental information, and low interaction efficiency.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A tractor intelligent cockpit human-machine interaction system includes a multimodal data acquisition module, an environmental intelligent perception module, an AI big data processing module, a control module, an actuator component, and an adaptive display module.

[0007] The multimodal data acquisition module is used to collect data related to the driver within the intelligent cockpit.

[0008] The environmental intelligent sensing module is used to collect data on the environment surrounding the tractor and the tractor itself.

[0009] The AI ​​big data processing module processes the data collected by the multimodal data acquisition module and the environmental intelligent perception module, and transmits the processing results to the cockpit control module.

[0010] The control module is used to make corresponding control commands based on the data processing results transmitted by the AI ​​big data processing module and pass them to the actuator components.

[0011] The actuator component is used to complete the corresponding actions according to the instructions issued by the control module, and displays the results through the adaptive display module.

[0012] Furthermore, the multimodal data acquisition module includes an in-cabin vision sensor and a voice sensor installed in the cockpit. The in-cabin vision sensor is used to acquire facial data, gesture language, and image data of the driver's driving status, while the voice sensor is used to acquire the driver's voice commands.

[0013] Furthermore, the environmental intelligent perception module includes an external vision sensor and an onboard millimeter-wave radar installed on the tractor. The vision sensor is used to collect photos of the environment around the tractor, while the onboard radar and lidar are used to collect distance information and three-dimensional data of distant objects.

[0014] Furthermore, the AI ​​big data processing module is used to filter, calibrate, and synchronize the received data, perform data fusion, improve the accuracy and reliability of the data, and realize driver face recognition, gesture language recognition, voice command recognition, as well as tractor surrounding environment recognition, distance and 3D recognition of distant objects based on the processed data.

[0015] Furthermore, the adaptive display module includes AR display and monitor display for displaying navigation, operation information, and visualization of machine status.

[0016] A human-machine interaction method for a tractor intelligent cockpit, based on the aforementioned interaction system, includes the following steps: S1. Data is collected from the cockpit and the surrounding environment of the deammoniation waste through the multimodal data acquisition module and the environmental intelligent sensing module; S2, the AI ​​big data processing module processes the collected data; S3. The control module issues corresponding control commands based on the processed data. S4. The actuator assembly completes the corresponding action according to the control command; S5, the adaptive display module displays the results.

[0017] By adopting the above technical solution, the present invention has the following beneficial effects: 1. This invention realizes gesture command operation and voice command operation through a multimodal data acquisition module and an AI big data processing module. Compared with traditional mechanical button operation and touch screen operation, it realizes non-contact and low-intervention operation, making the operation process more in line with human natural habits, improving the operation experience while reducing the need for driver's attention. 2. This invention achieves active detection of the environment around the tractor through an intelligent environmental perception module and an AI big data processing module. It can also convey the recognition results to the driver by distinguishing the degree of importance through multiple display methods, so that the driver can focus on the information with high importance and improve the driver's efficiency in processing environmental information. 3. By designing a feedback mechanism, this invention can proactively and promptly transmit the execution results to the driver, avoiding the problem of low interaction efficiency caused by the driver needing to actively confirm the results. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall structure of the interactive system in this invention.

[0019] Figure 2 This is a schematic diagram of the workflow of the interaction method in this invention.

[0020] Figure 3 This is a schematic diagram illustrating the principle of personnel identification in the interactive system in this embodiment of the invention.

[0021] Figure 4 This is a schematic diagram illustrating the principle of voice command recognition in the interactive system according to an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram illustrating the principle of how the interactive system identifies and displays the surrounding environment in an embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the principle of the AI ​​big data processing module of the interactive system in this embodiment of the invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the features and performance of a tractor intelligent cockpit human-machine interaction system and interaction method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] Please see the appendix Figures 1-2 A tractor intelligent cockpit human-machine interaction system includes a multimodal data acquisition module, an environmental intelligent perception module, an AI big data processing module, a control module, an actuator component, and an adaptive display module.

[0026] The multimodal data acquisition module is used to collect driver-related data within the intelligent cockpit. The module includes an in-cabin vision sensor and a voice sensor located within the cockpit. The in-cabin vision sensor collects facial data, gestures, and image data related to the driver's driving status, while the voice sensor collects the driver's voice commands.

[0027] The intelligent environmental sensing module is used to collect data on the environment surrounding the tractor and the tractor itself. The module includes an external vision sensor mounted on the tractor and an onboard millimeter-wave radar. The vision sensor captures images of the environment around the tractor, while the onboard radar and lidar collect distance information and 3D data of distant objects.

[0028] The AI ​​big data processing module processes the data collected by the multimodal data acquisition module and the environmental intelligent perception module, and transmits the processing results to the cockpit control module. This module filters, calibrates, and synchronizes the received data, performs data fusion to improve data accuracy and reliability, and uses the processed data to achieve driver facial recognition, gesture recognition, voice command recognition, as well as recognition of the tractor's surrounding environment, distance to distant objects, and 3D recognition.

[0029] The control module is used to make corresponding control commands based on the data processing results transmitted by the AI ​​big data processing module and pass them to the actuator components.

[0030] The actuator component performs corresponding actions according to the instructions issued by the control module and displays the results through the adaptive display module. The adaptive display module includes an AR display and a monitor display, used to display navigation, operation information, and visualization of machine status.

[0031] A human-machine interaction method for a tractor intelligent cockpit, based on the aforementioned interaction system, includes the following steps.

[0032] S1. Data is collected from the cockpit and the surrounding environment of the deammoniation waste through the multimodal data acquisition module and the environmental intelligent sensing module; S2, the AI ​​big data processing module processes the collected data; S3. The control module issues corresponding control commands based on the processed data. S4. The actuator assembly completes the corresponding action according to the control command; S5, the adaptive display module displays the results.

[0033] In a specific implementation, a tractor intelligent cockpit human-machine interaction system includes a multimodal data acquisition module, an environmental intelligent perception module, an AI big data processing module, a control module, an actuator component, an adaptive display module, and a feedback mechanism.

[0034] The multimodal data acquisition module is a key component of the tractor intelligent cockpit human-machine interaction system. It acquires various types of data from the tractor driver, including visual, auditory, and tactile data, through multiple sensors and data acquisition technologies, enabling the human-machine interaction system to better understand the driver's state. The multimodal data acquisition module utilizes multiple sensors, including in-cabin visual sensors, voice sensors, and tactile sensors, to collect relevant data from the driver.

[0035] The in-cabin visual sensors include monocular and binocular cameras, capable of capturing facial images and body language images such as gestures, as well as images of the driver's driving status. Voice sensors can capture the driver's voice commands. Tactile sensors are located on the steering wheel, seat, and other points of contact between the driver and the vehicle, used to detect signals such as pressure, vibration, and temperature.

[0036] The intelligent environmental perception module includes external vision sensors, millimeter-wave radar, and a GPS module, used to collect information about the tractor's surroundings, providing a comprehensive understanding of the vehicle's environment and enhancing the capabilities of the driver and driver assistance systems. This module is the core of the tractor's intelligent cockpit human-machine interface system, improving driving safety and providing the driver with a more comfortable and convenient driving experience.

[0037] The AI ​​big data processing module is used to process various types of data information collected by the multimodal data acquisition module and the environmental intelligent perception module, including filtering, calibration, synchronization, and data fusion to improve the accuracy and reliability of the data. Based on this data, it executes decision-making and control commands. The AI ​​big data processing module consists of two main parts: hardware and software. The hardware part includes a central processing unit (CPU), a graphics processing unit (GPU), storage devices, and network connection devices, while the software part includes an operating system, data processing algorithms, machine learning models, and a user interface.

[0038] The control module generates corresponding control commands based on the data processing results transmitted from the AI ​​big data processing module and sends them to the actuator components. The actuator components then perform the corresponding actions according to the commands issued by the control module and provide feedback on the execution results.

[0039] The adaptive display module displays the processing results data. Its main function is to enhance the readability of the information and provide drivers with a personalized user experience. The adaptive display module uses a high-resolution screen and virtual reality (AR) for displaying results. The high-resolution screen uses an OLED or LCD high-resolution display to provide clear visuals and a wide field of view. The AR display uses AR technology built into the windshield or goggles to provide intuitive data such as navigation, work information, and machine status.

[0040] The feedback mechanism can send signals back to the driver based on the collected data, providing real-time feedback to the operator through means such as seat vibration, sound prompts, and visual signals, ensuring the accuracy and safety of operation.

[0041] In addition to the components mentioned above, the tractor intelligent cockpit human-machine interaction system also includes a data transmission module and a communication module. The data transmission module can complete the transmission of various data within the interaction system, while the communication module can complete the communication tasks between various sensors, the AI ​​big data processing module, and other components within the system.

[0042] A human-machine interaction method for a tractor intelligent cockpit, such as Figure 2 As shown, the multimodal data acquisition module and the environmental intelligent perception module first collect various data in the cab and around the tractor. Then, the collected data is transmitted to the AI ​​big data processing module to process the various types of data accordingly. The processing results are then transmitted to the control module, which controls the actuator components to complete the corresponding actions and displays the results.

[0043] Among them, when performing driver identification and monitoring of driving status, such as Figure 3 As shown, the driver's image information is first collected by a monocular camera in the tractor's intelligent cockpit. The collected image information is then transmitted to the AI ​​big data processing module, where the driver's identity is confirmed by the built-in facial recognition algorithm. Once the driver's identity is confirmed, the driver can operate the tractor.

[0044] Meanwhile, the AI ​​big data processing module utilizes a built-in target detection algorithm to perform real-time target recognition on driver images captured by the binocular camera, segmenting facial features such as the eyes and mouth in the driver's image. Based on facial expression features, it determines the driver's current driving status. If the driver is determined to be in a state of fatigue, a feedback mechanism will remind the driver.

[0045] When recognizing driver voice commands, gestures, and other body language, and performing corresponding actions based on the recognition results, such as... Figure 4 As shown, the raw data of the driver's voice and body movements are first obtained through the voice sensor and binocular camera in the intelligent cockpit of the tractor.

[0046] A voice sensor collects the driver's voice commands from inside the cockpit and transmits the data to an AI big data processing module for processing. The AI ​​big data processing module first performs voice identification to confirm the driver's identity. Once the voice identification is successful, the AI ​​big data processing module processes the voice commands. The control module issues corresponding control commands based on the processing results, and the actuator components complete the corresponding actions according to the control commands. After the voice command is executed, a feedback mechanism will notify the driver that the command has been completed. If the driver's voice identification fails, the voice command will not be executed.

[0047] Meanwhile, the onboard binocular camera in the tractor's intelligent cab can capture images of the driver's gestures and other body language, transmitting these images to the AI ​​big data processing module for visual processing. The AI ​​big data processing module first recognizes the gestures and other body language in the images to determine the corresponding commands. The corresponding commands are then transmitted to the driver for confirmation via a feedback mechanism. If the driver confirms again, the AI ​​big data processing module's result is passed to the control module for execution. If the driver does not respond or provide feedback, the command will not be executed.

[0048] When collecting and processing data on the tractor's surrounding environment and its own condition, and adaptively displaying the results, such as... Figure 5 As shown, firstly, the external vision sensors around the tractor collect images of the surrounding environment. After image data processing by the AI ​​big data processing module, the processing results are used for target recognition of objects around the tractor and lane keeping.

[0049] Meanwhile, the millimeter-wave radar on the vehicle can collect 3D point cloud data of objects around the tractor. Through the point cloud data processing algorithm built into the AI ​​big data processing module, the distance and speed of surrounding objects can be measured, assisting the vehicle in obstacle avoidance and automatic braking. The onboard GPS module provides the vehicle with precise location and navigation information, providing the technological foundation for the tractor's autonomous driving and path planning. AR display technology and high-resolution displays can provide intuitive data such as navigation, work information, and machine status.

[0050] like Figure 6 As shown, the AI ​​big data processing module, as the data processing center of the intelligent cockpit, plays a crucial role in the realization of the entire intelligent cockpit's functions. This module mainly consists of three parts: software, algorithms, and hardware. The software includes frameworks such as PyCharm, Matlab, Python, and PyTorch. The algorithms include face recognition algorithms (FaceNet model), object detection algorithms (Yolo series models), and speech recognition algorithms (RNN, Transformer models), etc. Most of the above algorithms are based on deep learning models and require high-performance computer processors to complete. The hardware is the computer equipment used to run the software and algorithms.

[0051] It should be noted that the parts not described in detail in this solution are all prior art. The above embodiments are only used to illustrate the present invention, but the present invention is not limited to the above embodiments. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A human-machine interaction system for an intelligent driver's cab of a tractor, characterized in that: It includes a multimodal data acquisition module, an intelligent environmental sensing module, an AI big data processing module, a control module, actuator components, and an adaptive display module. The multimodal data acquisition module is used to collect driver-related data within the intelligent cockpit. The environmental intelligent sensing module is used to collect data on the tractor's surrounding environment and the tractor itself. The AI ​​big data processing module processes the data collected by the multimodal data acquisition module and the environmental intelligent perception module, and transmits the processing results to the cockpit control module. The control module is used to generate corresponding control commands based on the data processing results transmitted by the AI ​​big data processing module and then pass them to the actuator components. The actuator component is used to complete the corresponding actions according to the instructions issued by the control module, and displays the results through the adaptive display module.

2. The intelligent driver's cab human-machine interaction system for tractors as described in claim 1, characterized in that: The multimodal data acquisition module includes an in-cabin vision sensor and a voice sensor installed in the cockpit. The in-cabin vision sensor is used to acquire facial data, gesture language and image data of the driver's driving status, while the voice sensor is used to acquire the driver's voice commands.

3. The tractor intelligent cockpit human-machine interaction system as described in claim 1, characterized in that: The environmental intelligent perception module includes an external vision sensor and an onboard millimeter-wave radar installed on the tractor. The vision sensor is used to collect photos of the environment around the tractor, while the onboard radar and lidar are used to collect distance information and three-dimensional data of distant objects.

4. The tractor intelligent cockpit human-machine interaction system as described in claim 1, characterized in that: The AI ​​big data processing module is used to filter, calibrate, and synchronize the received data, perform data fusion, improve the accuracy and reliability of the data, and realize driver face recognition, gesture language recognition, voice command recognition, as well as tractor surrounding environment recognition, distance and 3D recognition of distant objects based on the processed data.

5. The intelligent driver's cab human-machine interaction system for tractors as described in claim 1, characterized in that: The adaptive display module includes AR display and monitor display, used to display navigation, operation information, and visualization of tractor vehicle status.

6. A human-machine interaction method for a tractor intelligent cockpit, based on the interaction system as described in claim 1, characterized in that: Includes the following steps, S1. Data is collected from the cab and the environment around the tractor through the multimodal data acquisition module and the intelligent environmental perception module; S2, the AI ​​big data processing module processes the collected data; S3. The control module issues corresponding control commands based on the processed data. S4. The actuator assembly completes the corresponding action according to the control command; S5, the adaptive display module displays the results.

Images