Intelligent agricultural machine driving simulation system and method, storage medium and electronic device

Abstract

This application discloses a driving simulation system, method, storage medium, and electronic device for intelligent agricultural machinery. The driving simulation system includes: a physical control console for collecting user operation commands for the simulated control equipment; a simulation module for scheduling a simulation model to process the user operation commands and obtain simulation state data; a rendering module for rendering the simulation state data to obtain a simulated driving screen; and a display module for displaying the simulated driving screen. The simulation model includes: an agricultural machinery dynamics model for simulating the dynamic characteristics of the intelligent agricultural machinery, an agricultural scene model for simulating a three-dimensional agricultural scene, and an operation interaction model for simulating the interaction between the intelligent agricultural machinery and the three-dimensional agricultural scene. This application can achieve high-fidelity simulation of real three-dimensional agricultural scenes and agricultural machinery operation effects, overcoming the limitation of general driving simulation systems being disconnected from the real farmland environment, and improving the simulation accuracy of the driving simulation system.

Description

Technical Field

[0001] This application relates to the field of simulation technology, specifically to a driving simulation system, method, storage medium, and electronic device for intelligent agricultural machinery. Background Technology

[0002] With the development of smart agriculture, the application of intelligent agricultural machinery such as autonomous driving and remote-controlled agricultural machinery is becoming increasingly widespread, which also places higher demands on the simulation effect of driving simulation systems for intelligent agricultural machinery. Traditional driving simulation systems for intelligent agricultural machinery usually use general driving simulation systems with simple modifications, providing only basic driving simulation functions without effectively integrating with agricultural scenarios. This results in significant differences between the simulated virtual scenarios and the real agricultural environment, making it difficult to reproduce the real operating effects of intelligent agricultural machinery. Summary of the Invention

[0003] This application provides a driving simulation system, method, storage medium, and electronic device for intelligent agricultural machinery, which can achieve high-fidelity simulation of real agricultural three-dimensional scenes and agricultural machinery operation effects, breaking through the limitation of general driving simulation systems being disconnected from real farmland environments, and improving the simulation accuracy of intelligent agricultural machinery driving simulation systems.

[0004] In a first aspect, embodiments of this application provide a driving simulation system for intelligent agricultural machinery. The driving simulation system includes: a physical control console, comprising multiple simulated control devices for the intelligent agricultural machinery, used to collect user operation commands for the simulated control devices; a simulation module connected to the physical control console, used to schedule a simulation model to simulate and process the user operation commands to obtain simulation state data; a rendering module connected to the simulation module, used to render the simulation state data to obtain a simulated driving screen; and a display module connected to the rendering module, used to display the simulated driving screen. The simulation model includes an agricultural machinery dynamics model, an agricultural scene model, and an operation interaction model; the agricultural machinery dynamics model is used to simulate the dynamic characteristics of the intelligent agricultural machinery, the agricultural scene model is used to simulate a three-dimensional agricultural scene, and the operation interaction model is used to simulate the interaction effect between the intelligent agricultural machinery and the three-dimensional agricultural scene.

[0005] In some embodiments, the operation interaction model is a mathematical model developed based on a physics engine; wherein, the input of the operation interaction model includes at least one of the following: the type of implement of the intelligent agricultural machine, the operation parameters of the intelligent agricultural machine, the driving parameters of the intelligent agricultural machine, and soil parameters; the output of the operation interaction model includes at least one of the following: virtual load parameters acting on the intelligent agricultural machine, and visual parameters of the operation effect of the intelligent agricultural machine.

[0006] In some embodiments, the simulation model further includes an abnormal event model; wherein the abnormal event model is used to simulate abnormal events encountered by the intelligent agricultural machinery during operation.

[0007] In some embodiments, the abnormal event includes at least one of the following: agricultural machinery malfunction event, abnormal weather event, obstacle event, and crop abnormal event.

[0008] In some embodiments, each of the simulated control devices includes: a data acquisition module for acquiring user operation commands for the simulated control device; and a driving feedback module for executing driving feedback operations corresponding to the user operation commands.

[0009] In some embodiments, the physical control console further includes a vibration feedback module for performing vibration feedback operations; wherein the vibration feedback operations are used to simulate the vibration excitation experienced by the intelligent agricultural machinery during operation.

[0010] In some embodiments, the simulation module includes an external interaction interface for connecting to an external controller, so that the simulation module and the external controller can perform closed-loop simulation tests.

[0011] In some embodiments, the driving simulation system further includes an evaluation module connected to the simulation module, configured to output simulation evaluation data based on the user operation instructions and / or the simulation state data.

[0012] In some embodiments, the evaluation indicators in the simulation evaluation data include at least one of the following: operation accuracy indicator, operation efficiency indicator, and anomaly handling indicator.

[0013] Secondly, embodiments of this application provide a driving simulation method for intelligent agricultural machinery, applied to the aforementioned driving simulation system for intelligent agricultural machinery. The driving simulation method includes: responding to user operation commands for a simulated control device, scheduling a simulation model to simulate and process the user operation commands to obtain simulation state data; performing rendering processing based on the simulation state data to obtain a simulated driving screen; and displaying the simulated driving screen. The simulation model includes an agricultural machinery dynamics model, an agricultural scene model, and an operation interaction model; the agricultural machinery dynamics model is used to simulate the dynamic characteristics of the intelligent agricultural machinery, the agricultural scene model is used to simulate a three-dimensional agricultural scene, and the operation interaction model is used to simulate the interaction effect between the intelligent agricultural machinery and the three-dimensional agricultural scene.

[0014] In some embodiments, the driving simulation method further includes: outputting simulation evaluation data based on the user operation instructions and / or the simulation state data.

[0015] In some embodiments, the driving simulation method further includes: sending virtual scene data of an agricultural three-dimensional scene to an external controller; and, in response to a device control command from the external controller, scheduling the simulation model to perform simulation processing on the device control command in order to update the virtual scene data.

[0016] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps in the driving simulation method for intelligent agricultural machinery as described above.

[0017] Fourthly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program or instructions, and when the computer program or instructions are executed by the processor, they implement the steps in the intelligent agricultural machinery driving simulation method described above.

[0018] In summary, the technical solution provided in this application provides a driving simulation system for intelligent agricultural machinery, comprising a physical control console, a simulation module, a rendering module, and a display module. The physical control console collects user operation commands for the simulated control equipment of the intelligent agricultural machinery. The simulation module schedules the simulation model to process the user operation commands to obtain simulation state data. The rendering module renders the simulation state data to obtain a simulated driving screen. The display module displays the simulated driving screen. The simulation model may include an agricultural machinery dynamics model for simulating the dynamic characteristics of the intelligent agricultural machinery, an agricultural scene model for simulating a three-dimensional agricultural scene, and an operation interaction model for simulating the interaction between the intelligent agricultural machinery and the three-dimensional agricultural scene. Therefore, this application embodiment sets up simulation models specifically for agricultural scenes, achieving high-fidelity simulation of real three-dimensional agricultural scenes and agricultural machinery operation effects. This overcomes the limitation of general driving simulation systems being disconnected from real farmland environments, improving the simulation accuracy of the intelligent agricultural machinery driving simulation system. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a driving simulation system for intelligent agricultural machinery provided in an embodiment of this application; Figure 2 This is a schematic diagram of another intelligent agricultural machinery driving simulation system provided in an embodiment of this application; Figure 3This is a schematic diagram of a driving simulation method for intelligent agricultural machinery provided in an embodiment of this application; Figure 4 This is a schematic diagram of another intelligent agricultural machinery driving simulation method provided in the embodiments of this application; Figure 5 This is a schematic diagram of an intelligent agricultural machine provided in an embodiment of this application; Figure 6 This is a schematic diagram of another intelligent agricultural machine provided in an embodiment of this application; Figure 7 This is a schematic diagram of another intelligent agricultural machinery driving simulation system provided in an embodiment of this application; Figure 8 This is a schematic diagram of the system framework of a driving simulation system provided in an embodiment of this application; Figure 9 This is a schematic diagram of a principle disassembly function interface provided in an embodiment of this application; Figure 10 This is a schematic diagram of another principle-based disassembly function interface provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, specific embodiments of this application will be illustrated with reference to steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be referred to several times as being performed by a computer. Computer performance as referred to in this application includes operations performed by a computer processing unit on electronic signals represented by data in a structured format. This operation transforms the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise alter the operation of the computer in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of this application are illustrated with specific embodiments and are not intended to be limiting. Those skilled in the art will understand that many of the steps and operations described below can also be implemented in hardware.

[0023] The terms "module" or "unit" as used in this application can be considered as software objects executing on the computing system. The different components, modules, engines, and services described in this application can be considered as implementation objects on the computing system. While the apparatus and methods described in this application are preferably implemented in software, they can also be implemented in hardware, both of which are within the scope of protection of this invention.

[0024] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used in the embodiments of this application may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that when an element is “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein may include wireless connection or wireless coupling. The term “and / or” as used herein includes all or any unit and all combinations of one or more associated listed items.

[0025] Driving simulation systems for intelligent agricultural machinery can be used for operator skills training and for testing and verifying intelligent agricultural machinery control algorithms. However, traditional driving simulation systems for intelligent agricultural machinery have the following drawbacks: First, insufficient scene realism: the 3D scene model is fixed and cannot be dynamically generated and changed according to agronomic parameters (such as crop growth cycle, planting row spacing, etc.), lacking physical realism; second, lack of interactive models: it only simulates the basic movements of agricultural machinery, without constructing mechanical and effect models of the interaction between specific agricultural implements and soil and crops such as plowing, sowing, fertilizing, and harvesting, and cannot simulate actual operational loads and effects; third, closed system: it cannot perform hardware-in-the-loop or software-in-the-loop integration testing with real automatic driving controllers (such as drive-by-wire chassis, planning and decision-making algorithms), limiting its application. Therefore, traditional driving simulation systems for intelligent agricultural machinery have limited training effects on operators and cannot be effectively used for the development and testing of intelligent agricultural machinery control algorithms.

[0026] In view of this, embodiments of this application provide a driving simulation system, driving simulation method, and storage medium for intelligent agricultural machinery, which can simulate real three-dimensional agricultural scenes and agricultural machinery operation effects with high fidelity, and effectively improve the skill training effect for operators and the testing and verification effect of intelligent agricultural machinery control algorithms.

[0027] Please see Figure 1 , Figure 1 This is a schematic diagram of a driving simulation system for intelligent agricultural machinery provided in an embodiment of this application. Figure 1As shown, the driving simulation system 100 may include a physical driver's console 110, a simulation module 120, a rendering module 130, and a display module 140.

[0028] The physical control console 110 can replicate the actual cab of the intelligent agricultural machinery, allowing the operator to recreate the control methods for the machinery. Based on this, the physical control console 110 can include multiple simulated control devices for the intelligent agricultural machinery. The layout of these simulated control devices within the physical control console 110 is consistent with the layout of the actual control devices within the intelligent agricultural machinery, thus highly replicating the actual cab. This application embodiment does not limit the specific type of simulated control devices; in practical applications, they can be flexibly configured according to requirements and the actual control devices of the intelligent agricultural machinery. For example, multiple simulated control devices may include, but are not limited to, at least one of the following: steering wheel, accelerator pedal, brake pedal, gear lever, hydraulic control lever, operation control buttons, multi-function armrest box, handbrake, ignition switch, etc. The operator can operate any one or more simulated control devices. Therefore, the physical control console 110 can be used to collect user operation commands for the simulated control devices and send the collected user operation commands to the simulation module 120.

[0029] The simulation module 120 runs a simulation engine to manage the simulation process, such as scheduling the simulation model and processing input and output signals. In this embodiment, the simulation module 120 is connected to the physical console 110, such as via hardwired connections or a network, allowing data interaction between the simulation module 120 and the physical console 110. Based on this, the simulation module 120 responds to user operation commands from the physical console 110, scheduling the simulation model to process the user operation commands and obtain simulation state data. The simulation model can be stored in the simulation module 120 or on a server connected to the simulation module 120 at high speed; this embodiment does not limit the storage location.

[0030] To achieve high-fidelity simulation of real-world 3D agricultural scenes and agricultural machinery operation effects, the simulation model in this application embodiment may include an agricultural machinery dynamics model, an agricultural scene model, and an operation interaction model. The agricultural machinery dynamics model is used to simulate the dynamic characteristics of intelligent agricultural machinery, such as the kinematics and dynamic characteristics of different or specific models of tractors, harvesters, and other intelligent agricultural machinery. The agricultural scene model is used to simulate 3D agricultural scenes, such as dynamically generating virtual 3D agricultural scenes based on GIS (Geographic Information System) data and agronomic parameters (such as crop growth cycles, planting row spacing, etc.). These 3D agricultural scenes may include, but are not limited to, terrain undulations, soil type zoning, crop row distribution, and / or 3D models of crops in different growth states. The operation interaction model is used to simulate the interaction between intelligent agricultural machinery and the 3D agricultural scene. The intelligent agricultural machinery is equipped with one or more implements (such as plows, seeders, fertilizer applicators, harvesting platforms, etc.), thus the operation interaction model can be used to simulate the interaction between the intelligent agricultural machinery and its implements and the 3D agricultural scene.

[0031] The rendering module 130 runs a rendering engine, which can be used to render simulation-related images. In this embodiment, the simulation module 120 and the rendering module 130 can be integrated into a single main control unit or into different main control units. This main control unit can be a computer (such as a high-performance industrial computer), a chip, etc. The rendering module 130 is connected to the simulation module 120, such as via a hardwired connection or network, allowing data interaction between them. Based on this, the rendering module 130 responds to simulation state data from the simulation module 120, performs rendering processing based on the simulation state data, and obtains a simulated driving image. Since the simulation module 120 schedules the simulation model to obtain simulation state data, the simulation state data can include, but is not limited to, at least one of the following: agricultural machinery status data of intelligent agricultural machinery, virtual scene data of agricultural three-dimensional scenes, and operational effect data of intelligent agricultural machinery. The rendering module 130 can render a visualized simulated driving scene based on the simulation state data. The simulated driving scene may include, but is not limited to, at least one of the following: a virtual screen of the agricultural machinery dashboard of the intelligent agricultural machinery, a virtual screen of the three-dimensional agricultural scene (such as crop texture, field landscape, etc.), and a screen of the operation effect of the intelligent agricultural machinery (such as turned soil, cut straw, etc.).

[0032] The display module 140 is used to display the simulation effect of the driving simulation system. In this embodiment, the display module 140 is connected to the rendering module 130, such as via hardwired connections or a network, so that the display module 140 and the rendering module 130 can interact with each other. Based on this, the display module 140 responds to the simulated driving screen from the rendering module 130 and displays the simulated driving screen. The display module 140 may include one or more displays, and the display module 140 can display the simulated driving screen on the displays. When the display module 140 includes multiple displays, the multiple displays can be installed at different positions on the physical driver's console 110 (such as in front of the seat on the physical driver's console 110, or to the side front of the physical driver's console 110); and the simulated driving screens displayed on the multiple displays may be the same or different (for example, the simulated driving screen based on a first-person perspective is displayed on the display screen in front of the seat, and the simulated driving screen based on other auxiliary perspectives such as the rearview mirror is displayed on the display screen to the side front).

[0033] In summary, the technical solution provided in this application provides a driving simulation system for intelligent agricultural machinery, comprising a physical control console, a simulation module, a rendering module, and a display module. The physical control console collects user operation commands for the simulated control equipment of the intelligent agricultural machinery. The simulation module schedules the simulation model to process the user operation commands to obtain simulation state data. The rendering module renders the simulation state data to obtain a simulated driving screen. The display module displays the simulated driving screen. The simulation model may include an agricultural machinery dynamics model for simulating the dynamic characteristics of the intelligent agricultural machinery, an agricultural scene model for simulating a three-dimensional agricultural scene, and an operation interaction model for simulating the interaction between the intelligent agricultural machinery and the three-dimensional agricultural scene. Therefore, this application embodiment sets up simulation models specifically for agricultural scenes, achieving high-fidelity simulation of real three-dimensional agricultural scenes and agricultural machinery operation effects. This overcomes the limitation of general driving simulation systems being disconnected from real farmland environments, improving the simulation accuracy of the intelligent agricultural machinery driving simulation system.

[0034] In some embodiments, the above-mentioned operation interaction model is a mathematical model developed based on a physics engine (such as Bullet, PhysX, etc.). The inputs to the operation interaction model may include, but are not limited to, at least one of the following: the type of agricultural implement of the intelligent agricultural machinery (such as plow, seeder, fertilizer applicator, harvester, etc.), the operation parameters of the intelligent agricultural machinery (such as operation depth, etc.), the driving parameters of the intelligent agricultural machinery (such as driving speed, etc.), and soil parameters (such as soil moisture, soil hardness, etc.). The outputs of the operation interaction model may include, but are not limited to, at least one of the following: virtual load parameters acting on the intelligent agricultural machinery (such as virtual load torque, virtual load resistance, etc.), and visual parameters of the operation effect of the intelligent agricultural machinery.

[0035] This application embodiment achieves precise quantitative simulation of the interaction process between intelligent agricultural machinery and the three-dimensional agricultural scene by setting the operation interaction model as a mathematical model developed based on a physics engine and explicitly defining its multi-dimensional input parameters and multi-type output results. Based on this, the simulation module can dynamically calculate the mechanical feedback acting on the intelligent agricultural machinery and the visual representation of the operation effect according to different agronomic conditions. This not only provides real-time and accurate driving data for the physical control console, but also enables the rendering module to present dynamic scene changes that are highly consistent with the actual operation process, significantly improving the simulation fidelity and physical credibility of the driving simulation system in terms of operation load perception and operation effect visualization.

[0036] In some embodiments, the simulation model may further include an anomaly event model. The anomaly event model is used to simulate anomaly events encountered by the intelligent agricultural machinery during operation. For example, the anomaly event model can randomly simulate anomaly events that the intelligent agricultural machinery may encounter during its operation or travel. This application embodiment does not limit the specific types and number of anomaly events simulated by the anomaly event model. In practical applications, it can be flexibly set according to requirements. For example, the anomaly events simulated by the anomaly event model may include, but are not limited to, at least one of the following: agricultural machinery malfunction events (such as engine stalling), abnormal weather events (such as sudden heavy fog or rainstorms), obstacle events (such as stones or animals appearing in the field), and crop anomaly events (such as crop lodging or pest / disease patches).

[0037] This application's embodiments introduce an abnormal event model, enabling the simulation module to randomly trigger various abnormal events during the operation or driving of intelligent agricultural machinery, thus achieving dynamic simulation of unexpected situations in real farmland operations. Based on this, this application's embodiments overcome the limitations of traditional driving simulation systems that only support ideal operating conditions. Operators can repeatedly train their ability to identify and handle emergencies such as machinery malfunctions, severe weather, obstacles, and crop abnormalities in a virtual environment. Simultaneously, the random injection mechanism of abnormal events provides reproducible stress testing conditions for the robustness testing of intelligent agricultural machinery control algorithms, effectively supporting the reliability verification and optimization iteration of intelligent agricultural machinery control algorithms under extreme conditions. Therefore, this application's embodiments effectively expand the application depth and scenario coverage of driving simulation systems in skills training and algorithm testing.

[0038] In some embodiments, each simulated control device in the physical control console 110 includes a data acquisition module and a driving feedback module. The data acquisition module is used to acquire user operation commands for the simulated control device; for example, the data acquisition module may include sensors, etc. The driving feedback module is used to execute driving feedback operations corresponding to the user operation commands; for example, the driving feedback module may include a high-precision force feedback device, etc. The user operation commands and driving feedback operations may differ depending on the type of simulated control device. For example, when the simulated control device is a steering wheel, the user operation commands include rotation commands (such as rotation angle, rotation angular velocity, etc.), and the driving feedback operations include damping feedback operations or power assist feedback operations; when the simulated control device is a pedal, the user operation commands include pedal commands (such as pedal depth, pedal speed, etc.), and the driving feedback operations include damping feedback operations.

[0039] This application embodiment achieves precise mapping between user operation commands and simulated working condition feedback by independently configuring data acquisition modules and driving feedback modules for each simulated control device. Based on this, this application embodiment can enable the physical driving platform to reproduce the mechanical response characteristics of real agricultural machinery in typical operations such as steering, braking, and implement control, thereby enhancing the immersive experience and training realism of the driving simulation system.

[0040] In some embodiments, the physical control console 110 may further include a vibration feedback module. The vibration feedback module performs vibration feedback operations to simulate the vibration excitation experienced by the intelligent agricultural machinery during operation. This application embodiment does not limit the specific type of vibration excitation; in practical applications, it can be flexibly set according to requirements. For example, vibration excitation may include, but is not limited to, at least one of the following: road surface excitation experienced by the intelligent agricultural machinery during driving (such as bumps caused by uneven road surfaces), and work load excitation experienced by the intelligent agricultural machinery during operation (such as vibrations generated when implements cut into soil, cut crops, or encounter rocks or roots).

[0041] Based on this, the vibration feedback module can perform vibration feedback operations according to the road excitation signal and / or work load excitation signal output by the simulation module 120. For example, the vibration feedback module can calculate vibration parameters (such as frequency, amplitude, direction, etc.) according to the road excitation signal and / or work load excitation signal, and drive the physical driver's cab 110 to generate corresponding vibrations according to the vibration parameters. The embodiments of this application do not limit the vibration type of the physical driver's cab 110. In practical applications, it can be flexibly set according to requirements. For example, the vibration feedback module can drive the physical driver's cab 110 to generate at least one of the following types of vibrations: vertical vibration (such as up and down bumping), pitch vibration (such as front and back nodding), side tilt vibration (such as left and right swaying), yaw vibration (such as horizontal torsion), etc.

[0042] This application embodiment incorporates a vibration feedback module into the physical control console to drive it to vibrate, simulating the vibration excitation experienced by the intelligent agricultural machinery during operation. This achieves accurate reproduction of the external vibrations experienced by the intelligent agricultural machinery during operation. Furthermore, because the vibration feedback module allows the operator to perceive bumps caused by uneven road surfaces and impact vibrations when implements cut into the soil or encounter obstacles in the simulated environment, it effectively broadens the feedback dimensions of the driving simulation system, enhances the immersion and physical realism of the system, and makes operator skills training more closely resemble real farmland operation scenarios, effectively improving the transferability and reliability of simulation training.

[0043] In some embodiments, the simulation module 120 may include an external interaction interface for connecting to an external controller, enabling the simulation module 120 and the external controller to perform closed-loop simulation testing. The external interaction interface may include hardware interfaces (such as CAN (Controller Area Network) bus, Ethernet, RS232 serial port, etc.) and / or software interfaces (such as API (Application Programming Interface), communication protocols, etc.), thus, the closed-loop simulation testing may include hardware-in-the-loop testing and / or software-in-the-loop testing, which is not limited in this embodiment.

[0044] External controllers may include, but are not limited to, real autonomous driving domain controllers and remote driving controllers. The simulation module 120 and the external controller can interact via an external interface. For example, the simulation module 120 sends virtual scene data of an agricultural 3D scene to the external controller; the external controller generates device control commands for the simulated operating equipment based on the virtual scene data and feeds these commands back to the simulation module 120; in response to the device control commands, the simulation module 120 schedules the simulation model to process the device control commands, thereby updating the virtual scene data and achieving closed-loop simulation testing.

[0045] This application embodiment integrates an external interaction interface into the simulation module to achieve bidirectional data interaction and closed-loop simulation testing with external controllers such as real autonomous driving domain controllers and remote driving controllers. The simulation module can output high-fidelity virtual scene data of an agricultural 3D scene to the external controller and receive equipment control commands generated by the external controller based on intelligent agricultural machinery control algorithms (such as autonomous driving algorithms) to drive the simulation model, thereby updating the virtual scene data and forming a complete test closed loop. This application embodiment supports both hardware-in-the-loop testing to verify the integration performance of controller hardware and algorithms, and software-in-the-loop testing to accelerate algorithm iteration development. It effectively reduces the cost and risk of testing external controllers in real farmland environments, providing an efficient and reproducible laboratory testing environment for the safety verification and reliability optimization of intelligent agricultural machinery control algorithms.

[0046] In some embodiments, such as Figure 2 As shown, the driving simulation system 100 may further include an evaluation module 150. The evaluation module 150 can be used to evaluate the simulation process of the driving simulation system 100 to output simulation evaluation data. The simulation process of the driving simulation system 100 may include operator skills training or testing and verification of intelligent agricultural machinery control algorithms, thus the simulation evaluation data may include operator skills training reports or algorithm performance testing and verification reports. In this embodiment, the evaluation module 150 is connected to the simulation module 120, such as via hardwired connections or a network, allowing data interaction between the evaluation module 150 and the simulation module 120. Based on this, the evaluation module 150 outputs simulation evaluation data according to user operation instructions and / or simulation status data.

[0047] The evaluation module 150 can evaluate the simulation process based on preset evaluation indicators, so that the simulation evaluation data includes the evaluation results of each evaluation indicator. This application embodiment does not limit the specific type of evaluation indicators; in practical applications, they can be flexibly set according to requirements. For example, the evaluation indicators in the simulation evaluation data may include, but are not limited to, at least one of the following: operational accuracy indicators (such as operational path straightness, operational coverage, and duplicate / missed rate), operational efficiency indicators (such as simulated energy consumption per unit area and operational efficiency), and anomaly handling indicators (such as operator response time to abnormal events and processing accuracy).

[0048] This application embodiment, by setting up an evaluation module, realizes the quantitative evaluation and result output of the simulation process of the driving simulation system. In practical applications, the simulation evaluation data output by the evaluation module can not only provide objective and quantifiable improvement basis for operators' skills, but also provide standardized test benchmarks for the performance verification and iterative optimization of intelligent agricultural machinery control algorithms, thus expanding the application value of driving simulation systems in the fields of skills training and algorithm development.

[0049] Please see Figure 3 , Figure 3 This is a schematic diagram of a driving simulation method for intelligent agricultural machinery provided in an embodiment of this application. This driving simulation method for intelligent agricultural machinery can be applied to the driving simulation system for intelligent agricultural machinery described in the above embodiment, as described above. Figure 1 and Figure 2 The driving simulation system shown. (As shown) Figure 3 As shown, the driving simulation method may include the following steps S310 to S330.

[0050] Step S310: In response to the user operation command for the simulated control device, the simulation model is scheduled to perform simulation processing on the user operation command to obtain simulation state data. Step S320: Render the simulation state data to obtain the simulated driving scene; Step S330: Display the simulated driving screen.

[0051] The simulation models mentioned above may include agricultural machinery dynamics models, agricultural scene models, and operation interaction models. The agricultural machinery dynamics model is used to simulate the dynamic characteristics of intelligent agricultural machinery, the agricultural scene model is used to simulate three-dimensional agricultural scenes, and the operation interaction model is used to simulate the interaction effect between intelligent agricultural machinery and three-dimensional agricultural scenes.

[0052] In some embodiments, the above-mentioned operation interaction model is a mathematical model developed based on a physics engine; wherein, the input of the operation interaction model may include, but is not limited to, at least one of the following: the type of agricultural implements of the intelligent agricultural machine, the operation parameters of the intelligent agricultural machine, the driving parameters of the intelligent agricultural machine, and the soil parameters; the output of the operation interaction model may include, but is not limited to, at least one of the following: the virtual load parameters acting on the intelligent agricultural machine, and the visual parameters of the operation effect of the intelligent agricultural machine.

[0053] In some embodiments, the simulation model may further include an abnormal event model; wherein the abnormal event model is used to simulate abnormal events encountered by intelligent agricultural machinery during operation.

[0054] In some embodiments, abnormal events may include, but are not limited to, at least one of the following: agricultural machinery malfunction events, abnormal weather events, obstacle events, and crop abnormal events.

[0055] In some embodiments, the driving simulation method described above may further include: performing driving feedback operations corresponding to user operation commands.

[0056] In some embodiments, the driving simulation method described above may further include: performing a vibration feedback operation; wherein the vibration feedback operation is used to simulate the vibration excitation experienced by the intelligent agricultural machinery during operation.

[0057] In some embodiments, such as Figure 4 As shown, the above driving simulation method may further include: Step S340: Send virtual scene data of the agricultural 3D scene to the external controller; Step S350: In response to the device control command from the external controller, the simulation model is scheduled to simulate and process the device control command in order to update the virtual scene data.

[0058] In some embodiments, such as Figure 4 As shown, the above driving simulation method may further include: Step S360: Output simulation evaluation data based on user operation instructions and / or simulation status data.

[0059] In some embodiments, the evaluation indicators in the above simulation evaluation data may include, but are not limited to, at least one of the following: operation accuracy indicators, operation efficiency indicators, and anomaly handling indicators.

[0060] It should be understood that Figure 3 and Figure 4 In the driving simulation method shown, each step can be continuously executed during the simulation process. For example, steps S310 to S330 or steps S340 to S350 are continuously executed during the simulation until the simulation ends.

[0061] For further details regarding the steps and beneficial effects of the above-described driving simulation method, please refer to the embodiments of the driving simulation system described above; these details will not be elaborated upon here.

[0062] The technical solutions provided in the embodiments of this application will be described below with several examples.

[0063] Please see Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of an intelligent agricultural machine provided in an embodiment of this application. Figure 6 This is a schematic diagram of another intelligent agricultural machine provided in an embodiment of this application. For example... Figure 5 and Figure 6 As shown, the intelligent agricultural machine 510 can be equipped with corresponding agricultural implements. Figure 5 The intelligent agricultural machine 510 shown is equipped with a rotary tiller 521. Figure 6 The intelligent agricultural machine 510 shown is equipped with a hydraulic reversible plow 522.

[0064] Please see Figure 7 , Figure 7 This is a schematic diagram of another intelligent agricultural machinery driving simulation system provided in an embodiment of this application. For example... Figure 7As shown, the physical cab 110 in this driving simulation system is used to replicate the real cab of intelligent agricultural machinery at a 1:1 scale, and can imitate the layout of mainstream or specific tractor cabs. For example... Figure 7 As shown, the physical control console 110 may include multiple simulated control devices for the intelligent agricultural machinery (such as a steering wheel, pedals, multi-functional armrest box, etc.); in addition, the physical control console 110 may also include a seat, where the operator can sit and operate one or more simulated control devices to replicate the real operation of the intelligent agricultural machinery.

[0065] The simulation and rendering modules in a driving simulation system can be integrated into a single main control unit, which can be a high-performance industrial computer.

[0066] The simulation module can run simulation management software or a simulation engine developed based on the ROS (Robot Operating System) framework. This engine is responsible for scheduling the simulation model and calculating the simulated world state at a specific frequency (e.g., 100Hz). Before the simulation begins, the operator or other users can configure the required agricultural scene and agricultural machinery parameters so that the simulation model can acquire the relevant parameters for simulation. In the simulation model, the agricultural machinery dynamics model can be built and exported based on multibody dynamics software (such as Adams). The operation interaction model is the core, simulating the interaction between intelligent agricultural machinery and its implements and the three-dimensional agricultural scene. For example, taking tillage as an example, the operation interaction model can be used to control the steering wheel to provide realistic steering force feedback to simulate the damping feel of different road conditions (such as rugged fields and hard surfaces), and to control the pedals to also have corresponding damping feel.

[0067] The rendering module can run a 3D rendering engine based on Unreal Engine 4 and communicate with the simulation engine in the simulation module through a custom plugin. The rendering module can render in real-time detailed crop textures, field landscapes that change as harvesting progresses, and high-definition images of the virtual dashboard of the intelligent agricultural machinery. The rendering module can output the rendered driving simulation footage to displays in front of the physical driver's seat (such as three LCD screens and a rearview mirror display).

[0068] The evaluation module in the driving simulation system can continuously record data such as the trajectory, fuel consumption, and grain loss rate of intelligent agricultural machinery in the virtual simulation world, and generate simulation evaluation data after the simulation ends. When operators undergo skills training through the driving simulation system, this simulation evaluation data can be used to indicate problems that occur during operation, such as the header colliding with the simulated field ridge due to failure to slow down in advance when turning at the edge of the field.

[0069] Please see Figures 8 to 10 , Figure 8 This is a schematic diagram of the system framework of a driving simulation system provided in an embodiment of this application. Figure 9This is a schematic diagram of a principle disassembly function interface provided in an embodiment of this application. Figure 10 This is a schematic diagram of another principle-based disassembly function interface provided in an embodiment of this application. For example... Figure 8 As shown, the driving simulation system 100 can be used for theoretical learning of intelligent agricultural machinery (including learning libraries, question banks, theoretical assessments, and score inquiries), principle disassembly (including equipment recognition, disassembly, and assembly), simulation training (including simulation training and actual assessments), and drive systems (including user management, system logs, system backups, and assessment management). Figure 9 As shown, the principle disassembly function interface 910 displays various function buttons, such as structural composition, working principle, assembly simulation, disassembly simulation, scene debugging, assessment, and evaluation. Operators can operate the corresponding function buttons to enter the corresponding function for simulation skills training. Figure 10 As shown, the learning tasks and assessment requirements for the corresponding functions can be displayed in the principle disassembly function interface 920, so that the operator can understand the learning objectives of this skills training.

[0070] Driving simulation systems can also be used for testing and verifying intelligent agricultural machinery control algorithms (such as autonomous driving algorithms). For example, the autonomous driving domain controller of a real intelligent agricultural machine can be connected to the main controller via a CAN bus. The simulation module sends virtual scene data of the agricultural 3D scene to the autonomous driving domain controller, such as virtual GPS / IMU data, camera video streams (generated by the rendering module), and LiDAR point clouds. The autonomous driving domain controller perceives the virtual scene data to make path planning and lateral and longitudinal control decisions, generates equipment control commands (such as steering angle and throttle opening), and feeds them back to the simulation module via the CAN bus. The simulation module schedules the simulation model to process the equipment control commands, drives the virtual agricultural machinery in the virtual simulation world, and updates the virtual scene data to form a complete closed-loop test, which can fully verify the performance of the algorithm in various complex farmland environments in a laboratory environment.

[0071] In summary, the beneficial effects of the technical solution provided in this application include: First, high simulation and professionalism: by integrating high-fidelity simulation models (such as agricultural scene models and operation interaction models), it can accurately simulate the real operation process under different agronomic conditions, improving the realism and effectiveness of the simulation process; Second, multi-purpose: it can be used for operator skills training and assessment, as well as for the development, testing and verification of intelligent agricultural machinery control algorithms and remote driving systems, reducing R&D testing costs and risks; Third, hardware-in-the-loop integration capability: the open interface design allows access to real external controllers, realizing the leap from pure software simulation to semi-physical simulation, and the test results are closer to the performance of the real system; Fourth, quantitative evaluation and closed-loop improvement: the built-in evaluation module can perform multi-dimensional quantitative scoring of operation or algorithm performance, providing clear data guidance for skill improvement and algorithm optimization, forming a closed loop of "training / testing-evaluation-improvement".

[0072] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11 The electronic device 1100 can be used to implement the steps described in the above method embodiments.

[0073] Electronic device 1100 may include one or more processors 1110. The processor 1110 supports the electronic device 1100 in implementing the steps described in the preceding method embodiments. The processor 1110 may be a general-purpose processor or a special-purpose processor. For example, the processor 1110 may be implemented as a Central Processing Unit (CPU); or, the processor 1110 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor 1110 may be any conventional processor, etc.

[0074] The electronic device 1100 may further include one or more memories 1120. The memories 1120 store computer programs or instructions. The memories 1120 may be independent of the processor 1110 or integrated into the processor 1110. The computer programs or instructions in the memories 1120 can be executed by the processor 1110, causing the processor 1110 to perform the intelligent agricultural machinery driving simulation method described in the above embodiments.

[0075] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0076] Therefore, this application also provides a computer-readable storage medium storing a computer program or instructions thereon, which is loaded by a processor to execute the intelligent agricultural machinery driving simulation method described in the above embodiments.

[0077] For details on the implementation of each of the above operations / steps, please refer to the previous examples, which will not be repeated here.

[0078] Computer-readable storage media may include: read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc.

[0079] Since the computer program stored in the computer-readable storage medium can execute the steps in any of the above method embodiments provided in the embodiments of this application, the beneficial effects that the methods described in any of the above method embodiments can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0080] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0081] The above provides a detailed description of a driving simulation system, method, storage medium, and electronic device for intelligent agricultural machinery provided by the embodiments of this application. Specific examples have been used in this application to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A driving simulation system for intelligent agricultural machinery, characterized in that, The driving simulation system includes: The physical control console includes multiple simulated control devices for the intelligent agricultural machinery, used to collect user operation commands for the simulated control devices; The simulation module, connected to the physical control console, is used to schedule the simulation model to process the user operation commands and obtain simulation state data. The simulation model includes an agricultural machinery dynamics model, an agricultural scene model, and an operation interaction model. The agricultural machinery dynamics model is used to simulate the dynamic characteristics of the intelligent agricultural machinery, the agricultural scene model is used to simulate a three-dimensional agricultural scene, and the operation interaction model is used to simulate the interaction effect between the intelligent agricultural machinery and the three-dimensional agricultural scene. A rendering module, connected to the simulation module, is used to perform rendering processing based on the simulation state data to obtain a simulated driving scene. The display module is connected to the rendering module and is used to display the simulated driving screen.

2. The driving simulation system according to claim 1, characterized in that, The task interaction model is a mathematical model developed based on a physics engine; whereby, The input to the operation interaction model includes at least one of the following: the type of agricultural implement of the intelligent agricultural machine, the operation parameters of the intelligent agricultural machine, the driving parameters of the intelligent agricultural machine, and soil parameters; The output of the operation interaction model includes at least one of the following: virtual load parameters applied to the intelligent agricultural machinery, and visual parameters of the operation effect of the intelligent agricultural machinery.

3. The driving simulation system according to claim 1, characterized in that, The simulation model also includes an abnormal event model; wherein the abnormal event model is used to simulate abnormal events encountered by the intelligent agricultural machinery during operation.

4. The driving simulation system according to claim 3, characterized in that, The abnormal events include at least one of the following: agricultural machinery failure events, abnormal weather events, obstacle events, and crop abnormal events.

5. The driving simulation system according to claim 1, characterized in that, Each of the aforementioned simulation control devices includes: The data acquisition module is used to acquire the user operation commands for the simulated control device; The driving feedback module is used to execute driving feedback operations corresponding to the user's operation commands.

6. The driving simulation system according to claim 1, characterized in that, The physical control console also includes: A vibration feedback module is used to perform vibration feedback operations; wherein, the vibration feedback operations are used to simulate the vibration excitation experienced by the intelligent agricultural machinery during operation.

7. The driving simulation system according to claim 1, characterized in that, The simulation module includes: An external interaction interface is provided for connecting to an external controller, enabling the simulation module and the external controller to perform closed-loop simulation tests.

8. The driving simulation system according to any one of claims 1 to 7, characterized in that, The driving simulation system also includes: An evaluation module, connected to the simulation module, is used to output simulation evaluation data based on the user operation instructions and / or the simulation status data.

9. The driving simulation system according to claim 8, characterized in that, The evaluation indicators in the simulation evaluation data include at least one of the following: operation accuracy indicator, operation efficiency indicator, and anomaly handling indicator.

10. A driving simulation method for intelligent agricultural machinery, characterized in that, A driving simulation system applied to the intelligent agricultural machinery according to any one of claims 1 to 9; the driving simulation method includes: In response to user operation commands for the simulated control device, the simulation model is scheduled to perform simulation processing on the user operation commands to obtain simulation state data. The simulation state data is rendered to obtain a simulated driving scene. Display the simulated driving screen; The simulation model includes an agricultural machinery dynamics model, an agricultural scene model, and an operation interaction model. The agricultural machinery dynamics model is used to simulate the dynamic characteristics of the intelligent agricultural machinery, the agricultural scene model is used to simulate a three-dimensional agricultural scene, and the operation interaction model is used to simulate the interaction effect between the intelligent agricultural machinery and the three-dimensional agricultural scene.

11. The driving simulation method according to claim 10, characterized in that, The driving simulation method also includes: Based on the user operation instructions and / or the simulation status data, output simulation evaluation data.

12. The driving simulation method according to claim 10, characterized in that, The driving simulation method also includes: Send virtual scene data of the agricultural 3D scene to an external controller; In response to a device control command from the external controller, the simulation model is scheduled to perform simulation processing on the device control command in order to update the virtual scene data.

13. A computer-readable storage medium, characterized in that, It stores computer programs or instructions, which, when executed by a processor, implement the steps in the driving simulation method for intelligent agricultural machinery as described in any one of claims 10 to 12.

14. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program or instructions, which, when executed by the processor, implement the steps in the driving simulation method for intelligent agricultural machinery as described in any one of claims 10 to 12.

Images