Method, device, controller and work machine for automatic control of work machine

Abstract

The application provides a method and device for automatically controlling a working machine, a controller and the working machine. The method comprises: determining the depth of soil penetration, the height from the ground and the anti-collision margin of the working device end according to the acquired multi-modal terrain data and the pose angle; converting the instruction into a text sequence through a language large model; converting the multi-modal terrain data into a visual sequence through a visual large model; determining the action output of the working machine according to the depth of soil penetration, the height from the ground, the anti-collision margin, the text sequence and the visual sequence through a minimalist iterative strategy model; the model performs reasoning through twice fixed forward propagation; and executing the action output. By introducing the explicit geometric state scalar such as the depth of soil penetration, the working machine can accurately grasp the cutting depth, and the task success rate and the full bucket rate are improved; the minimalist iterative strategy replaces the multi-step denoising integration process in the traditional strategy with high time consumption, compresses the response delay, and meets the high-frequency control demand of the hydraulic system.

Description

Technical Field

[0001] This application relates to the field of intelligent control of construction machinery, and in particular to a method, device, controller and construction machinery for automatic control of construction machinery. Background Technology

[0002] The intelligent and unmanned operation of construction machinery has become a key cutting-edge development direction in the industrial field. For example, unmanned excavators have become core equipment in complex scenarios such as earthmoving, open-pit mining, disaster relief, and waste disposal. Their level of automation directly determines the overall operation efficiency, economic benefits, and personnel safety at the construction site.

[0003] Unmanned excavator control typically employs a hierarchical architecture combining perception, decision-making, planning, and execution, integrating sensor sensing, physical models, and optimization algorithms to achieve operational control. However, such solutions based on dynamic models or traditional mechanical equations are highly dependent on accurate prior physical models and complex soil mechanics parameters. Due to the variability of real-world construction environments, these parameters are difficult to acquire and dynamically update in real-time during actual operations. This results in traditional methods having extremely poor environmental adaptability and are prone to failure when encountering unknown terrain or hard obstacles, leading to problems such as dry digging or shallow digging.

[0004] Therefore, how to improve the environmental understanding, real-time control and anti-disturbance capabilities of unmanned excavators under complex dynamic working conditions, and improve the bucket full rate and operating efficiency, has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a method, device, controller, and machine for automatic control of operating machinery, which improves the machine's ability to understand complex scenarios, control real-time performance, and adaptability to disturbances, thereby increasing the bucket fullness and operating efficiency during operation.

[0006] In a first aspect, this application provides a method for automatic control of a work machine, the method comprising:

[0007] Acquire multimodal terrain data, the pose angle of the working device, and commands;

[0008] Based on the multimodal terrain data and the pose angle, determine the soil penetration depth, ground clearance, and rotation anti-collision margin of the working device end;

[0009] The instructions are converted into a text sequence using a large language model;

[0010] The multimodal terrain data is converted into a visual sequence using a large visual model;

[0011] Based on the soil penetration depth, ground clearance, slewing anti-collision margin, text sequence, and visual sequence, the action output of the operating machinery is determined through a simplified iterative strategy model; wherein, the simplified iterative strategy model adopts a regression network with two fixed forward propagations;

[0012] Perform the action and output the result.

[0013] Furthermore, the step of converting the multimodal terrain data into a visual sequence using a large visual model includes:

[0014] Based on the multimodal terrain data, a top-view elevation map is determined;

[0015] Based on the multimodal terrain data, a forward-looking depth map is determined;

[0016] The top-view elevation map, the forward-looking depth map, and the forward-looking RGB image are stitched together along the channel dimension to obtain the RGB-D input tensor;

[0017] The RGB-D input tensor is input into the large visual model to obtain the visual sequence.

[0018] Furthermore, the step of determining the action output of the operating machinery using a simplified iterative strategy model based on the soil penetration depth, the ground clearance, the slewing collision avoidance margin, the text sequence, and the visual sequence includes:

[0019] By concatenating the soil penetration depth, the ground clearance, the rotational anti-collision margin, and the pose angle, a low-dimensional body geometric state vector is obtained.

[0020] The text sequence, the visual sequence, and the low-dimensional ontology geometric state vector are concatenated to obtain a high-dimensional comprehensive context feature vector.

[0021] The high-dimensional comprehensive context feature vector is input into the simplified iterative strategy model to obtain the action output.

[0022] Furthermore, the multimodal terrain data includes three-dimensional point cloud data, and the step of determining the top-view elevation map based on the multimodal terrain data includes:

[0023] Using the rotation center of the operating machinery as the origin of the plane coordinate system, the ground plane is selected as the XY plane, and the effective working area in front of the operating machinery is divided into a two-dimensional grid matrix with a preset resolution.

[0024] The highest Z-axis coordinate value among all point clouds falling within each two-dimensional grid is taken as the gray value of the two-dimensional grid matrix; wherein, the Z-axis is perpendicular to the XY plane.

[0025] Furthermore, the method also includes:

[0026] For two-dimensional graticules without point cloud coverage, a spatial interpolation algorithm is used for smooth filling.

[0027] Furthermore, determining the forward-looking depth map based on the multimodal terrain data includes:

[0028] Based on homogeneous coordinate transformation, the point cloud data is projected onto the two-dimensional imaging pixel plane of the forward-looking camera, and the forward-looking depth map is generated by interpolation.

[0029] Secondly, this application provides an automatic control device for operating machinery, the device comprising:

[0030] The acquisition module is used to acquire multimodal terrain data, the pose angle of the working device, and operator instructions;

[0031] The geometric state module is used to determine the soil penetration depth, ground clearance, and rotational anti-collision margin of the working device end based on the multimodal terrain data and the pose angle.

[0032] The first processing module is used to convert the instructions into a text sequence through a large language model;

[0033] The second processing module is used to convert the multimodal terrain data into a visual sequence through a large visual model;

[0034] The third processing module is used to determine the action output of the operating machinery based on the soil penetration depth, the ground clearance, the slewing anti-collision margin, the text sequence, and the visual sequence through a simplified iterative strategy model; wherein the simplified iterative strategy model adopts a regression network with two fixed forward propagations.

[0035] An execution module is used to execute the action output.

[0036] Thirdly, this application provides a controller, including: a memory and a processor;

[0037] The memory stores computer-executed instructions;

[0038] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any of the first aspects.

[0039] Fourthly, this application provides a work machine, including a work machine body, the work machine body including a work device and a controller, the controller being used to perform the method as described in any of the first aspects.

[0040] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0041] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0042] This application overcomes the depth perception blind spots of traditional pure visual models in single-texture scenes such as loess and sand by fusing global and local elevation maps generated by LiDAR and introducing explicit geometric state scalars such as "penetration depth". This enables excavators to accurately grasp the cutting depth, improve task success rate and bucket full rate, and solve the "empty digging" or "shallow digging" problems common in traditional methods. A simplified iterative strategy is adopted to replace the time-consuming multi-step denoising integration process in traditional flow matching or diffusion models. This architecture requires only two deterministic forward propagations, compressing the end-to-end system response delay and meeting the high-frequency control requirements of hydraulic systems. It enables the lifting and unloading of the boom at millisecond speeds when encountering sudden high cutting resistance, reducing the system pressure buildup shutdown rate. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 Flowchart of the method for automatic control of operating machinery provided in this application Figure 1 ;

[0045] Figure 2 Flowchart of the method for automatic control of operating machinery provided in this application Figure 2 ;

[0046] Figure 3 Flowchart of the method for automatic control of operating machinery provided in this application Figure 3 ;

[0047] Figure 4 A schematic diagram of the structure of the automatic control device for the operating machinery provided in this application;

[0048] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0049] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0051] Existing unmanned excavator control schemes typically begin by using sensors such as LiDAR and cameras to collect terrain geometry and on-site image information. This information is then combined with joint angles, attitude parameters, and position information to estimate the terrain contours, obstacle locations, and the motion state of the working device. Based on this, the system further determines the cutting depth, contact state, and feasible trajectory according to soil mechanics models, equipment kinematic models, or empirical rules. Finally, it uses optimization algorithms to solve for the control quantities of each actuator, and the hydraulic system executes the predetermined actions. While these schemes can achieve a certain degree of automation in structured or relatively unchanging working conditions, their core logic is generally based on prior parameters, static geometric analysis, and hierarchical transmission mechanisms, making them prone to significant shortcomings in complex dynamic environments.

[0052] On the one hand, terrain perception often focuses on geometric point clouds or image processing, and lacks a comprehensive understanding of soil surface features, on-site semantic information and operational intentions. This makes it difficult for the system to accurately identify low-texture surfaces, areas with strong light shading or complex material stockpiling boundaries, resulting in problems such as unseen excavation, shallow excavation or untimely obstacle avoidance during turning.

[0053] On the other hand, when the work object changes from soft soil to hard rock, when the bucket is obstructed causing a sudden change in load, or when temporary obstacles appear in the site, the control link that relies on fixed models and layer-by-layer planning often responds with lag, making it difficult to correct the action output in time, which can easily cause discontinuous trajectory, hydraulic pressure buildup, work interruption, or even equipment collision.

[0054] In addition, traditional solutions often do not directly utilize the key geometric states at the end of the working device. Parameters such as penetration depth, ground clearance, and rotational anti-collision margin are not used as a unified decision-making basis for action generation, resulting in a deviation between the control results and the actual operating state, making it difficult to balance real-time performance, stability, and anti-disturbance capability.

[0055] To address the aforementioned issues, this application proposes a method for automatic control of construction machinery. During operation, multimodal terrain data, the pose angles of the working device, and operator commands are first acquired. Based on the multimodal terrain data and pose angles, the depth of penetration, ground clearance, and slewing collision avoidance margin of the working device's end effector are determined. The commands are then converted into a text sequence using a large-scale language model, and the multimodal terrain data is converted into a visual sequence using a large-scale visual model. Subsequently, based on the penetration depth, ground clearance, slewing collision avoidance margin, text sequence, and visual sequence, a simplified iterative strategy model is used to determine the action output of the construction machinery, and this action output is executed. This technical approach collaboratively utilizes environmental information, geometric state, and operational semantics in a unified decision-making process. It is suitable for deployment in the automatic control systems of construction machinery such as tracked hydraulic excavators, helping to improve the efficiency of action generation, real-time control, and anti-disturbance capabilities in complex scenarios.

[0056] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0057] Figure 1 Flowchart of the method for automatic control of operating machinery provided in this application Figure 1 ,like Figure 1 As shown, it includes the following steps:

[0058] S101: Acquire multimodal terrain data, the pose angle of the working device, and operator instructions.

[0059] In this step, multimodal terrain data is used as input for environmental perception, providing basic data for subsequent determination of soil penetration depth, ground clearance, turning collision avoidance margin, and generation of visual sequences. Specifically, it can include 3D point cloud data, which can be further reconstructed into representations such as top-view elevation maps, forward-looking depth maps, and forward-looking RGB images.

[0060] The working device is the object whose actions are controlled and executed. In the context of a tracked hydraulic excavator, it can include the boom, stick, bucket, and slewing mechanism. Its end-effector geometry directly determines the safety and operational efficiency of the digging, lifting, and slewing processes. The pose angle characterizes the current spatial attitude of the working device, specifically including joint angle information such as the boom angle, stick angle, bucket angle, and top slewing angle, and may also include pitch, roll, and yaw angles output by the inertial attitude unit. Commands carry the operator's task intent, providing semantic constraints for subsequent action generation. These commands can be received by a remote terminal, remote control, dispatching platform, or onboard communication module.

[0061] Specifically, in this embodiment, the executing entity can be an automatic controller mounted on the main body of the operating machinery. The automatic controller may include an industrial computer, an embedded processor, an edge computing unit, or a control platform composed of the aforementioned hardware. The automatic controller is connected to an image acquisition device, a terrain sensing device, an angle acquisition device, an inertial measurement unit, a satellite positioning unit, a hydraulic controller, and a communication device to complete data acquisition and control output in a unified time frame.

[0062] In one possible embodiment, the image acquisition device includes a forward-looking RGB camera and a depth camera, the terrain sensing device includes a lidar, and the angle acquisition device includes an absolute encoder or angle sensor mounted on the articulation points of the boom, stick, and bucket. After the machine enters automatic operation mode, the automatic controller triggers each acquisition unit to operate according to a preset control cycle. The lidar outputs a 3D point cloud of the current working area, the camera outputs the current frame image, the angle sensing unit outputs the current angle values ​​of each joint of the working device, the inertial measurement unit outputs the vehicle's attitude, and the remote communication device receives the operation commands input by the operator.

[0063] In some embodiments, to ensure the accuracy of subsequent geometric calculations and multimodal fusion, the acquired results can be further synchronized in time, aligned in coordinates, and validated for data validity. Time synchronization can employ a unified clock stamping mechanism to bind point clouds, images, angle values, and commands acquired within the same control cycle into the same frame of data. Coordinate alignment can be achieved by converting terrain perception results from different sources to a unified coordinate system based on the vehicle coordinate system or the world coordinate system, using the sensor extrinsic parameter matrix. When a short-term anomaly is detected in a single sensor, the controller can compensate using valid data from the previous cycle and results from adjacent sensors to maintain the continuity of the control link.

[0064] S102: Based on multimodal terrain data and pose angles, determine the soil penetration depth, ground clearance, and rotational anti-collision margin of the working device's end.

[0065] In this step, the penetration depth indicates the depth at which the end of the working device cuts into the ground, controlling the excavation depth; the ground clearance indicates the height of the end of the working device relative to the ground, preventing collisions due to excessive low placement or idle rotation due to excessive height; and the slewing anti-collision margin indicates the safety margin to avoid collisions during slewing. The pose angle, along with the above three quantities, reflects the current geometric state of the working machinery.

[0066] Due to the undulating terrain, irregular material pile boundaries, and dynamic appearance of temporary obstacles at the work site, it is difficult to accurately infer the actual state of the end of the working device by relying solely on joint angles. Therefore, it is necessary to jointly calculate multimodal terrain data and pose angles.

[0067] Specifically, the automatic controller first performs a forward kinematics solution based on the mechanism parameters and pose angles of the working device to obtain the spatial position and attitude of the bucket tooth tip or other selected end reference point in a unified coordinate system. The mechanism parameters may include the boom length, stick length, bucket geometry, relative positions of each hinge point, and installation offset. These parameters can be pre-written into the controller during the equipment calibration stage.

[0068] The controller reconstructs the current surface model based on multimodal terrain data. The surface model can be a rasterized elevation map, a local triangulation model, or a voxel map. In one specific implementation, the 3D point cloud can first be segmented into ground points, outliers removed, and interpolated for completion. Then, a top-down elevation map centered on the area in front of the working machinery is generated, and the elevation map is corrected for local errors at steep slope edges and obstructed areas by combining it with a forward-looking depth map. The penetration depth can be calculated by the distance of the end reference point relative to the local ground surface along the direction of gravity. When the reference point is below the ground surface, it is defined as a negative value to reflect the actual cutting depth. The ground clearance can be determined by the minimum vertical distance between the end reference point and the ground surface or the upper surface of an obstacle, to characterize the lifting clearance. The slewing collision avoidance margin can be obtained by scanning the distances of obstacles, material pile boundaries, and adjacent structures of the vehicle body within a predetermined slewing sector based on the current upper vehicle slewing center and the envelope model of the working device, to obtain the minimum safe interval along the potential slewing path.

[0069] In one possible embodiment, the calculation of the turning collision avoidance margin considers not only static terrain but also the predicted location of dynamic obstacles. The controller can detect the speed direction of nearby vehicles, personnel, or mobile devices based on multiple consecutive frames of point cloud or image data, and expand their occupied area within a short-term prediction window, thereby obtaining a safety margin that better reflects actual operating conditions. To adapt to environmental changes within the control cycle, the penetration depth, ground clearance, and turning collision avoidance margin are updated in real time, with the update frequency consistent with the control cycle.

[0070] S103: Convert instructions into text sequences using a large language model.

[0071] In this step, instructions are used to express the operator's work objectives, work area, work method, and work constraints, such as specifying the digging location, loading object, unloading direction, leveling requirements, or safety restrictions. The language model is used to convert the operator's input instructions into a machine-processable text sequence, mapping natural language or structured control statements to semantically related expressions. The text sequence represents the semantics of the instructions and, together with the visual sequence and low-dimensional geometric state, constitutes the comprehensive contextual information required for subsequent action decisions.

[0072] Specifically, after receiving an instruction, the automatic controller or its connected edge inference unit first standardizes the input. If the operator inputs a natural language voice instruction, it can be converted into text by a speech recognition module; if the input is a structured task message issued by the scheduling system, the field information can be expanded into standardized descriptive statements. The standardized instruction is then input into a language model, which can be a locally deployed lightweight model, a cloud-edge collaborative model, or a domain-fine-tuned dedicated semantic encoding model. This model performs semantic parsing on the action words, target objects, region descriptions, directional constraints, order requirements, and security constraints in the instruction, and generates a text token sequence based on a preset vocabulary or word segmentation strategy.

[0073] For example, when the instruction is "dig the left side of the material pile in front and turn around to the right rear to unload the truck", the language big model can output a text sequence containing semantic units such as "dig target area", "left side material pile", "turning direction" and "right rear unloading point" so that the subsequent model can understand the current task intent and action bias.

[0074] To improve the matching degree between semantic information and the control system, length pruning, key field enhancement, and task tag appending can be performed after the text sequence is generated. Length pruning is used to ensure stable inference latency and avoid excessively long inputs consuming computing resources; key field enhancement can assign higher weights to key information such as work objects, dangerous areas, and restricted areas; task tag appending can add work mode identifiers to the front of the text sequence, such as digging mode, loading mode, unloading mode, or leveling mode.

[0075] When instructions are ambiguous, such as "dig forward a little" without specifying the depth or direction boundaries, the system can perform semantic completion based on the current working conditions, the most recent confirmed task, and safety rules, or encode the ambiguous information as an uncertainty marker and pass it into the subsequent strategy model, so that the model tends to output conservatively when generating actions.

[0076] S104: Convert multimodal terrain data into visual sequences using a large visual model.

[0077] In this step, the large visual model is used to extract visual features and sequentially represent multimodal terrain data. The visual sequence is used to characterize visual, spatial, and terrain structure information in the environment. In this embodiment, the multimodal terrain data includes not only the original 3D point cloud but also a top-down elevation map, a forward-looking depth map, and a forward-looking RGB image constructed from point cloud and image fusion. By uniformly inputting multiple representation forms into the large visual model, the model can simultaneously perceive terrain undulations, obstacle distribution, material pile boundaries, surface textures, and local occlusion relationships, providing richer contextual information for subsequent action outputs than a single geometric input.

[0078] Specifically, after completing synchronization and coordinate alignment in S101, the automatic controller converts the multimodal terrain data at the target time into an input tensor suitable for visual model processing. In one specific implementation, the top-view elevation map, the forward-looking depth map, and the forward-looking RGB image can be stitched together according to the channel dimension to form a unified RGB-D input tensor. Among them, the top-view elevation map mainly reflects the terrain undulations and material distribution within the working area, the forward-looking depth map mainly reflects the near-field spatial structure and obstacle distances in front of the working device, and the forward-looking RGB image mainly reflects surface color, texture, shadows, and semantic cues.

[0079] Large-scale visual models can employ visual Transformer networks, hybrid convolutional and Transformer networks, or multimodal coding networks trained specifically for engineering machinery scenarios. After performing block embedding, positional encoding, and multi-layer feature extraction on the input tensor, the model outputs a sequence of visual tokens with a sequential structure. Each element in this visual sequence corresponds to different spatial regions or features at different scales in the input scene, thus simultaneously preserving both global terrain structure and local obstacle details.

[0080] In one possible embodiment, to improve robustness in complex lighting and dusty environments, image enhancement and data correction processing can be added before the large visual model is input.

[0081] In another possible embodiment, the visual big model can also fuse data from multiple consecutive frames and extract obstacle movement trends and terrain change trends through temporal coding, so that the visual sequence reflects not only the static scene but also the short-term dynamic environmental features.

[0082] Converting multimodal terrain data into visual sequences can overcome the limitations of traditional control schemes that focus on static geometric analysis, and enhance the system's comprehensive understanding of low-texture surfaces, complex material boundaries, and occluded areas. On the other hand, visual sequences, in the form of compressed high-dimensional semantic features, can participate in subsequent reasoning, which can control the input scale while ensuring information richness.

[0083] S105: Based on the soil penetration depth, ground clearance, slewing collision avoidance margin, text sequence, and visual sequence, the action output of the operating machinery is determined through a minimal iterative strategy model; wherein, the minimal iterative strategy model adopts a regression network with two fixed forward propagations.

[0084] In this step, the simplified iterative strategy model is the core decision model of the control strategy. It is used to determine the action output of the operating machinery based on the low-dimensional ontological geometry state and the high-dimensional semantics and visual context. The action output is the final control result sent to the operating machinery for execution, which can be expressed as the target displacement increment, target velocity, target angular velocity, hydraulic valve opening command, or a combination thereof for the boom, stick, bucket, and slewing mechanism within the current control cycle.

[0085] The embedment depth, ground clearance, and turning collision avoidance margin are used as explicit geometric constraints. Text sequences provide task semantic constraints, while visual sequences provide environmental visual semantics and spatial structural constraints. The minimalist iterative strategy model performs inference through two fixed forward propagations. This means it does not rely on multiple rounds of sampling or an indefinite number of optimization iterations, but rather completes action generation and correction within two fixed forward computations. This reduces inference latency and meets the real-time control requirements of engineering machinery. The first forward propagation inference yields a noise-free initial blind guess, while the second forward propagation inference yields noisy, refined residual data.

[0086] Specifically, the controller first combines the soil penetration depth, ground clearance, and swerve collision avoidance margin into a low-dimensional ontological geometric state vector. It can also add current pose angle, hydraulic pressure, vehicle attitude, and historical action summaries to form a structured state input. Simultaneously, the text sequence output from S103 and the visual sequence output from S104 are input into the cross-modal fusion module, where an attention mechanism or feature concatenation mechanism is used to obtain a high-dimensional comprehensive contextual feature vector.

[0087] Subsequently, the simplified iterative strategy model performs its first fixed forward propagation, generating initial action results based on the geometric state vector and the comprehensive context feature vector. This initial action result focuses on coarse-grained determination of the current action trend to be taken, such as continuing to press down and cut in, lifting and disengaging, decelerating and turning, stopping the approach to the obstacle, or adjusting the bucket posture. Afterward, the initial action result, along with the original input, is fed back into the same model or a correction subnetwork sharing its parameters, performing a second fixed forward propagation to refine the initial action result.

[0088] Fine-grained corrections can include reducing excessive downward pressure, increasing lift when ground clearance is insufficient, reducing turning speed when the turning collision avoidance margin is too small, and increasing the range of motion that meets safety requirements when the text sequence explicitly requires rapid loading.

[0089] In one specific implementation, the action output can be represented as a vector A = [θ1, θ2, θ3, ωr], where θ1, θ2, and θ3 correspond to the target angles of the boom, stick, and bucket, respectively, and ωr corresponds to the target angle of the slewing mechanism. The purpose of this representation is to decompose complex operational actions into multiple control quantities that can be directly mapped to the actuators, enabling the strategy model to simultaneously constrain digging depth, ground clearance, and slewing safety within a unified space.

[0090] In some embodiments, if the penetration depth is denoted as d, the height above ground is denoted as h, and the turning collision avoidance margin is denoted as m, the model can use a built-in constraint mechanism during training or deployment to ensure that when d exceeds a predetermined threshold, the downward pressure is suppressed, when h is below a safety threshold, the lifting component is increased, and when m decreases, the absolute value of ωr is reduced, thereby enabling the geometric state to directly modulate the action output.

[0091] The technical significance of the two fixed forward propagations in this step lies in the fact that the first propagation quickly forms an executable action skeleton, while the second propagation, while preserving the action trend, incorporates current working condition details to complete the correction, avoiding the response lag caused by traditional layer-by-layer planning and multi-round optimization. The simplified iterative reasoning mechanism using two fixed forward propagations can balance action generation efficiency, real-time control, and safety stability under complex dynamic working conditions, thereby alleviating problems such as dry digging, shallow digging, hydraulic pressure buildup, trajectory discontinuity, and untimely obstacle avoidance during slewing.

[0092] S106: Execute action output.

[0093] Specifically, the automatic controller sends the motion output generated by S105 to the motion control unit of the operating machine, and the motion control unit maps the control quantity according to the underlying execution architecture of the equipment.

[0094] In one possible embodiment, after executing the action output, the system also records the current action result, actual trajectory, load changes, and environmental changes for subsequent online diagnosis, task evaluation, or model iteration updates. For example, when the actual soil penetration depth is detected to be significantly less than the target soil penetration depth and the bucket load is low for several consecutive control cycles, it can be determined that there is a tendency for empty digging, and the cutting angle and cutting position can be automatically adjusted through the strategy model in subsequent cycles; when the actual anti-collision margin shrinks rapidly during the slewing execution, the slewing speed can be converged in real time and the slewing can be terminated in advance.

[0095] In this embodiment, by incorporating multimodal terrain perception results, key geometric states at the end of the working device, and operator task intent into a unified decision-making chain, and utilizing a large language model and a large visual model to complete semantic and visual encoding respectively, and then generating action output using a minimalist iterative strategy model employing two fixed forward propagations, the operating machinery can maintain a high control response speed and action stability even under complex working conditions with significant terrain undulations, frequent changes in material properties, and dynamic appearance of obstacles. Furthermore, the soil penetration depth, ground clearance, and slewing collision avoidance margin, as explicit geometric variables that directly reflect the actual performance and safety boundaries of the operation, can provide immediate constraints on action generation; text sequences and visual sequences supplement the shortcomings of traditional geometric control from the perspectives of task semantics and environmental understanding, respectively; and the two fixed forward propagation mechanisms reduce reasoning complexity while completing coarse generation and fine correction of actions, thereby effectively improving problems such as empty digging, shallow digging, lag in slewing collision avoidance, trajectory discontinuity, and insufficient anti-disturbance capability in existing technologies.

[0096] Example 2

[0097] Based on the foregoing embodiments, the following embodiment describes the process of converting multimodal terrain data into a visual sequence through a large visual model.

[0098] Figure 2 Flowchart of the method for automatic control of operating machinery provided in this application Figure 2 ,like Figure 2 As shown, it includes the following steps:

[0099] S1041. Determine the top elevation map based on multimodal terrain data.

[0100] Among them, the top-view elevation map is used to represent the height of the terrain undulations in the work area from a top-down perspective.

[0101] In one specific implementation, the effective working area ahead is divided into a two-dimensional grid matrix, with the rotation center of the working machinery as the origin and the ground plane as the XY plane. The highest Z-axis coordinate value of the point cloud data in each grid is then mapped to the corresponding grayscale value, thereby forming an elevation representation that can be recognized by a large visual model. Since this elevation map is constructed based on the rotation center, its spatial representation is consistent with the machinery's steering, rotation, and forward working range, which is beneficial for uniformly referencing the terrain condition and the machine's attitude when making action decisions.

[0102] The top-down elevation map transforms point cloud data into a two-dimensional height distribution consistent with the machine's coordinate system. This allows the machinery to directly perceive the terrain's undulations, accumulation boundaries, and local protrusions, providing structured spatial input for subsequent large-scale visual models. Based on rasterization and the use of the highest point for data processing, the fidelity of the actual terrain contours is enhanced, noise interference is reduced, and the accuracy of subsequent visual sequence generation is improved. This, in turn, helps improve the machinery's environmental perception, real-time control, and disturbance rejection capabilities during automatic control.

[0103] For two-dimensional rasters without point cloud coverage, a spatial interpolation algorithm is used for smooth filling. The spatial interpolation algorithm is used to continuously estimate the missing elevation based on the spatial distribution relationship of neighboring known rasters, so that the raster matrix maintains a coherent representation of the terrain surface.

[0104] In the specific implementation, after the top-down elevation map is generated, it is first checked whether each two-dimensional raster has corresponding point cloud coverage. When a raster does not obtain a valid grayscale value, it is marked as a raster to be filled, and the grayscale values ​​of known rasteres within a preset neighborhood range around the raster are extracted. The spatial interpolation algorithm can use any of the following: inverse distance weighted interpolation, bilinear interpolation, cubic convolution interpolation, or kriging interpolation, or a combination of interpolation based on terrain undulation and point cloud density. By continuously estimating the known grayscale values ​​in the neighborhood, the filling value of the target raster is obtained, and this filling value is written into the two-dimensional raster matrix, thereby completing the smooth compensation for the missing area.

[0105] After the infilling is complete, the 2D raster matrix continues to participate in the subsequent construction of the top-view elevation map. Because the infilled result has a continuous spatial variation relationship with the neighboring raster, it can mitigate abrupt elevation changes caused by local gaps, making the input tensor more geometrically complete and consistent. Therefore, the large-scale visual model can obtain more stable terrain context information when generating visual sequences, reducing feature bias caused by missing point clouds.

[0106] S1042. Determine the forward depth map based on multimodal terrain data.

[0107] The forward-looking depth map is used to characterize the distance distribution in the space ahead of the work direction. It is obtained by coordinate projection of the depth information corresponding to the forward viewpoint in the multimodal terrain data, and then by interpolation to form a continuous image to supplement the information of obstacles in the space ahead that are difficult to be directly reflected from the top-down view.

[0108] S1043. The top-view elevation map, the front-view depth map, and the front-view RGB image are stitched together by channel dimension to obtain the RGB-D input tensor.

[0109] The RGB-D input tensor is used as a unified input carrier for large visual models. It is formed by stitching together the top-view elevation map, the front-view depth map, and the front-view RGB image in the channel dimension, so that the model can simultaneously receive geometric information, distance information, and texture information.

[0110] S1044. Input the RGB-D input tensor into the large visual model to obtain the visual sequence.

[0111] The visual big model is used to extract features and serialize the fused input, and output a visual sequence with temporal or spatial order representation for subsequent action decision-making.

[0112] In this specific implementation, the multimodal terrain data can consist of lidar point clouds, forward-looking camera images, and depth data related to the work direction. After acquisition, time synchronization and coordinate system unification are performed first, and then the point cloud and image are mapped to the same spatial reference system based on the extrinsic parameter calibration relationship. When determining the top-view elevation map, a grayscale matrix can be generated based on the highest point height of the area covered by each grid in the point cloud, and the grayscale range is normalized to a numerical range acceptable to the visual model.

[0113] When determining the forward-looking depth map, the point cloud can be projected onto the pixel plane of the forward-looking camera through homogeneous coordinate transformation, and neighborhood interpolation or bilinear completion can be performed on the hole regions to improve the continuity of the depth map.

[0114] Subsequently, the top-view elevation map, forward-looking depth map, and forward-looking RGB image are concatenated into a multi-channel tensor along the channel dimension and then input into the large-scale visual model. The large-scale visual model can employ a combination structure of a visual encoder and a serialization mapping module to extract fused features hierarchically and generate visual sequences. This large-scale visual model can use a visual backbone network with multi-channel input capability to preserve the correlation information between different modalities during the same encoding process. In practical applications, other models of this component can also be selected, and this application embodiment does not limit this.

[0115] The above-described method can uniformly convert the top-down geometric undulations, forward spatial depth, and on-site RGB semantic texture of the work area into a visual sequence, enabling the large visual model to obtain a more complete environmental context. This improves the ability to recognize complex terrain, temporary obstacles, and boundary areas, and provides a stable visual basis for subsequent action output. Since the elevation map and depth map supplement the terrain representation from different perspectives, their stitching enhances the model's understanding of low-texture, heavily occluded, and unstructured scenes, thereby improving the real-time performance, robustness, and collision avoidance capabilities of the automated control of the machinery.

[0116] Example 3

[0117] Building upon any of the aforementioned embodiments, the input to the simplified iterative strategy model is further subjected to feature concatenation. Specifically, the penetration depth, ground clearance, turning collision avoidance margin, and pose angle are concatenated to obtain a low-dimensional ontological geometric state vector; the text sequence, visual sequence, and low-dimensional ontological geometric state vector are concatenated to obtain a high-dimensional comprehensive contextual feature vector; and the high-dimensional comprehensive contextual feature vector is input into the simplified iterative strategy model to obtain the action output.

[0118] The working mechanism of this method lies in simultaneously feeding the end-effector geometric constraints, task semantics, and environmental visual information into the same decision model, avoiding control deviations caused by relying solely on a single geometric analysis or static rules. Since the action output is directly constrained by the penetration depth, ground clearance, and turning collision avoidance margin, it can maintain stable penetration in complex scenarios, avoid ground scraping or collisions, and improve the responsiveness to load changes and environmental disturbances.

[0119] After adopting this specific implementation method, the matching degree between the action output of the operating machinery and the current operating state is improved, the control response is more timely, the trajectory continuity is better, and the probability of problems such as empty digging, shallow digging, slewing interference and hydraulic pressure buildup can be reduced, thereby improving the safety, stability and execution efficiency of automatic operation in complex unstructured scenarios.

[0120] Example 4

[0121] Based on the foregoing embodiments, the generation of the forward depth map will be specifically described below with an example.

[0122] Based on homogeneous coordinate transformation, point cloud data is projected onto the two-dimensional imaging pixel plane of the forward-looking camera, and then interpolated to generate a forward-looking depth map.

[0123] The forward-looking depth map is used to convert the spatial distance relationships of a 3D point cloud into a 2D depth representation consistent with the forward-looking viewpoint. The 2D imaging pixel plane of the forward-looking camera corresponds to the observation direction in front of the working machinery, and can directly reflect the changes in the distance to terrain undulations, pothole boundaries, and obstacles. Homogeneous coordinate transformation is used to transform the point cloud from the terrain coordinate system or vehicle coordinate system to the forward-looking camera coordinate system, so that point clouds at different spatial locations can be mapped to the same imaging plane according to the camera's intrinsic and extrinsic parameters, thereby forming a depth distribution with consistent viewpoint. The point cloud data comes from LiDAR or multi-sensor fusion results. After coordinate unification, an initial depth map can be generated based on the depth value and projection position of each point in the forward-looking camera coordinate system.

[0124] In its implementation, the system first establishes a homogeneous transformation matrix between the point cloud coordinate system and the camera coordinate system based on the extrinsic parameters of the forward-looking camera. Then, it combines the intrinsic parameters of the camera to complete the mapping from 3D points to 2D pixel coordinates, obtaining the pixel position and corresponding depth value. Due to the discrete sampling of the point cloud, occlusion relationships, and missing field-of-view edges, holes or sparse regions may appear on the projected pixel plane. Therefore, interpolation is needed to complete the missing pixels. Interpolation can be performed using bilinear, bicubic, or distance-weighted depth estimation based on the effective depth values ​​in the neighborhood to maintain the continuity of the depth map and suppress the influence of local outliers on the visual input.

[0125] The forward-looking depth map and the forward-looking RGB image together represent the geometric and appearance information of the terrain ahead. In practical applications, the forward-looking camera can also be configured with different focal lengths and resolutions, which is not limited in this embodiment.

[0126] After adopting the above method, the 3D point cloud is converted into a forward-looking depth expression consistent with the working direction, which can improve the perception accuracy of the spatial structure of the terrain ahead, reduce the depth ambiguity caused by relying solely on image texture, enhance the ability of the operating machinery to identify potholes, material stack boundaries and obstacles in complex scenarios, and enhance the real-time performance and safety of action decisions.

[0127] The following section uses a tracked hydraulic excavator as an example to provide a detailed introduction to this technical solution with specific examples.

[0128] Figure 3 Flowchart of the method for automatic control of operating machinery provided in this application Figure 3 , refer to Figure 3 It includes the following steps:

[0129] 1. Hardware system deployment and sensor spatiotemporal calibration configuration.

[0130] (1) Dual-view vision sensor group: Two high-definition RGB cameras with a large field of view are installed on the top front edge of the excavator cab. They are responsible for capturing global workspace images including the boom, stick, and open terrain in front, as well as focusing on the bucket and its adjacent local cutting surface, providing details of soil texture and contact status at the moment of excavation.

[0131] (2) Three-dimensional geometric perception radar: A high-resolution lidar is installed at the top front of the excavator cab to obtain a high-precision dense point cloud of the surrounding environment. In order to meet the response cycle of the underlying hydraulic actuator, the scanning output frequency is set to 10Hz.

[0132] (3) Multi-source fusion body state sensor: In order to calculate the excavator's pose in real time, high-precision inertial measurement units (IMUs) with strict protection are installed on the boom, stick, bucket linkage mechanism and upper slewing body of the excavator. In addition, a real-time dynamic differential global navigation satellite system (GNSS) antenna module is configured on the top of the vehicle body to obtain the absolute three-dimensional positioning and high-precision heading angle of the excavator base in the global coordinate system.

[0133] In the hardware data acquisition process, all sensor data streams (including image frames, point cloud packets, IMU attitude angles and RTK coordinates) are strictly time-stamped at the hardware level based on the 10Hz hardware pulse frequency of LiDAR, and are uniformly converted to the main vehicle coordinate system with the excavator's slewing center as the origin through a multi-sensor joint calibration algorithm, thereby completely eliminating motion distortion caused by data asynchrony.

[0134] 2. Dimensionality reduction generation and fusion preprocessing of multimodal terrain representation data.

[0135] Due to its disordered nature, raw radar 3D point cloud data is difficult to extract effective local topological features when directly fed into the model. Therefore, the data preprocessing module of this system transforms the point cloud into a structured 2.5D image format through coordinate projection and spatial interpolation algorithms.

[0136] (1) Construction of the global elevation map from above: Extract the real-time point cloud of the current frame of the lidar after synchronization. Taking the excavator's slewing center as the origin of the plane coordinate system, select the XY plane parallel to the track ground plane and divide the effective working area in front of the excavator into a two-dimensional grid matrix with a fixed resolution (e.g., 0.1m × 0.1m). Traverse the point cloud data and extract the highest coordinate value of the Z-axis (perpendicular to the ground upwards) of all points falling into each grid as the gray value of that pixel. For blind grids without point cloud coverage, a spatial interpolation algorithm based on the neighborhood mean is used for smooth filling. This global elevation map provides a macroscopic topological overview of the terrain slope, soil pile volume, and static obstacle distribution for subsequent model planning of cross-regional slewing and unloading trajectories.

[0137] (2) Projection and generation of dense depth map in front view: The 3D point cloud is projected onto the two-dimensional imaging pixel plane of the front view camera using homogeneous coordinate transformation, and a dense depth map is generated by interpolation. This process enables the Visual Language Model (VLM) to achieve pixel-level alignment between visual texture (such as soil material) and spatial geometry (such as slope and distance).

[0138] (3) Multimodal feature channel stitching: The generated top-down global elevation map and forward-looking dense depth map are used as additional channels and directly hard-stitched with the RGB image tensor output by the forward-looking camera in the channel dimension, thereby forming a high-dimensional RGB-D format input tensor. This enables the subsequent visual encoder to capture pixel-level visual material texture and precise spatial geometric absolute scale within a single receptive field.

[0139] 3. Real-time forward solution and feature injection of explicit geometric state scalars

[0140] The "forward kinematics and geometry solver" running inside the system rapidly executes the following purely physical and mathematical calculation process within each 10Hz sensing cycle:

[0141] First, the system reads tilt sensor data distributed on the boom, stick, and bucket to obtain the current absolute attitude angles of each working device, and converts them into relative joint variables between adjacent links through geometric calculation. Combining the known fixed physical dimensions of each structural component of the excavator, the system constructs a link coordinate system using the classic Denavit-Hartenberg (DH) parametric method. Through continuous homogeneous transformation matrix multiplication, it forward calculates the three-dimensional spatial coordinates (X, Y, F, Z) of the bucket's foremost tooth tip in the excavator's base coordinate system at the current moment. tip Y tip Z tip ).

[0142] Subsequently, the system calculates the horizontal coordinates (X... tip Y tip The absolute elevation Z of the terrain surface corresponding to the coordinate is directly indexed from the generated top-down global elevation map. terrain Based on this, the system calculates and outputs the following three key explicit geometric scalars that determine the excavator's behavior patterns in real time:

[0143] (1) Scalar of relative height / depth of penetration Z rel Its mathematical definition is Z rel = Z tip - Z terrain .

[0144] When the solution result Z rel When the value is > 0, it indicates that the bucket is currently in a state of hovering or moving in mid-air. After receiving this feature, the model tends to output a sequence of actions to continue lowering the boom to get closer to the ground.

[0145] When the solution result Z rel= 0 represents the instant the bucket teeth just touch the soil surface. This critical signal acts as a strong state trigger, inducing the model to smoothly switch from a spatial free trajectory planning mode to a high-impedance contact digging mode (e.g., initiating a combined action of retracting the bucket and the inward retraction of the boom).

[0146] When the solution result Z rel A value less than 0 indicates that the bucket has deeply penetrated the underground soil layer. This scalar is the core input variable for overcoming dry excavation and high-frequency pressure buildup. The absolute value of its negative value directly and accurately reflects the current cutting depth. Since the cutting depth is strongly positively correlated with the expected soil resistance in physics, if this negative value is too large, the network can use this explicit prior to quickly predict the need to perform a boom lifting action for dynamic unloading and drag reduction before substantial hydraulic overflow jamming occurs.

[0147] (2) Ground height scalar Z gnd Defined as the absolute height distance between the tooth tip and the plane where the excavator track contacts the parking surface. This feature is forcibly injected into the network for chassis collision avoidance constraints, preventing the model from misjudging the distance and impacting the track structure when retracting the bucket, and also guiding the safe lifting height of the fully loaded bucket after excavation is completed.

[0148] (3) Slewing anti-collision margin Z swing Defined as the height of the tooth tip relative to the excavator's rotation center. This ensures sufficient safety clearance at the bottom of the bucket during rotation to prevent scraping against protruding obstacles in the surrounding unknown terrain.

[0149] The three scalars obtained from the solution are first encoded using a multilayer perceptron (MLP) for feature upsizing, and then directly concatenated with the current low-dimensional joint angle vector of the excavator to form a complete low-dimensional ontological geometric state vector o that is rich in physical constraints. prop .

[0150] 4. Cross-modal feature fusion extraction based on Visual-Language Encoder (VLM Encoder)

[0151] Once the sensory data is ready, the system's sensory center begins operation. This center is based on a lightweight visual-language model architecture and has been fine-tuned specifically for efficiently handling data mining scenarios.

[0152] 1. High-dimensional extraction of visual features: The system feeds a dataset containing RGB images, forward-looking RGB-D tensors, and top-down elevation maps into a pre-trained SigLIP visual encoder network. To enable visual encoders, which are traditionally trained only on natural images, to understand the specific geometric and topological semantics contained in the elevation maps, this system deliberately unfreezes the adapter weights of the last few specific layers of the visual encoder during the fine-tuning training phase. After forward propagation, the encoder outputs a highly condensed sequence of visual semantic tokens.

[0153] 2. Cross-modal context fusion: Operator-issued natural language instructions (such as "level the soil area ahead" or "dig up stones and load them onto the truck") are converted into a text token sequence by a lightweight LLM tokenizer. Subsequently, the system concatenates the visual token sequence, the text token sequence, and the previously generated ontology state vector o(prop) containing strong geometric constraints in all dimensions, and feeds them into the language decoder.

[0154] 3. Inference optimization mechanism for edge computing power: Considering the bandwidth bottleneck of the in-vehicle platform, this central module deploys a skip-layer attention mechanism and a minimalist visual token downsampling strategy without losing core interaction features. This significantly reduces unnecessary background region computation overhead during the forward propagation stage, ultimately outputting a high-dimensional comprehensive context feature condition vector C that includes both spatial three-dimensional geometric understanding and operational intent, which is then passed to the action expert module.

[0155] 5. Network Training and Ultra-Low Latency Closed-Loop Inference Based on Minimalist Iterative Strategy (MIP) Expert Modules

[0156] This module is the core architecture that completely replaces the high-latency flow matching model and diffusion strategy. It is specifically responsible for generating the action sequence for several future control cycles based on the context feature C, so as to achieve continuous and smooth coordinated control of the four major joints: boom, stick, bucket and slewing.

[0157] (1) Training design and loss function construction of MIP network:

[0158] The theoretical foundation of the Minimalist Iterative Process (MIP) lies in abandoning precise fitting of high-dimensional continuous complex distributions and instead forcing the network to learn error-correcting residuals by artificially introducing state perturbations. Assuming a real, perfect action sequence from a dataset of experienced human operators is denoted as 'a', the system samples equal-dimensional random Gaussian noise z~N(0,1) from a standard normal distribution during each training iteration. A fixed noise level (or time step) hyperparameter t* is defined. The system constructs an intermediate, deviated state I, formed by the real expert's actions being strongly disturbed by noise. t*Its linear interpolation formula is defined as:

[0159]

[0160] The parameters θ of the MIP expert network are trained by minimizing the following joint loss function L. MIP This is achieved by including a regression mean squared error (MSE) term that supervises the two-step prediction separately:

[0161]

[0162] The first term on the right side of the equation (the direct prediction term) constrains the network π. θ Without any initial noise input (i.e., time step t=0 and input state I0=0), the network can directly "blindly guess" the real action a based solely on the contextual features C; the second term on the right side of the equation (the denoising correction term) strictly requires the network to even when receiving a noisy state I that is significantly off track. t* Even so, it can still accurately calculate the corrected gradient and forcibly recover the original expert trajectory 'a'. This training mechanism, which combines "random injection" and "supervised iteration", endows the neural network with extremely strong manifold adhesion capabilities. When the bucket hits a rock in real-world operations, causing the physical action to deviate from expectations (i.e., generating an external disturbance offset similar to z), the network can immediately use its learned denoising capabilities to spontaneously pull the trajectory back to a safe range.

[0163] (2) "Two-step deterministic iteration (Guess-and-Refine)" ultra-low latency inference mechanism:

[0164] During the real-time inference phase deployed on the excavator, the MIP module completely abandons the time-consuming random sampling and lengthy integration processes of diffusion models or flow matching. The model network performs only two extremely straightforward fixed forward propagations:

[0165] Step 1 (Guess Preliminary Exploration and Prediction): Set the input time step variable t to 0, and set the input initial motion tensor to a noise-free matrix of all zeros. Based on the high-dimensional environmental features C input from the VLM, the network quickly outputs a preliminary coarse prediction of the future motion trajectory (including the angles of joints such as the boom and stick):

[0166]

[0167] Step 2 (Refine Residual Polishing and Smoothing): The coarse predicted trajectory output from Step 1 is scaled proportionally according to the set hyperparameter time step t*, and this scaled-up state is fed back into the network as a new input state. Simultaneously, the network's time step conditional input is set to t*. The network, combined with deep physical geometry features, performs fine-grained polishing of the motion trajectory, outputting a smooth motion sequence block that can be directly deployed.

[0168]

[0169] This application abandons pure RGB input and utilizes point cloud projection reconstruction to construct a top-down global elevation map and a forward-looking depth map. These maps are then fused and stitched together with RGB color images and natural language text commands at the token level, serving as the unified perception central input for the policy network. This multimodal design directly solves the serious depth misjudgment problem of vision-based methods in single low-texture scenes such as loess and sand, or in low-light environments at dusk. This enables excavators to maintain an extremely stable and efficient operating rhythm in all weather conditions (unaffected by strong backlight or low light).

[0170] This application utilizes forward kinematics matrix and elevation terrain data in real time to calculate the absolute physical error parameters of the bucket tooth tip relative to the terrain directly below. These parameters are then directly injected as a robust prior condition vector into the multi-layer feature stitching layer of the expert action generation network. The introduction of explicit scalars directly establishes a "fast error correction channel" from the physical space state to the network's hidden layer parameters. This significantly improves the bucket fullness rate and operational efficiency while maintaining absolute system stability.

[0171] This application abandons the multi-step ordinary differential equation (ODE) integration and denoised random sampling of the flow matching algorithm. Instead, it employs a second-order regression network with a training loss function combining "real motion and Gaussian noise linear interpolation" and an ultra-low latency inference mechanism of "two-step deterministic forward propagation" including noiseless Guess and scaled-noise-added Refine, as the final trajectory decoder. This halves the system inference computation time, making stable and precise 50Hz ultra-high frequency dynamic control of the excavator's hydraulic valve control system a practical engineering reality.

[0172] Figure 4 A schematic diagram of the structure of the automatic control device for the operating machinery provided in this application is shown below. Figure 4 As shown, the automatic control device 40 for operating machinery provided in this embodiment includes:

[0173] The acquisition module 401 is used to acquire multimodal terrain data, the pose angle of the working device, and the operator's instructions;

[0174] The geometry module 402 is used to determine the soil penetration depth, ground clearance, and rotational anti-collision margin of the working device end based on multimodal terrain data and pose angle.

[0175] The first processing module 403 is used to convert instructions into text sequences through a large language model;

[0176] The second processing module 404 is used to convert multimodal terrain data into a visual sequence through a large visual model;

[0177] The third processing module 405 is used to determine the action output of the operating machinery based on the soil penetration depth, ground clearance, slewing anti-collision margin, text sequence, and visual sequence through a simplified iterative strategy model; wherein, the simplified iterative strategy model adopts a regression network with two fixed forward propagations.

[0178] Execution module 406 is used to execute action output.

[0179] Furthermore, the second processing module 404 is specifically used for:

[0180] Determine the top-view elevation map based on multimodal terrain data;

[0181] Determine the forward depth map based on multimodal terrain data;

[0182] The top-view elevation map, the front-view depth map, and the front-view RGB image are stitched together along the channel dimension to obtain the RGB-D input tensor;

[0183] The RGB-D input tensor is input into the large visual model to obtain the visual sequence.

[0184] Furthermore, the third processing module 405 is specifically used for:

[0185] By splicing together the burial depth, ground clearance, rotational collision avoidance margin, and pose angle, a low-dimensional ontological geometric state vector is obtained.

[0186] The text sequence, visual sequence, and low-dimensional ontology geometric state vector are concatenated to obtain a high-dimensional comprehensive context feature vector;

[0187] The high-dimensional integrated context feature vector is input into the minimal iterative strategy model to obtain the action output.

[0188] Furthermore, the second processing module 404 is specifically used for:

[0189] Using the rotation center of the operating machinery as the origin of the plane coordinate system, the ground plane is selected as the XY plane, and the effective working area in front of the operating machinery is divided into a two-dimensional grid matrix with a preset resolution.

[0190] The highest Z-axis coordinate value among all point clouds falling within each 2D grid is taken as the gray value of the 2D grid matrix; where the Z-axis is perpendicular to the XY plane.

[0191] For two-dimensional graticules without point cloud coverage, a spatial interpolation algorithm is used for smooth filling.

[0192] Furthermore, the second processing module 404 is specifically used for:

[0193] Based on homogeneous coordinate transformation, point cloud data is projected onto the two-dimensional imaging pixel plane of the forward-looking camera, and then interpolated to generate a forward-looking depth map.

[0194] The apparatus provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0195] Figure 5 This is a schematic diagram of the electronic device provided in this application. The electronic device can be a controller, such as… Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0196] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0197] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0198] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0199] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0200] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0201] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0202] This application also provides a work machine, including a work machine body, the work machine body including a work device and a controller, the controller being used to perform the method as described in any of the foregoing claims.

[0203] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0204] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0205] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0206] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0209] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0211] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for automatic control of operating machinery, characterized in that, The method includes: Acquire multimodal terrain data, the pose angle of the working device, and operator instructions; Based on the multimodal terrain data and the pose angle, determine the soil penetration depth, ground clearance, and rotation anti-collision margin of the working device end; The instructions are converted into a text sequence using a large language model; The multimodal terrain data is converted into a visual sequence using a large visual model; Based on the soil penetration depth, ground clearance, slewing collision avoidance margin, text sequence, and visual sequence, the action output of the operating machinery is determined through a simplified iterative strategy model; wherein, the simplified iterative strategy model employs a regression network with two fixed forward propagations. Perform the action and output the result.

2. The method according to claim 1, characterized in that, The step of converting the multimodal terrain data into a visual sequence using a large visual model includes: Based on the multimodal terrain data, a top-view elevation map is determined; Based on the multimodal terrain data, a forward-looking depth map is determined; The top-view elevation map, the forward-looking depth map, and the forward-looking RGB image are stitched together along the channel dimension to obtain the RGB-D input tensor; The RGB-D input tensor is input into the large visual model to obtain the visual sequence.

3. The method according to claim 1, characterized in that, The process of determining the action output of the operating machinery based on the soil penetration depth, ground clearance, slewing collision avoidance margin, text sequence, and visual sequence using a simplified iterative strategy model includes: By concatenating the soil penetration depth, the ground clearance, the rotational anti-collision margin, and the pose angle, a low-dimensional body geometric state vector is obtained. The text sequence, the visual sequence, and the low-dimensional ontology geometric state vector are concatenated to obtain a high-dimensional comprehensive context feature vector. The high-dimensional comprehensive context feature vector is input into the simplified iterative strategy model to obtain the action output.

4. The method according to claim 2, characterized in that, The multimodal terrain data includes three-dimensional point cloud data. The step of determining a top-view elevation map based on the multimodal terrain data includes: Using the rotation center of the operating machinery as the origin of the plane coordinate system, the ground plane is selected as the XY plane, and the effective working area in front of the operating machinery is divided into a two-dimensional grid matrix with a preset resolution. The highest Z-axis coordinate value among all point clouds falling within each two-dimensional grid is taken as the gray value of the two-dimensional grid matrix; wherein, the Z-axis is perpendicular to the XY plane.

5. The method according to claim 4, characterized in that, The method further includes: For two-dimensional graticules without point cloud coverage, a spatial interpolation algorithm is used for smooth filling.

6. The method according to claim 4, characterized in that, The step of determining the forward-looking depth map based on the multimodal terrain data includes: Based on homogeneous coordinate transformation, the point cloud data is projected onto the two-dimensional imaging pixel plane of the forward-looking camera, and the forward-looking depth map is generated by interpolation.

7. A device for automatic control of operating machinery, characterized in that, The device includes: The acquisition module is used to acquire multimodal terrain data, the pose angle of the working device, and operator instructions; The geometric state module is used to determine the soil penetration depth, ground clearance, and rotational anti-collision margin of the working device end based on the multimodal terrain data and the pose angle. The first processing module is used to convert the instructions into a text sequence through a large language model; The second processing module is used to convert the multimodal terrain data into a visual sequence through a large visual model; The third processing module is used to determine the action output of the operating machinery based on the soil penetration depth, the ground clearance, the slewing anti-collision margin, the text sequence, and the visual sequence through a simplified iterative strategy model; wherein the simplified iterative strategy model adopts a regression network with two fixed forward propagations. The execution module is used to execute the action output.

8. A controller, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A type of operating machinery, characterized in that, The method includes a main body of a working machine, which includes a working device and a controller, the controller being used to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images