Fuel efficiency optimization method and system based on multi-modal visual fusion

Abstract

The present application relates to the field of engineering machinery control technology, specifically to a fuel efficiency optimization method and system based on multi-modal visual fusion, obtaining the environment image around the engineering machinery, generating a parallax map and performing three-dimensional reconstruction; according to the reconstructed image and the pre-saved engineering machinery joint model, obtaining the three-dimensional coordinates and spatial posture of the boom joint; according to the three-dimensional coordinates and spatial posture of the boom joint, the state parameters of the engineering machinery and the reconstructed image, classifying the current working condition of the engineering machinery, and determining the action state and slope correction angle under different working condition types; according to the working condition type, the action state and the slope correction angle, based on the pre-constructed fuel mapping database, obtaining the corresponding load intensity coefficient, engine speed, main pump pressure, oil supply correction coefficient and torque compensation, adjusting the current state parameters of the engineering machinery, and realizing fuel optimization.

Description

Technical Field

[0001] This invention relates to the field of engineering machinery control technology, specifically to a fuel efficiency optimization method and system based on multimodal vision fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] During the operation of construction machinery, the driver's experience is used to determine the characteristics of materials and the current working conditions, thereby controlling the fuel supply to achieve power control. This method relies too heavily on subjective human experience, which can easily lead to a mismatch between power and load. Some existing technologies attempt to use sensors to acquire the status of the construction machinery, combined with the computing power of the onboard computer, to help the driver make judgments on the working conditions or material types, and even switch modes. However, this method has too high latency and large errors, and cannot distinguish precise operation actions. Summary of the Invention

[0004] To address the technical problems mentioned above, this invention provides a fuel efficiency optimization method and system based on multimodal visual fusion. This method utilizes visual recognition to acquire material attributes (material type and wettability) and operating condition attributes (operating condition type and operating state). By using a pre-built fuel economy mapping database, it queries and obtains compensation coefficients corresponding to the combination of material attributes and operating condition attributes, thereby adjusting parameters such as the speed, torque, and fuel supply of the construction machinery to optimize fuel efficiency.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a fuel efficiency optimization method based on multimodal vision fusion, comprising the following steps: Acquire and preprocess environmental images of the space surrounding the construction machinery; The material location is determined based on the preprocessed image. The texture and color features of the material in the image are extracted. The material type is determined based on the contrast and energy in the texture features. The material moisture level is determined based on the brightness standard deviation and saturation in the color features. The preprocessed image is converted into a disparity map and reconstructed in three dimensions. Combined with the pre-saved joint model of the engineering machinery, the three-dimensional coordinates and spatial posture of the boom joint are obtained. Furthermore, combined with the state parameters of the engineering machinery, the current working condition type of the engineering machinery and the action state under different working conditions are determined. Based on the combination of material type, moisture level, operating condition type, and action state, a pre-built fuel mapping database is used to obtain the corresponding load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient, and torque compensation value, thereby optimizing the fuel supply.

[0006] Furthermore, texture features are quantized using a gray-level co-occurrence matrix, and contrast, energy, and homogeneity are obtained based on the gray-level index, the original frequency of co-occurrence of the corresponding gray levels, and the normalized probability value.

[0007] Furthermore, the color features are converted from RGB space to HSV space to obtain hue, saturation, and lightness, and the standard deviation of lightness and hue distribution entropy are determined.

[0008] Furthermore, the material type is determined based on the contrast and energy in the texture features, specifically: When the contrast ratio is greater than threshold A and the energy is less than threshold a, the material is stone. When the contrast is between threshold A and threshold B, and the energy is between threshold a and threshold b, the material is earth and rock. All other cases involve earthwork.

[0009] Furthermore, the material's moisture content is determined based on the standard deviation of lightness and saturation in its color characteristics, specifically: If the standard deviation of brightness is greater than the threshold X and the saturation is less than the threshold X, the material is considered dry. If the standard deviation of brightness is greater than the threshold Y and the saturation is less than the threshold y, the wettability of the material is slightly wet. If the standard deviation of brightness is greater than the threshold Z and the saturation is less than the threshold Z, the wettability of the material is moderate. In all other cases, the humidity is extremely high.

[0010] Furthermore, based on the three-dimensional coordinates and spatial orientation of the boom joint, combined with the state parameters of the construction machinery, the current working condition type of the construction machinery is determined, specifically: The bucket trajectory of the construction machinery is distributed in a fan shape, the turning cycle of the machine body is within the set range, and the working condition is tossing and dumping. The bucket height of the construction machinery changes beyond the set value and reciprocates between fixed positions; the working condition is loading. The bucket height of the construction machinery fluctuates less than the set value and moves along a horizontal trajectory. The ground texture changes continuously during the movement, and the working condition is flat ground. The angle between the bucket of the construction machinery and the ground exceeds a set value, and this angle changes continuously within a set time period. The working condition is slope repair.

[0011] Furthermore, by utilizing the changes in bucket height, hydraulic pressure, and body rotation angle of the engineering vehicle, as well as the texture features of the surrounding environment images during these changes, the operational status under different working conditions can be determined.

[0012] Furthermore, fuel supply optimization includes adaptive speed adjustment, dynamic torque compensation, and closed-loop fuel supply control. The rotational speed is adaptively adjusted, as shown in the following formula: n target =n base +Δn+K n ·（K L -1） ; in, n base Based on the base speed, Δn The speed compensation value is obtained based on the fuel mapping database. K n The rotational speed sensitivity coefficient; Dynamic torque compensation is shown in the following formula: ΔT final =ΔT base ·K L +K T ·(t actual -t target ） ; ΔT base The torque compensation value is obtained based on the fuel mapping database. K T =0.1, t actual For real-time torque, t target For the target torque under economic operating conditions, P Pi (π) is the mathematical constant of a circle. The closed-loop control of fuel supply is shown in the following formula: Q fuel =Q calibrated ·β·（0.9+0.1·K L ）·（n target / n econ ） ; Q calibrated To calibrate the fuel supply to the engine. β The fuel supply correction factor is obtained based on the fuel mapping database. n econ This is the reference speed for the set economic speed.

[0013] Furthermore, the load intensity coefficient in the fuel mapping database K L As shown in the following formula: K L =α·（P current / P max )+β·(1+θ×0.015)+γ·(v / v max ） ; in, α For hydraulic pressure weighting, β As the slope resistance weight, c Weighted by motion speed. P current For real-time hydraulic pressure, i To adjust the slope angle, v The speed at which the bucket moves. P max (The maximum pressure of the fuel system for construction machinery). v max This is the maximum design speed of the bucket.

[0014] A second aspect of the present invention provides a fuel efficiency optimization system based on multimodal vision fusion, comprising: The image acquisition module is configured to acquire and preprocess environmental images of the space surrounding the construction machinery. The visual recognition module is configured to: determine the area where the material is located based on the preprocessed image, extract the texture and color features of the material in the image, determine the material type based on the contrast and energy in the texture features, and determine the material moisture level based on the brightness standard deviation and saturation in the color features. The visual recognition module is also configured to: convert the preprocessed image into a disparity map and reconstruct it in three dimensions; combine it with the pre-saved engineering machinery joint model to obtain the three-dimensional coordinates and spatial posture of the boom joint; and further combine it with the state parameters of the engineering machinery to determine the current working condition type of the engineering machinery and the action state under different working conditions. The fuel optimization module is also configured to optimize the fuel supply by using a pre-built fuel mapping database to obtain the corresponding load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient, and torque compensation value based on the combination of material type, wetness, operating condition type, and action state.

[0015] Compared with existing technologies, one or more of the above technical solutions have the following beneficial effects: 1. After acquiring images of the environment surrounding the construction machinery, visual recognition is used to obtain material properties (material type and moisture level) and operating condition properties (operating condition type and motion state). A pre-built fuel economy mapping database is used to retrieve compensation coefficients corresponding to combinations of material and operating condition properties. This allows for adjustments to parameters such as the machinery's speed, torque, and fuel supply, optimizing fuel efficiency. The depth information provided by visual recognition can resolve subtle changes that traditional sensors cannot distinguish, reducing errors and improving fuel optimization. The identification of material and operating condition properties can be achieved through pre-training, accelerating the machinery's response to control commands. The fuel economy mapping database digitizes engineers' experience, transforming discrete experience into a continuous parameter space beforehand, eliminating ambiguity from manual operation and ensuring optimal parameters for any combination of operating conditions. Through visual recognition and a pre-built database, as many subdivided scenarios as possible can be covered, improving fuel efficiency.

[0016] 2. By combining images acquired through visual means with time information and existing sensors on the construction machinery, the construction conditions (shoveling / loading / leveling / slope repair), action status (excavation / unloading, etc.), material type (earthwork / rockwork / earth-rockwork), and moisture level (dry / lightly moist / moderately moist / heavily moist) can be identified more precisely, enabling more precise understanding of the construction machinery's operating scenarios and achieving adaptive adjustment of key control parameters.

[0017] 3. Traditional control processes rely on sensor data from the entire vehicle. Taking a digging event as an example, traditional solutions require the use of handle signals and high-pressure sensor values ​​to determine whether digging is in progress. Only after digging has begun can the digging event be detected and relevant control commands output, resulting in a delay. In contrast, vision-based methods can acquire recognition results and determine corresponding control parameters in a very short time before digging begins. This allows for different strategy responses to different working conditions, enabling more refined control strategies, improving control effectiveness while reducing control latency. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0019] Figure 1 This is a schematic diagram of the overall process of the fuel efficiency optimization method based on multimodal vision fusion provided by one or more embodiments of the present invention; Figure 2 This is a schematic diagram of the structure of a fuel efficiency optimization system based on multimodal vision fusion provided in one or more embodiments of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] As described in the background section, traditional construction machinery relies on the driver's experience for control. Some existing technologies use various signals acquired by sensors combined with the computing power of onboard computers to achieve assisted control, thereby attempting to optimize fuel efficiency. However, there are delays, and it is impossible to adaptively adjust key machine control parameters according to changes in materials and operating conditions, making it difficult to distinguish fine-grained operational actions.

[0023] Therefore, the fuel efficiency optimization method and system based on multimodal visual fusion presented in the following embodiments utilize visual recognition to obtain material attributes (material type and wetness) and operating condition attributes (operating condition type and action state). By querying a pre-built fuel economy mapping database, the compensation coefficient corresponding to the combination of material attributes and operating condition attributes is obtained, thereby adjusting parameters such as the speed, torque and fuel supply of the construction machinery to optimize fuel efficiency.

[0024] Example 1: like Figure 1 As shown, the fuel efficiency optimization method based on multimodal vision fusion includes the following steps: Acquire and preprocess environmental images of the space surrounding the construction machinery; The material location is determined based on the preprocessed image. The texture and color features of the material in the image are extracted. The material type is determined based on the contrast and energy in the texture features. The material moisture level is determined based on the brightness standard deviation and saturation in the color features. The preprocessed image is converted into a disparity map and reconstructed in three dimensions. Combined with the pre-saved joint model of the engineering machinery, the three-dimensional coordinates and spatial posture of the boom joint are obtained. Furthermore, combined with the state parameters of the engineering machinery, the current working condition type of the engineering machinery and the action state under different working conditions are determined. Based on the combination of material type, moisture level, operating condition type, and action state, a pre-built fuel mapping database is used to obtain the corresponding load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient, and torque compensation value, thereby optimizing the fuel supply.

[0025] After acquiring the image, material identification is achieved through texture and color dual-channel feature fusion, and working condition identification is achieved through three-dimensional reconstruction and spatiotemporal fusion algorithm.

[0026] Regarding material identification, images within a certain range in front of the construction machinery are acquired. After preprocessing and target detection, images of the area where the material is located are obtained. Texture and color features are extracted from the material images. GLCM texture analysis is performed on the texture features, and HSV color space conversion is performed on the color features. The texture analysis results and the converted color features are then used for feature fusion to make classification decisions.

[0027] Texture features are extracted using the Gray-Level Co-occurrence Matrix (GLCM), including the following steps: By reducing the grayscale value, the computational complexity is reduced, specifically: I g (x, y) = [Gray(x, y) / 16] × 16, used to compress 256 levels of grayscale into 16 levels (16 × 16 GLCM matrix), where I g (x, y) represents the quantized grayscale value (0-15), G ray (x, y) represents the gray value (0-255) of the original image at coordinates (x, y); Generate the GLCM matrix (distance d=1, angle θ=0°); Extract texture features as shown in the following formula: ; ; ; in, For contrast, For energy, Homogeneity i and j This represents the gray level index of the grayscale image, with a value range of 0-15 (because 256 gray levels are compressed to 16 levels). i The grayscale level of the reference pixel; j The gray level of adjacent pixels; P(i,j) grayscale i and j The original frequency values ​​of co-occurrence; P n (i, j) This is the normalized probability value.

[0028] Normalized probability value P n (i, j) As shown in the following formula: ; in, m This is the row index of the matrix (value range: 0 to 15). n The column index of the matrix (value range: 0 to 15).

[0029] Color features were converted from RGB space to HSV space, the hue H channel was extracted, the 0-360° hue was divided into 12 intervals, each interval being 30°, the number of pixels in each interval was counted, and the probability p was calculated. k Calculate the hue distribution entropy.

[0030] Color characteristics are converted from RGB space to HSV space: R'=R / 255, G'=G / 255, B'=B / 255; M = max(R', G', B'); m = min(R', G', B'); Δ=Mm.

[0031] Hue H: If M=R': H=60×((G'-B') / Δ); If M=G': H=60×((B'-R') / Δ+2); If M=B': H=60×((R'-G') / Δ+4); If H < 0: H = H + 360.

[0032] Mean saturation: N = total number of pixels.

[0033] Standard deviation of lightness: ; m v =Mean brightness.

[0034] Hue distribution entropy divides the hue range of 0-360° into 12 intervals (each interval being 30°), specifically: ; in, p k Let be the probability that the hue value falls within the k-th interval. p k = Number of pixels in the k-th interval / Total number of pixels (k=1, 2, ..., 12).

[0035] Feature fusion is shown in the following formula: ; Among them, w t =0.6 (texture weight), w c =0.4 (color weight) Feature normalization: all feature values ​​are scaled to [0, 1].

[0036] The classification decision is as follows: ; Where, sign is the sign function, which outputs +1 when the result inside the parentheses is greater than 0, and -1 when it is less than 0; α i For the first i The Lagrange multipliers (weights) corresponding to each support vector are obtained through optimization during training. y i For the first i The true class label (+1 or -1) of each support vector; K(F) i ，F） =exp(- c||F i -F||2） ; c This is the kernel width parameter (0.5 in this system). |F i -F| The distance is Euclidean. b This is the bias term (the intercept of the decision function), obtained during training.

[0037] The decision-making process includes the following steps: (1) Calculate similarity: Calculate input features F With each support vector F i Similarity: s i =K(F i ，F） ; (2) Weighted summation: Multiply the similarity by the label weight and then sum them, as shown in the following formula: ; in, α i For the first i The Lagrange multipliers (weights) corresponding to each support vector. y i For the first i The true class labels of the support vectors. s i Similarity; (3) Add bias terms b : S′=S+b (4) Determination of sign function: ; When classifying materials according to the same type using symbolic functions, each material type is determined using a corresponding symbolic function. For example: Earthwork classifier output f( F ) =-1 means it does not belong to earthwork; Earthwork classifier output f( F ) =-1 means it does not belong to earthwork; Stone Classifier Output f( F ) =+1 indicates that it belongs to the stone square; Final assessment: It belongs to the stone section.

[0038] When making a decision, the material is classified into a category by the "+1" output of the sign function in the three classifiers. For example, when the sign function output in the rock classifier is "+1", the material is classified into "rock"; when the sign function output in the earthwork classifier is "+1", the material is classified into "earthwork".

[0039] The material classification rules are as follows: When the contrast ratio is >0.35 (threshold A) and the energy is <0.2 (threshold a), the material is stone. When the contrast ratio is 0.25-0.35 (between threshold A and threshold B) and the energy is 0.2-0.4 (between threshold a and threshold b), the material is earth and stone. All other cases involve earthwork.

[0040] Contrast ratio reflects the hardness and edge sharpness of materials. High contrast indicates a large difference in grayscale values ​​between adjacent pixels in an image, corresponding to sharp edges and rough surfaces of materials (such as the sharp edges of rocks or the unevenness of broken rocks). Low contrast indicates a gradual change in grayscale values, corresponding to uniform particle size and smooth surfaces of materials (such as loose soil or fine sand).

[0041] In rock formations, the fracture surfaces of the rock produce a large number of sharp edges, which significantly increases the number of off-diagonal elements in the GLCM, resulting in a higher contrast value.

[0042] In the earthwork, the soil particles are small and evenly distributed, and the GLCM is concentrated near the diagonal, resulting in low contrast.

[0043] In earthwork, the mixture of gravel and soil results in a texture that is somewhere in between, thus having moderate contrast.

[0044] Energy is used to characterize the complexity of a material's structure. At low energy, GLCM elements are dispersed, indicating a chaotic and disordered texture (such as the random fracture surfaces of rock). At high energy, GLCM elements are concentrated, indicating a uniform and repetitive texture (such as the periodic particle arrangement of compacted soil).

[0045] The humidity level is determined as follows: If the standard deviation of lightness is >0.7 (threshold X) and the saturation is <0.3 (threshold x), the material is considered dry. If the standard deviation of lightness is >0.6 (threshold Y) and the saturation is <0.5 (threshold y), the material's wettability is slightly wet. The material is moderately wet if the standard deviation of lightness is >0.4 (threshold Z) and the saturation is <0.7 (threshold z). In all other cases, the humidity is extremely high.

[0046] The standard deviation of lightness reflects the uneven distribution of moisture on the surface of a material. Dry materials have rough surfaces, uniform diffuse reflection of light, and small variations in lightness (low V standard deviation). In moist but unsaturated materials, water films accumulate in depressions, forming alternating bright spots (high lightness) and dry areas (low lightness), resulting in drastic fluctuations in lightness. In heavily moist materials (such as mud), moisture completely covers the surface, creating uniform specular reflection, and the lightness tends to be consistent.

[0047] The mean saturation value (S-mean) is used to characterize the dilution effect of water on color. Water dilutes the surface color, reducing saturation. For example, the surface color of dry materials is bright (e.g., loess S≈0.6); water molecules in wet materials scatter light, diluting the color (reducing the S-value); in extreme cases, mud saturates close to 0 due to the water film covering it.

[0048] Regarding work condition recognition, the preprocessed image is used to generate a disparity map through stereo matching, and then the 3D coordinates of the boom joint are obtained through point cloud reconstruction and key point detection. The 3D pose estimate is obtained through PnP pose solving. Based on the 3D pose estimate, the work conditions are classified and the action state is determined. At the same time, the slope adjustment angle is calculated to form the recognition result and output it.

[0049] The reconstructed image is used for working condition identification using a spatiotemporal fusion algorithm. Specifically, based on the reconstructed image and the three-dimensional coordinates of the boom joint, a three-dimensional attitude estimate for the construction machinery is obtained, and further, working condition labels, action status, and slope repair angle are obtained.

[0050] 3D pose estimation is performed by solving the spatial pose of the boom and bucket joints using the PnP algorithm (input: 2D key points + camera calibration parameters). Specifically, the coordinates of key points in the 2D images captured by the binocular camera are combined with the coordinates of the 3D model points of the construction machinery (such as an excavator) and the camera intrinsic parameter matrix. The PnP solver then solves for the 6-DOF pose (3D position + 3D rotation) of the boom joint in space, and the output pose matrix R|t is obtained.

[0051] The coordinates of 12 2D keypoints were obtained through HRNet keypoint detection. The pose matrix R / t was obtained through the EPnP solver, and the 6-DOF pose, spatial coordinates, attitude angles, and spatiotemporal features were further determined. The slope trimming angle θ=arctanΔz / Δx was calculated based on the spatial coordinates and attitude angles. The working conditions were classified and the state was further determined using ResNet-34 based on the spatiotemporal features.

[0052] The key point detection network uses the lightweight HRNet (parameter count < 1.2M), and the input image is 1920x1080. Through multi-resolution parallel processing, three types of features are obtained: 256x144, 128x72, and 64x36. The feature fusion outputs a heatmap, and the key points are determined based on the heatmap.

[0053] The closed-form solution algorithm for EPNP is as follows: (1) Definition of control points: Let the coordinates of the four control points in the world coordinate system be: ; in, , , Control points j Three-dimensional coordinates in the world coordinate system.

[0054] (2) 3D point parameterization: Any 3D point P w i Represented as a weighted combination of control points: ; α ij For the weight of the centroid coordinates, i For 3D point indices (i=1,2,...,n).

[0055] (3) Camera projection equation: ; in, l i is the depth scale factor, representing the distance from a 3D point to the camera. u i 、v i These are the pixel coordinates on the 2D image plane, and K is the camera intrinsic parameter matrix; K=| fx 0 u0 | K = | 0 fyv0 | K = |001|.

[0056] in, fx、fy The focal length (in pixels) is the focal length along the x and y axes. u0 The horizontal coordinates (in pixels) of the main point. v0 The vertical coordinates (in pixels) of the master point. C cj For the first jThe 3D coordinates of each control point in the camera coordinate system.

[0057] (4) Construct a system of linear equations: eliminate l i The constraint equations are obtained as follows: ; ; The pose calculation formula is as follows: ; in,[ u, v [R|t] represents the 2D image coordinates, K is the camera intrinsic parameter matrix, [R|t] is the rotation matrix + translation vector to be determined, and [X, Y, Z] are the 3D model coordinates.

[0058] The working condition classification model is based on an improved ResNet-34, and the model architecture is as follows: input layer (800x600x3), convolutional layer (64 filters, 7x7, stride=2) + BN + ReLU, max pooling (3x3, stride=2), [residual block group x 4] (each group doubles the number of channels: 64→128→256→512), global average pooling, fully connected layer (1024 units) + Dropout (0.5), output layer (4 units + softmax).

[0059] The working condition classification strategy is shown in Table 1.

[0060] Table 1 Working Condition Classification Strategy

[0061] The working condition types of construction machinery are obtained through a working condition classification model, including dumping, loading, leveling, and slope repair.

[0062] (1) Sliding operation: [Slinging] --> Excavation and loading: Bucket height < 1.2m & hydraulic pressure > 28MPa; Excavation and loading --> Rotation and positioning: Bucket height > 2.0m & machine body rotation > 30°; Rotary positioning --> Spreading and unloading: Bucket reaches the unloading area & opening >75%; Discharge and unload --> Return to reset: Pressure drop <15MPa & opening <20%; Return to reset --> Excavation and loading: Bucket height < 1.0m & pointing towards the stockpile.

[0063] (2) Loading conditions: [Loading] --> Precision digging: Bucket height < 0.8m & pressure > 30MPa; Precise digging --> Smooth lifting: Bucket height > 1.5m & vibration < 0.1g; Smooth improvement --> Mining card positioning: Mining card outline detected (visual); Truck positioning --> Precise unloading: Bucket inside the truck bed & height <1.0m; Precise unloading --> Efficient return: Opening degree > 85% & pressure < 18MPa; Efficient return --> Precise excavation: Bucket height < 0.7m & pointing towards the material pile.

[0064] A rectangular outline (length > 5m, height > 2m) and texture feature "wheel + truck bed" were detected: confirmed as a mining truck.

[0065] In this embodiment, a rectangular outline is detected to identify the cargo bed, and texture features are used to identify the wheels, which is achieved through circular detection and tire tread analysis. Specifically: Circular detection: (xa)² - (yb)² = r²; Tread pattern analysis: Local Binary Pattern (LBP) is used to extract tread patterns, and Histogram of Oriented Gradients (HOG) is used to capture radial textures; Matching rules: Four circles were detected + LBP variance is greater than 120 + HOG orientation consistency is greater than 0.7.

[0066] In addition, the YOLOv5 model can be used for image training. After training, the open-source model can be directly called for mining truck recognition. This method has a better recognition effect than the rectangular outline + texture feature method. However, considering the computing power limitation of the computer on the construction machinery, this solution adopts the rectangular outline + texture feature judgment method.

[0067] (3) Flat ground conditions: [Flat Ground] --> Scraper Grounding: Bucket Inclination Angle < 5° & Pressure < 15MPa; Scraper grounding --> forward leveling: speed 0.3-0.8m / s & height fluctuation <5cm; Forward leveling --> Lifting and turning: Body rotation > 20° & height > 0.3m; Lifting plate for steering --> Reversing and positioning: Rotation angle reaches target value ±5°; Reversal positioning --> scraper grounding: bucket re-touches the ground; Forward leveling --> Operation completed: No elevation change for 3 consecutive times.

[0068] (4) Slope repair conditions: [Slope Repair] --> Slope Location: Detected slope & gradient >10°; Slope positioning --> Cutting into the slope: Bucket height < 0.5m & pressure > 25MPa; Cut into the slope --> Slope adjustment: Move along the slope and change the slope by <1° / s; Slope adjustment --> Fine-tuning inspection: Moving speed <0.2m / s; Fine-tuning inspection --> Acceptance completed: Slope error <1° for 5 seconds.

[0069] Slope angle calculation: θ = arctan(Δz / Δx), where Δz is the height difference between the bucket and the machine body, and Δx is the horizontal projection distance.

[0070] Based on the combination of material type, moisture level, operating condition type and action state obtained from the above visual recognition part, the corresponding load intensity coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient and torque compensation value are obtained by using the pre-built fuel mapping database, so as to optimize the fuel supply.

[0071] The fuel mapping database is a four-dimensional database, which includes material type (3 categories) × wettability (4 levels) × operating condition (4 categories) × operating state (5 types), forming a total of 240 combinations. Each combination has a corresponding load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient and torque compensation value.

[0072] By determining the corresponding fuel control parameters under different operating conditions and action states through preliminary experiments or simulations, a fuel mapping database is formed. Due to the large amount of data, this embodiment shows a portion of the database as an example, as shown in Table 2.

[0073] Table 2 Fuel Mapping Database

[0074] The parameters involved in the fuel mapping database are explained in Table 3.

[0075] Table 3 Parameter Explanation

[0076] Based on the obtained load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient, and torque compensation value, the fuel supply quantity is optimized. Among these, the load strength coefficient... K L As shown in the following formula: K L =α·（P current / P max )+β·(1+θ×0.015)+γ·(v / v max ） ; The above formula includes three parameters: weighting coefficients, dynamic inputs, and constants.

[0077] The weighting coefficients are: α =0.6 (hydraulic pressure weight). β =0.3 (slope resistance weight). c =0.1 (motion speed weight).

[0078] Dynamic input is: P current Real-time hydraulic pressure (CAN bus read). i For slope repair angle (calculated by visual system). v The bucket moving speed (calculated using three-dimensional coordinate difference).

[0079] The constant is: P max =50MPa (maximum system pressure). v max =2.0m / s (maximum design speed of the bucket).

[0080] The fuel supply is optimized through adaptive speed adjustment, dynamic torque compensation, and closed-loop control of the fuel supply.

[0081] (1) Adaptive speed adjustment, as shown in the following formula: n target =n base +Δn+K n ·（K L -1） ; in, n base Based on the base speed, Δn The speed compensation value is obtained by looking up a table in the database. K n =20 is the speed sensitivity coefficient (measured optimized value).

[0082] (2) Dynamic torque compensation, as shown in the following formula: ΔT final =ΔT base ·K L +K T ·(t actual -t target ） ; ΔTbase The torque compensation value is obtained by looking up a table in the database. K T =0.1 is the coefficient, t actual Real-time torque via CAN bus. t target For the target torque under economic operating conditions, t target =60·P econ ·1000 / 2Π·n target , P econ The most economical power for the current operating conditions (provided by the database). P Let π be the mathematical constant pi.

[0083] (3) Closed-loop control of fuel supply, as shown in the following formula: Q fuel =Q calibrated ·β·（0.9+0.1·K L ）·（n target / n econ ） ; Q calibrated To calibrate the fuel supply to the engine. β The fuel supply correction factor is obtained by looking up a table in the database. n econ Economic speed benchmark (provided by database).

[0084] Economic speed reference n econ For the set values, this embodiment sets: SF = 1750 rpm, ZC = 1800 rpm, PD = 1700 rpm, and XP = 1900 rpm.

[0085] By visually identifying the texture and color of materials, and using contrast and energy in texture features to distinguish material hardness and classify material types, and using saturation and brightness in color features to determine the moisture level of materials, the control parameters that can be queried later can cover more complex scenarios, solving the problem that empirical parameters cannot cover complex scenarios.

[0086] By combining images acquired through visual means with temporal information and existing sensors on the construction machinery, the working conditions (shoveling / loading / leveling / slope repair) and operational states (excavation / unloading, etc.) of the construction machinery can be identified more accurately. This allows for a more precise understanding of the working conditions and states of the construction machinery, reducing reliance on high-precision industrial sensors (such as LiDAR). Since the cost of binocular cameras is lower than that of high-precision industrial sensors, and the algorithm is transferable, it can be quickly adapted to equipment such as excavators and loaders by updating the key point definitions of the 3D model, thereby reducing the cost of the algorithm. This improves the overall control effect of the construction machinery and reduces control costs. Furthermore, the data continuously collected during the operation of the construction machinery can help the algorithm to be continuously optimized.

[0087] After visually identifying material and operating condition attributes, the resulting combinations can cover as many subdivided scenarios as possible. A pre-built fuel mapping database serves as a dynamic parameter optimization engine, digitizing engineers' experience. By pre-converting discrete experience into a continuous parameter space, the ambiguity of manual operation is eliminated. A four-dimensional database addresses scenario coverage issues dynamically. K L The coefficient addresses the real-time fluctuation problem, while the dual closed-loop control solves the delay problem, creating a cascading enhancement effect.

[0088] Example 2: A fuel efficiency optimization system based on multimodal vision fusion includes: The image acquisition module is configured to acquire and preprocess environmental images of the space surrounding the construction machinery. The visual recognition module is configured to: determine the area where the material is located based on the preprocessed image, extract the texture and color features of the material in the image, determine the material type based on the contrast and energy in the texture features, and determine the material moisture level based on the brightness standard deviation and saturation in the color features. The visual recognition module is also configured to: convert the preprocessed image into a disparity map and reconstruct it in three dimensions; combine it with the pre-saved engineering machinery joint model to obtain the three-dimensional coordinates and spatial posture of the boom joint; and further combine it with the state parameters of the engineering machinery to determine the current working condition type of the engineering machinery and the action state under different working conditions. The fuel optimization module is also configured to optimize the fuel supply by using a pre-built fuel mapping database to obtain the corresponding load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient, and torque compensation value based on the combination of material type, wetness, operating condition type, and action state.

[0089] By visually identifying the texture and color of materials, and using contrast and energy in texture features to distinguish material hardness and classify material types, and using saturation and brightness in color features to determine the moisture level of materials, the control parameters that can be queried later can cover more complex scenarios, solving the problem that empirical parameters cannot cover complex scenarios.

[0090] By combining images acquired through visual means with temporal information and existing sensors on the construction machinery, the working conditions (shoveling / loading / leveling / slope repair) and operational states (excavation / unloading, etc.) of the construction machinery can be identified more accurately. This allows for a more precise understanding of the working conditions and states of the construction machinery, reducing reliance on high-precision industrial sensors (such as LiDAR). Since the cost of binocular cameras is lower than that of high-precision industrial sensors, and the algorithm is transferable, it can be quickly adapted to equipment such as excavators and loaders by updating the key point definitions of the 3D model, thereby reducing the cost of the algorithm. This improves the overall control effect of the construction machinery and reduces control costs. Furthermore, the data continuously collected during the operation of the construction machinery can help the algorithm to be continuously optimized.

[0091] After visually identifying material and operating condition attributes, the resulting combination can cover as many subdivided scenarios as possible. By using a pre-built fuel mapping database as a dynamic parameter optimization engine, the engineer's experience is digitized. By pre-converting discrete experience into a continuous parameter space, the ambiguity of manual operation is eliminated. The scenario coverage problem is solved by a four-dimensional database, the real-time fluctuation problem is solved by dynamic KL coefficients, and the delay problem is solved by dual closed-loop control, forming a cascading enhancement effect.

[0092] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fuel efficiency optimization method based on multimodal visual fusion, characterized in that, Includes the following steps: Acquire environmental images around the construction machinery, generate a disparity map, and perform 3D reconstruction; based on the reconstructed images and a pre-saved joint model of the construction machinery, obtain the 3D coordinates and spatial orientation of the boom joint; Based on the three-dimensional coordinates and spatial posture of the boom joint, the state parameters of the construction machinery, and the reconstructed image, the current working condition of the construction machinery is classified, and the action state and slope repair angle under different working condition types are determined, specifically: The bucket trajectory of the construction machinery is distributed in a fan shape, the turning cycle of the machine body is within the set range, and the working condition is tossing and dumping. The bucket height of the construction machinery changes beyond the set value and reciprocates between fixed positions; the working condition is loading. The bucket height of the construction machinery fluctuates less than the set value and moves along a horizontal trajectory. The ground texture changes continuously during the movement, and the working condition is flat ground. The angle between the bucket of the construction machinery and the ground exceeds a set value, and this angle changes continuously within a set time period. The working condition is slope repair. Based on the working condition type, operating status, and slope repair angle, and using a pre-built fuel mapping database, the corresponding load intensity coefficient, engine speed, main pump pressure, fuel supply correction coefficient, and torque compensation are obtained. The current state parameters of the construction machinery are then adjusted to achieve fuel optimization.

2. The fuel efficiency optimization method based on multimodal vision fusion as described in claim 1, characterized in that, Texture features are quantized using a gray-level co-occurrence matrix. Contrast, energy, and homogeneity are obtained based on the gray-level index, the original frequency of co-occurrence of the corresponding gray levels, and the normalized probability value.

3. The fuel efficiency optimization method based on multimodal vision fusion as described in claim 1, characterized in that, Color features are converted from RGB space to HSV space to obtain hue, saturation, and lightness, and the standard deviation of lightness and hue distribution entropy are determined.

4. The fuel efficiency optimization method based on multimodal vision fusion as described in claim 1, characterized in that, The material type is determined based on the contrast and energy in the texture features, specifically: When the contrast ratio is greater than threshold A and the energy is less than threshold a, the material is stone. When the contrast is between threshold A and threshold B, and the energy is between threshold a and threshold b, the material is earth and rock. All other cases involve earthwork.

5. The fuel efficiency optimization method based on multimodal vision fusion as described in claim 1, characterized in that, The degree of material moisture is determined based on the standard deviation of lightness and saturation in color characteristics, specifically: If the standard deviation of brightness is greater than the threshold X and the saturation is less than the threshold X, the material is considered dry. If the standard deviation of brightness is greater than the threshold Y and the saturation is less than the threshold y, the wettability of the material is slightly wet. If the standard deviation of brightness is greater than the threshold Z and the saturation is less than the threshold Z, the wettability of the material is moderate. In all other cases, the humidity is extremely high.

6. A system for implementing the fuel efficiency optimization method based on multimodal vision fusion as described in any one of claims 1-5, characterized in that, include: The image acquisition module is configured to acquire and preprocess environmental images of the space surrounding the construction machinery. The visual recognition module is configured to: determine the area where the material is located based on the preprocessed image, extract the texture and color features of the material in the image, determine the material type based on the contrast and energy in the texture features, and determine the material moisture level based on the brightness standard deviation and saturation in the color features. The visual recognition module is also configured to: convert the preprocessed image into a disparity map and reconstruct it in three dimensions; combine it with the pre-saved engineering machinery joint model to obtain the three-dimensional coordinates and spatial posture of the boom joint; and further combine it with the state parameters of the engineering machinery to determine the current working condition type of the engineering machinery and the action state under different working conditions. The fuel optimization module is also configured to optimize the fuel supply by using a pre-built fuel mapping database to obtain the corresponding load strength coefficient, speed compensation value, pressure compensation value, fuel supply correction coefficient, and torque compensation value based on the combination of material type, wetness, operating condition type, and action state.

Images