Engineering machinery image fusion display method and device based on attitude guidance

Abstract

The invention relates to the technical field of engineering machinery, and discloses an engineering machinery image fusion display method and device based on attitude guidance. Collecting a multi-dimensional inclination angle parameter for indicating a real-time operation inclination angle condition of a multi-dimensional part of the engineering machinery and a multi-dimensional environment image parameter for indicating a real-time environment condition in a preset range of the engineering machinery; according to the multi-dimensional inclination angle parameters, attitude parameters, at least including bucket tip coordinate parameters and joint position coordinate parameters of at least one mechanical joint, of the engineering machinery are calculated; based on the attitude parameters and the multi-dimensional environment image parameters, a multi-dimensional display image of the engineering machine is generated, and the multi-dimensional display image is at least used for indicating one or more of landing point projection information of the bucket, bucket elevation information, distance information of the bucket relative to a cockpit corresponding to the engineering machine and real-time highest position information. Therefore, the accuracy and comprehensiveness of engineering machinery image display can be improved.

Description

Technical Field

[0001] This invention relates to the field of engineering machinery technology, and in particular to an image fusion display method and apparatus for engineering machinery based on posture guidance. Background Technology

[0002] When operators use construction machinery to control operations, they often rely on on-site video transmitted from the field imaging equipment for operation control.

[0003] However, practice has shown that when performing more delicate operations, the lack of depth information or inaccurate depth information in the on-site video makes it difficult for operators to accurately determine the landing point of attachments (such as buckets). This is especially true in delicate working conditions such as trenching, grooving, and crushing, where the working depth and accuracy cannot be guaranteed. This results in low efficiency in actual engineering operations, difficulty in controlling the quality of work, and may even lead to safety hazards.

[0004] Therefore, it is particularly important to propose a technical solution to improve the accuracy and comprehensiveness of engineering machinery image display. Summary of the Invention

[0005] This invention provides a method and apparatus for image fusion display of engineering machinery based on posture guidance, which can improve the accuracy and comprehensiveness of image display of engineering machinery.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses a posture-guided image fusion and display method for construction machinery. The method is applied to construction machinery, which includes a bucket. The method comprises:

[0007] The system collects multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within a preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions of the construction machinery within the preset range.

[0008] Based on the multi-dimensional tilt angle parameters, the attitude parameters of the construction machinery are calculated. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery.

[0009] Based on the attitude parameters and the multi-dimensional environmental image parameters, a multi-dimensional display image of the construction machinery is generated. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

[0010] As an optional implementation, in the first aspect of the present invention, the multi-dimensional tilt angle parameters include vehicle body rotation angle parameters, boom rotation angle parameters, stick rotation angle parameters, and bucket rotation angle parameters. The step of calculating the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters includes:

[0011] The multi-dimensional distance parameters of the construction machinery are collected. The multi-dimensional distance parameters include at least a first distance parameter between the upper hinge point and the lower hinge point of the boom, a second distance parameter between the upper hinge point of the boom and the bucket pin, a third distance parameter between the bucket pin and the bucket tip, a first horizontal component distance parameter between the vehicle rotation center and the lower hinge point of the boom, and a second horizontal component distance parameter between the lower hinge point of the boom cylinder and the lower hinge point of the boom.

[0012] The attitude parameters of the engineering machinery are calculated based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters.

[0013] As an optional implementation, in the first aspect of the present invention, the joint position coordinate parameters include at least the lower hinge point coordinate parameters of the boom, the upper hinge point coordinate parameters of the boom, and the bucket pin coordinate parameters. The calculation of the attitude parameters of the construction machinery based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters includes:

[0014] Based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters, calculate the bucket tip coordinate parameters of the bucket;

[0015] The bucket tip coordinate parameters are calculated based on the bucket tip coordinate parameters, the third distance parameters, and the multi-dimensional tilt angle parameters.

[0016] The coordinate parameters of the lower hinge point of the boom are calculated based on the first horizontal component distance parameter, the second horizontal component distance parameter, and the vehicle body rotation angle parameter.

[0017] The coordinate parameters of the upper hinge point of the boom are calculated based on the coordinate parameters of the lower hinge point of the boom, the first distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter.

[0018] As an optional implementation, in the first aspect of the present invention, the joint position coordinate parameters further include at least one of the following: the upper hinge point coordinate parameters of the stick cylinder, the lower hinge point coordinate parameters of the stick cylinder, the upper hinge point coordinate parameters of the bucket cylinder, the lower hinge point coordinate parameters of the bucket cylinder, and the bucket connecting rod pin coordinate parameters.

[0019] Furthermore, the step of calculating the attitude parameters of the engineering machinery based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters further includes:

[0020] Collect the first included angle from the upper hinge point of the boom cylinder to the lower hinge point of the boom cylinder and the fourth distance parameter between the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder;

[0021] The coordinate parameters of the upper hinge point of the boom cylinder are calculated based on the lower hinge point coordinate parameters of the boom, the first included angle, the fourth distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter.

[0022] The second included angle from the lower hinge point of the boom cylinder to the upper hinge point of the boom to the bucket pin is measured.

[0023] The coordinate parameters of the lower hinge point of the boom cylinder are calculated based on the coordinate parameters of the upper hinge point of the boom, the second included angle, the rotation angle parameters of the vehicle body, and the boom rotation angle parameters.

[0024] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0025] Calculate the target coordinate parameters of the stick idler link hinge point based on the fifth distance parameter and the bucket pin coordinate parameters;

[0026] Based on the target coordinate parameters, the boom upper hinge point coordinate parameters, and the bucket cylinder upper hinge point, calculate the bucket cylinder upper hinge point coordinate parameters;

[0027] Based on the target coordinate parameters, the coordinate parameters of the upper hinge point of the bucket cylinder, and the lower hinge point of the bucket cylinder, calculate the coordinate parameters of the lower hinge point of the bucket cylinder.

[0028] The coordinate parameters of the bucket connecting rod pin are calculated based on the coordinate parameters of the bucket pin, the coordinate parameters of the bucket tip, and the bucket connecting rod pin.

[0029] As an optional implementation, in the first aspect of the present invention, the bucket rotation angle parameter is calculated based on the third included angle from the bucket pin to the boom idler joint hinge point to the bucket cylinder lower hinge point, and the calculation method of the bucket rotation angle parameter is as follows:

[0030] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0031] Collect the sixth distance parameter between the lower hinge point of the bucket cylinder and the hinge point of the stick idler connecting rod;

[0032] Based on the fifth distance parameter, the sixth distance parameter, and the third included angle, a seventh distance parameter is calculated. The seventh distance parameter is used to represent the distance between the bucket pin and the lower hinge point of the bucket cylinder.

[0033] Collect the fourth included angle from the bucket tip to the bucket pin to the bucket connecting rod pin;

[0034] The fifth included angle between the bucket connecting rod pin and the hinge point of the bucket pin and the stick idler connecting rod is collected;

[0035] Collect the eighth distance parameter between the lower hinge point of the bucket cylinder and the bucket connecting rod pin;

[0036] Collect the ninth distance parameter between the bucket pin and the bucket connecting rod pin;

[0037] The bucket turning angle parameter is calculated based on the fourth included angle, the fifth included angle, the fifth distance parameter, the sixth distance parameter, the seventh distance parameter, the eighth distance parameter, and the ninth distance parameter.

[0038] As an optional implementation, in the first aspect of the invention, before generating the multi-dimensional display image of the engineering machinery based on the attitude parameters and the multi-dimensional environmental image parameters, the method further includes:

[0039] Determine the pixel matrix parameters corresponding to the multi-dimensional environmental image parameters and the image acquisition intrinsic parameter matrix parameters of the corresponding sensing component. The image acquisition intrinsic parameter matrix parameters are at least used to indicate the image acquisition focal length and the image acquisition principal point. The pixel matrix parameters correspond to the pixel coordinate system.

[0040] Based on the pixel matrix parameters and the image acquisition intrinsic parameter matrix parameters, the perception matrix parameters of the sensing component are generated, and the perception matrix parameters correspond to the perception coordinate system of the sensing component.

[0041] The rotation matrix parameters and translation vector parameters corresponding to the sensing component are determined. The rotation matrix parameters are calculated based on the Euler angles of the sensing component, and the translation vector parameters are used to represent the position of the optical center corresponding to the sensing component in the vehicle coordinate system of the engineering machinery.

[0042] Based on the perception matrix parameters, the rotation matrix parameters, and the translation vector parameters, the vehicle matrix parameters of the construction machinery are generated, and the vehicle matrix parameters correspond to the vehicle coordinate system of the construction machinery.

[0043] And, generating a multi-dimensional display image of the engineering machinery based on the attitude parameters and the multi-dimensional environmental image parameters includes:

[0044] Based on the vehicle matrix parameters, the attitude parameters, and the multi-dimensional environmental image parameters, a multi-dimensional display image of the construction machinery is generated.

[0045] As an optional implementation, in the first aspect of the present invention, generating a multi-dimensional display image of the construction machinery based on the vehicle matrix parameters, the attitude parameters, and the multi-dimensional environmental image parameters includes:

[0046] Collect millimeter-wave radar detection parameters, which are used to represent the distribution of obstacles within the preset range of the engineering machinery;

[0047] The millimeter-wave radar detection parameters are correlated with the bucket tip coordinate parameters to generate obstacle correlation parameters, which are used to represent the distance and positional relationship between the obstacle and the bucket.

[0048] Based on the vehicle matrix parameters, the attitude parameters, the multi-dimensional environmental image parameters, and the obstacle association parameters, a multi-dimensional display image of the construction machinery containing obstacle warning indicators is generated;

[0049] The millimeter-wave radar detection parameters are generated by controlling the millimeter-wave radar to transmit detection signals and receive reflected signals, and are used to obtain the distribution range and shape of obstacles within the preset range in real time. The obstacle association parameters are obtained by calculating the spatial distance between the obstacle coordinates in the millimeter-wave radar detection parameters and the bucket tip coordinate parameters, and are used to determine whether the obstacle is located within the movement trajectory range of the bucket. The obstacle warning markers mark the obstacle distribution in the multi-dimensional display image through image generation technology, and dynamically adjust the display mode of the markers based on the distance relationship.

[0050] A second aspect of this invention discloses an image fusion display device for engineering machinery based on posture guidance. The device is applied to engineering machinery, which includes a bucket. The device comprises:

[0051] The acquisition module is used to acquire multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within a preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions of the construction machinery within the preset range.

[0052] The calculation module is used to calculate the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery.

[0053] The generation module is used to generate a multi-dimensional display image of the construction machinery based on the attitude parameters and the multi-dimensional environmental image parameters. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

[0054] As an optional implementation, in the second aspect of the present invention, the multi-dimensional tilt angle parameters include vehicle body rotation angle parameters, boom rotation angle parameters, stick rotation angle parameters, and bucket rotation angle parameters. The specific method by which the calculation module calculates the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters includes:

[0055] The multi-dimensional distance parameters of the construction machinery are collected. The multi-dimensional distance parameters include at least a first distance parameter between the upper hinge point and the lower hinge point of the boom, a second distance parameter between the upper hinge point of the boom and the bucket pin, a third distance parameter between the bucket pin and the bucket tip, a first horizontal component distance parameter between the vehicle rotation center and the lower hinge point of the boom, and a second horizontal component distance parameter between the lower hinge point of the boom cylinder and the lower hinge point of the boom.

[0056] The attitude parameters of the engineering machinery are calculated based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters.

[0057] As an optional implementation, in the second aspect of the present invention, the joint position coordinate parameters include at least the lower hinge point coordinate parameters of the boom, the upper hinge point coordinate parameters of the boom, and the bucket pin coordinate parameters. The specific method by which the calculation module calculates the attitude parameters of the construction machinery based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters includes:

[0058] Based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters, calculate the bucket tip coordinate parameters of the bucket;

[0059] The bucket tip coordinate parameters are calculated based on the bucket tip coordinate parameters, the third distance parameters, and the multi-dimensional tilt angle parameters.

[0060] The coordinate parameters of the lower hinge point of the boom are calculated based on the first horizontal component distance parameter, the second horizontal component distance parameter, and the vehicle body rotation angle parameter.

[0061] The coordinate parameters of the upper hinge point of the boom are calculated based on the coordinate parameters of the lower hinge point of the boom, the first distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter.

[0062] As an optional implementation, in the second aspect of the present invention, the joint position coordinate parameters further include at least one of the following: the upper hinge point coordinate parameters of the stick cylinder, the lower hinge point coordinate parameters of the stick cylinder, the upper hinge point coordinate parameters of the bucket cylinder, the lower hinge point coordinate parameters of the bucket cylinder, and the bucket connecting rod pin coordinate parameters.

[0063] Furthermore, the specific method by which the calculation module calculates the attitude parameters of the engineering machinery based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters further includes:

[0064] Collect the first included angle from the upper hinge point of the boom cylinder to the lower hinge point of the boom cylinder and the fourth distance parameter between the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder;

[0065] The coordinate parameters of the upper hinge point of the boom cylinder are calculated based on the lower hinge point coordinate parameters of the boom, the first included angle, the fourth distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter.

[0066] The second included angle from the lower hinge point of the boom cylinder to the upper hinge point of the boom to the bucket pin is measured.

[0067] The coordinate parameters of the lower hinge point of the boom cylinder are calculated based on the coordinate parameters of the upper hinge point of the boom, the second included angle, the rotation angle parameters of the vehicle body, and the boom rotation angle parameters.

[0068] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0069] Calculate the target coordinate parameters of the stick idler link hinge point based on the fifth distance parameter and the bucket pin coordinate parameters;

[0070] Based on the target coordinate parameters, the boom upper hinge point coordinate parameters, and the bucket cylinder upper hinge point, calculate the bucket cylinder upper hinge point coordinate parameters;

[0071] Based on the target coordinate parameters, the coordinate parameters of the upper hinge point of the bucket cylinder, and the lower hinge point of the bucket cylinder, calculate the coordinate parameters of the lower hinge point of the bucket cylinder.

[0072] The coordinate parameters of the bucket connecting rod pin are calculated based on the coordinate parameters of the bucket pin, the coordinate parameters of the bucket tip, and the bucket connecting rod pin.

[0073] As an optional implementation, in a second aspect of the invention, the bucket rotation angle parameter is calculated based on the third included angle from the bucket pin to the boom idler joint hinge point to the bucket cylinder lower hinge point, and the calculation method for the bucket rotation angle parameter is as follows:

[0074] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0075] Collect the sixth distance parameter between the lower hinge point of the bucket cylinder and the hinge point of the stick idler connecting rod;

[0076] Based on the fifth distance parameter, the sixth distance parameter, and the third included angle, a seventh distance parameter is calculated. The seventh distance parameter is used to represent the distance between the bucket pin and the lower hinge point of the bucket cylinder.

[0077] Collect the fourth included angle from the bucket tip to the bucket pin to the bucket connecting rod pin;

[0078] The fifth included angle between the bucket connecting rod pin and the hinge point of the bucket pin and the stick idler connecting rod is collected;

[0079] Collect the eighth distance parameter between the lower hinge point of the bucket cylinder and the bucket connecting rod pin;

[0080] Collect the ninth distance parameter between the bucket pin and the bucket connecting rod pin;

[0081] The bucket turning angle parameter is calculated based on the fourth included angle, the fifth included angle, the fifth distance parameter, the sixth distance parameter, the seventh distance parameter, the eighth distance parameter, and the ninth distance parameter.

[0082] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0083] The determination module is used to determine the pixel matrix parameters corresponding to the multi-dimensional environmental image parameters and the image acquisition intrinsic parameter matrix parameters of the corresponding sensing component before the generation module generates the multi-dimensional display image of the engineering machinery based on the attitude parameters and the multi-dimensional environmental image parameters. The image acquisition intrinsic parameter matrix parameters are used to indicate at least the image acquisition focal length and the image acquisition principal point. The pixel matrix parameters correspond to the pixel coordinate system.

[0084] The generation module is further configured to generate perception matrix parameters of the sensing component based on the pixel matrix parameters and the image acquisition intrinsic parameter matrix parameters, wherein the perception matrix parameters correspond to the sensing coordinate system of the sensing component;

[0085] The determining module is further configured to determine the rotation matrix parameters and translation vector parameters corresponding to the sensing component. The rotation matrix parameters are calculated based on the Euler angles of the sensing component, and the translation vector parameters are used to represent the position of the optical center corresponding to the sensing component in the vehicle coordinate system of the engineering machinery.

[0086] The generation module is further configured to generate the vehicle matrix parameters of the construction machinery based on the perception matrix parameters, the rotation matrix parameters, and the translation vector parameters, wherein the vehicle matrix parameters correspond to the vehicle coordinate system of the construction machinery.

[0087] Furthermore, the specific method by which the generation module generates the multi-dimensional display image of the engineering machinery based on the attitude parameters and the multi-dimensional environmental image parameters includes:

[0088] Based on the vehicle matrix parameters, the attitude parameters, and the multi-dimensional environmental image parameters, a multi-dimensional display image of the construction machinery is generated.

[0089] As an optional implementation, in a second aspect of the present invention, the specific method by which the generation module generates a multi-dimensional display image of the construction machinery based on the vehicle matrix parameters, the attitude parameters, and the multi-dimensional environmental image parameters includes:

[0090] Collect millimeter-wave radar detection parameters, which are used to represent the distribution of obstacles within the preset range of the engineering machinery;

[0091] The millimeter-wave radar detection parameters are correlated with the bucket tip coordinate parameters to generate obstacle correlation parameters, which are used to represent the distance and positional relationship between the obstacle and the bucket.

[0092] Based on the vehicle matrix parameters, the attitude parameters, the multi-dimensional environmental image parameters, and the obstacle association parameters, a multi-dimensional display image of the construction machinery containing obstacle warning indicators is generated;

[0093] The millimeter-wave radar detection parameters are generated by controlling the millimeter-wave radar to transmit detection signals and receive reflected signals, and are used to obtain the distribution range and shape of obstacles within the preset range in real time. The obstacle association parameters are obtained by calculating the spatial distance between the obstacle coordinates in the millimeter-wave radar detection parameters and the bucket tip coordinate parameters, and are used to determine whether the obstacle is located within the movement trajectory range of the bucket. The obstacle warning markers mark the obstacle distribution in the multi-dimensional display image through image generation technology, and dynamically adjust the display mode of the markers based on the distance relationship.

[0094] A third aspect of the present invention discloses another posture-guided image fusion display device for engineering machinery, the device comprising:

[0095] Memory containing executable program code;

[0096] A processor coupled to the memory;

[0097] The processor calls the executable program code stored in the memory to execute the posture-guided image fusion display method for engineering machinery disclosed in the first aspect of the present invention.

[0098] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the posture-guided engineering machinery image fusion display method disclosed in the first aspect of the present invention.

[0099] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0100] In this embodiment of the invention, the method is applied to construction machinery, and the construction machinery includes a bucket. The method includes: acquiring multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within a preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions within the preset range of the construction machinery; calculating the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery; and generating a multi-dimensional display image of the construction machinery based on the attitude parameters and the multi-dimensional environmental image parameters. The multi-dimensional display image is used to indicate at least one or more of the following: bucket landing point projection information, bucket elevation information, distance information of the bucket relative to the corresponding cab of the construction machinery, and real-time highest position information. As can be seen, implementing this invention enables real-time calculation of the attitude parameters of the construction machinery, including at least the bucket tip coordinate parameters and the joint position coordinate parameters of at least one mechanical joint, based on the collected multi-dimensional tilt angle parameters. This allows for real-time mathematical modeling of the physical tilt angle of the construction machinery, and the solution of the real-time attitude of the machinery, achieving real-time positioning of each working component. Compared to the convolutional neural network model used in existing technologies, this invention eliminates the need for model training, saving model training and maintenance costs. It also reduces the requirements for technical personnel and training data, and is unaffected by the operating environment (such as smoke, dust, rain, snow, and other harsh weather conditions). Starting from the state, the system guides the generation of multi-dimensional display images of construction machinery. By further combining the multi-dimensional environmental image parameters collected from the construction machinery within a preset range, it generates at least one or more of the following multi-dimensional display images of the construction machinery: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information. This improves the accuracy and comprehensiveness of the construction machinery image display, allowing operators to have a more intuitive and comprehensive understanding of the real-time spatial attitude of the construction machinery. While reducing human visual errors, it also further avoids collisions between the bucket and obstacles, improving the operational precision and safety of fine-working conditions (such as trenching, grooving, crushing, etc.). Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1This is a flowchart illustrating an image fusion and display method for engineering machinery based on posture guidance, as disclosed in an embodiment of the present invention.

[0103] Figure 2 This is a schematic diagram of the installation position of a multi-dimensional tilt sensor and environmental image acquisition device for engineering machinery disclosed in an embodiment of the present invention;

[0104] Figure 3 This is a projection diagram of the landing point of the bucket tip disclosed in an embodiment of the present invention;

[0105] Figure 4 This is a front view of a bucket disclosed in an embodiment of the present invention;

[0106] Figure 5 This is a cross-sectional view of an engineering machine disclosed in an embodiment of the present invention;

[0107] Figure 6 A schematic flowchart of another posture-guided image fusion and display method for engineering machinery disclosed in this embodiment of the invention;

[0108] Figure 7 This is a schematic diagram of the structure of an image fusion display device for engineering machinery based on posture guidance, as disclosed in an embodiment of the present invention.

[0109] Figure 8 This is a schematic diagram of another posture-guided image fusion display device for engineering machinery disclosed in an embodiment of the present invention;

[0110] Figure 9 This is a schematic diagram of the structure of another posture-guided image fusion display device for engineering machinery disclosed in an embodiment of the present invention. Detailed Implementation

[0111] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0112] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0113] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0114] This invention discloses a posture-guided image fusion display method and device for construction machinery. Based on the acquired multi-dimensional tilt angle parameters of the construction machinery, it can calculate in real time the posture parameters of the machinery, including at least the bucket tip coordinate parameters and the joint position coordinate parameters of at least one mechanical joint. This enables real-time mathematical modeling of the physical tilt angle of the construction machinery, and uses this model to solve for the real-time posture of the machinery, achieving real-time positioning of each working component. Compared to the convolutional neural network model used in existing technologies, this method eliminates the need for model training, saving on model training and maintenance costs. It also reduces the requirements for technical personnel and training data, and is unaffected by the operating environment (such as smoke, dust, rain, snow, and other harsh weather conditions). Starting from the actual spatial posture of the construction machinery, this method guides the generation of multi-dimensional display images of the machinery. By further combining multi-dimensional environmental image parameters collected within a preset range of the construction machinery, it generates at least one or more of the following multi-dimensional display images of the construction machinery: bucket landing point projection information, bucket elevation information, bucket distance information relative to the corresponding cab of the construction machinery, and real-time highest position information. This improves the accuracy and comprehensiveness of the construction machinery image display, allowing operators to more intuitively and comprehensively understand the real-time spatial posture of the construction machinery. It also reduces human visual error and further avoids collisions between the bucket and obstacles, improving the operational precision and safety in fine-tuning operations (such as trenching, grooving, and crushing). Detailed explanations follow.

[0115] Example 1

[0116] Please see Figure 1 , Figure 1This is a flowchart illustrating a posture-guided image fusion and display method for engineering machinery disclosed in an embodiment of the present invention. Wherein, Figure 1 The described posture-guided image fusion display method for construction machinery can be applied to construction machinery, including buckets, and can also be applied to intelligent devices associated with the construction machinery. These intelligent devices include, but are not limited to, one or more of battery devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. This invention does not limit the application of these methods. Figure 1 As shown, the posture-guided image fusion and display method for engineering machinery may include the following operations:

[0117] 101. Collect multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within the preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions within the preset range of the construction machinery.

[0118] In this embodiment of the invention, optionally, the above-mentioned multi-dimensional tilt angle parameters can be collected by controlling the boom tilt angle sensor, stick tilt angle sensor, bucket tilt angle sensor, body tilt angle sensor and slewing angle sensor installed on the construction machinery.

[0119] like Figure 2 As shown, Figure 2 This is a schematic diagram of the installation position of a multi-dimensional tilt sensor and environmental image acquisition device for engineering machinery disclosed in an embodiment of the present invention;

[0120] like Figure 2 As shown, 1 is the bucket tilt sensor, 2 is the stick tilt sensor, 3 is the boom tilt sensor, 4 is the binocular camera, 5 is the left monocular camera, 6 is the vehicle tilt sensor, 7 is the slewing angle sensor, and 8 is the millimeter-wave radar.

[0121] Optionally, the vehicle tilt sensor can simultaneously collect pitch angle and roll angle; the slewing angle sensor can collect the vehicle's horizontal rotation angle.

[0122] Optionally, the multi-dimensional environmental image parameters within the preset range of the aforementioned construction machinery can be acquired by controlling the single / dual-lens cameras (front view) and single / dual-lens cameras (left / right / rear view) installed on the construction machinery to cover the entire work area and reduce blind spots.

[0123] 102. Calculate the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters. The attitude parameters shall include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery.

[0124] In this embodiment of the invention, optionally, based on multi-dimensional tilt angle parameters and fixed geometric dimensions of mechanical components (such as boom length and stick hinge point spacing), a kinematic mathematical model is constructed to solve the problem in real time:

[0125] Bucket tip coordinates: Based on the origin of the vehicle coordinate system (the ground projection point of the vehicle rotation center), and combined with the boom rotation angle parameter α, the stick rotation angle parameter β, and the bucket rotation angle parameter γ, the bucket tip coordinate parameters V(x,y,z) are calculated step by step.

[0126] Joint position coordinates: Simultaneously calculate the spatial positions of key joints such as the lower hinge point of the boom, the upper hinge point of the boom, and the bucket pin;

[0127] In this embodiment of the invention, as an optional implementation, the aforementioned multi-dimensional tilt angle parameters include vehicle body rotation angle parameters, boom rotation angle parameters, stick rotation angle parameters, and bucket rotation angle parameters. Based on these multi-dimensional tilt angle parameters, the attitude parameters of the construction machinery are calculated, including:

[0128] Collect multi-dimensional distance parameters of the construction machinery. The multi-dimensional distance parameters include at least the first distance parameter between the upper hinge point and the lower hinge point of the boom, the second distance parameter between the upper hinge point of the boom and the bucket pin, the third distance parameter between the bucket pin and the bucket tip, the first horizontal component distance parameter between the body rotation center and the lower hinge point of the boom, and the second horizontal component distance parameter between the lower hinge point of the boom cylinder and the lower hinge point of the boom.

[0129] The attitude parameters of the engineering machinery are calculated based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters.

[0130] In this embodiment of the invention, optionally, the first distance parameter, the second distance parameter, the third distance parameter, the first horizontal component distance parameter, and the second horizontal component distance parameter all reflect the corresponding straight-line distance (e.g., the first distance parameter reflects the straight-line distance between the upper hinge point and the lower hinge point of the boom, and so on).

[0131] Alternatively, the first distance parameter, the second distance parameter, the third distance parameter, the first horizontal component distance parameter, and the second horizontal component distance parameter mentioned above can all be obtained by using the design drawings / actual physical measurements of the engineering machinery to obtain their fixed dimensions;

[0132] Alternatively, the above distance parameters and real-time tilt angles (vehicle rotation angle parameter θ1, boom rotation angle parameter α, etc.) can be input into the kinematic model:

[0133] The coordinates are calculated joint by joint using the vector superposition method;

[0134] The final output is the precise coordinates of the bucket tip and key joints.

[0135] As can be seen, implementing this optional embodiment can eliminate the errors that may accumulate from simply relying on angle data by introducing fixed geometric parameters (such as hinge point spacing and horizontal component distance) and combining them with tilt angle parameters to construct a kinematic mathematical model, ensuring the reliability of the bucket tip coordinates and improving the accuracy of attitude calculation; by utilizing the inherent dimensions of the mechanical design (without the need for additional high-precision sensors), the requirement for LiDAR or binocular cameras is reduced, hardware costs are reduced, and hardware dependence is simplified; attitude calculation based on the physical constraints of the mechanical structure is applicable to different types of engineering machinery, improving the versatility and portability of the solution, and enhancing the universality and adaptability of the model in the application environment.

[0136] In this optional embodiment, as an optional implementation, the aforementioned joint position coordinate parameters include at least the lower hinge point coordinate parameters of the boom, the upper hinge point coordinate parameters of the boom, and the bucket pin coordinate parameters. Based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters, the attitude parameters of the construction machinery are calculated, including:

[0137] Calculate the bucket tip coordinate parameters based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters;

[0138] Calculate the bucket pin coordinate parameters based on the bucket tip coordinate parameters, the third distance parameter, and the multi-dimensional tilt angle parameters;

[0139] The coordinate parameters of the lower hinge point of the boom are calculated based on the distance parameters of the first horizontal component, the distance parameters of the second horizontal component, and the vehicle body rotation angle parameters.

[0140] The coordinate parameters of the upper hinge point of the boom are calculated based on the coordinate parameters of the lower hinge point of the boom, the first distance parameter, the body rotation angle parameter, and the boom rotation angle parameter.

[0141] In this embodiment of the invention, optionally, the aforementioned bucket tip coordinate parameters V(x,y,z) can be specifically calculated using a first formula, which is:

[0142] x=cosθ1·[L3cos(α+β+γ)+L2cos(α+β)+L1cosα+L0], L0=L 01 +L 02 ;

[0143] y=sinθ1·[L3cos(α+β+γ)+L2cos(α+β)+L1cosα+L0], L0=L 01 +L 02 ;

[0144] z=L3sin(α+β+γ)+L2sin(α+β)+L1sinα+H0;

[0145] Where V represents the bucket tip, θ1 represents the vehicle body rotation angle parameter, α represents the boom angle parameter, β represents the stick angle parameter, γ represents the bucket angle parameter, L1 represents the first distance parameter, L2 represents the second distance parameter, L3 represents the third distance parameter, and L... 01 L is used to represent the first horizontal component distance parameter mentioned above. 02 The distance parameter of the second horizontal component mentioned above is used to represent the height parameter of the lower hinge point of the boom mentioned above, and the height parameter is a fixed value.

[0146] Further optional, the above-mentioned bucket pin coordinate parameter Q(x) q ,y q ,z q This can be specifically calculated using the second formula, which is as follows:

[0147] x q =x-L3·cos(α+β+γ)·cosθ1;

[0148] y q =y-L3·cos(α+β+γ)·sinθ1;

[0149] z q = z - L3·sin(α+β+γ);

[0150] Where Q represents the bucket pin, and the bucket pin represents the joint point connecting the bucket and the stick.

[0151] Further optional, the above-mentioned lower hinge point coordinate parameter C(x) of the boom. c ,y c ,z c This can be specifically calculated using the third formula, which is as follows:

[0152] x c =L0cosθ1;

[0153] y c =L0sinθ1;

[0154] z c =H0;

[0155] Where C is used to represent the lower hinge point of the boom, and the lower hinge point of the boom is used to represent the joint point connecting the boom and the vehicle body;

[0156] Further optional, the above-mentioned boom hinge point coordinate parameters F(x) f ,y f ,z fThis can be specifically calculated using the fourth formula, which is as follows:

[0157] x f =x c +L1·cosα·cosθ1;

[0158] y f =y c +L1·cosα·sinθ1;

[0159] z f =L1·sinα+H0;

[0160] Where F is used to represent the upper hinge point of the boom, and the upper hinge point of the boom is used to represent the joint point connecting the boom and the stick;

[0161] It should be noted that since the bucket tip coordinate parameters are calculated based on the aforementioned multi-dimensional distance parameters, multi-dimensional tilt angle parameters, and the first formula, although in the above embodiments, the calculation of the bucket pin coordinate parameters borrows the bucket tip coordinate parameters, and the calculation of the boom upper hinge point coordinate parameters borrows the boom lower hinge point coordinate parameters, in actual calculations, the calculation methods of the borrowed coordinate points can be directly substituted into the calculation of the target coordinate points. There is no strict order of execution, such as: x q =x-L3·cos(α+β+γ)·cosθ1

[0162] x q =cosθ1·[L3cos(α+β+γ)+L2cos(α+β)+L1cosα+L0]-L3·cos(α+β+γ)·cosθ1, L0=L 01 +L 02

[0163] Based on the aforementioned multi-dimensional distance parameters and multi-dimensional tilt angle parameters, the aforementioned bucket pin coordinate parameters and boom upper hinge point coordinate parameters can be directly obtained. The specific calculation order can be determined according to the actual application scenario. The same applies to the coordinate parameters proposed later.

[0164] In this embodiment of the invention, it should be noted that the above-mentioned attitude parameters are all calculated based on the whole vehicle coordinate system with the projection point of the ground corresponding to the rotation center of the vehicle body as the origin. This process does not conflict with the subsequent process of converting the pixel coordinate system to the whole vehicle coordinate system in the process of generating multi-dimensional display images by combining multi-dimensional environmental image parameters. It can be understood that the above-mentioned attitude parameter calculation process is a theoretical kinematic mathematical modeling, and the subsequent process of converting the pixel coordinate system to the whole vehicle coordinate system is a coordinate system transformation operation proposed in practice to improve the accuracy and comprehensiveness of multi-dimensional display image generation.

[0165] As can be seen, implementing this optional embodiment can employ a strategy combining forward calculation (calculating the bucket tip coordinates step by step from the lower hinge point of the boom) and reverse calculation (calculating the joint coordinates from the bucket tip coordinates), shortening the calculation chain, improving real-time performance, and optimizing the efficiency of joint coordinate calculation. By verifying the logical relationship between the bucket tip coordinates and the bucket pin coordinates through fixed dimensional parameters (such as the third distance parameter), it avoids attitude deviations caused by single-point calculation errors and ensures data consistency. The accurate joint position coordinates provide reliable input for the subsequent generation of front views and sectional views, enhance the comprehensiveness of auxiliary information, and provide basic support for the generation of multiple views.

[0166] In this optional embodiment, as another optional implementation, the joint position coordinate parameters mentioned above also include at least one of the following: the upper hinge point coordinate parameters of the stick cylinder, the lower hinge point coordinate parameters of the stick cylinder, the upper hinge point coordinate parameters of the bucket cylinder, the lower hinge point coordinate parameters of the bucket cylinder, and the bucket connecting rod pin coordinate parameters.

[0167] In this embodiment of the invention, optionally, the upper hinge point of the stick cylinder can be set to D, the lower hinge point of the stick cylinder to E, the upper hinge point of the bucket cylinder to G, the lower hinge point of the bucket cylinder to M, and the bucket connecting rod pin to K.

[0168] Optionally, the above-mentioned calculation of the attitude parameters of the engineering machinery based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters also includes:

[0169] Collect the first included angle from the upper hinge point of the boom cylinder to the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder, and the fourth distance parameter between the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder.

[0170] Based on the coordinate parameters of the lower hinge point of the boom, the first included angle, the fourth distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter, calculate the coordinate parameters of the upper hinge point of the stick cylinder.

[0171] Collect the second included angle from the lower hinge point of the boom cylinder to the upper hinge point of the boom to the bucket pin;

[0172] Calculate the coordinate parameters of the lower hinge point of the boom cylinder based on the coordinate parameters of the upper hinge point of the boom, the second included angle, the body rotation angle parameters, and the boom rotation angle parameters;

[0173] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0174] Calculate the target coordinate parameters of the stick idler link hinge point based on the fifth distance parameter and the bucket pin coordinate parameters;

[0175] Calculate the coordinate parameters of the upper hinge point of the bucket cylinder based on the target coordinate parameters, the upper hinge point coordinate parameters of the boom, and the upper hinge point of the bucket cylinder.

[0176] Calculate the coordinate parameters of the lower hinge point of the bucket cylinder based on the target coordinate parameters, the coordinate parameters of the upper hinge point of the bucket cylinder, and the coordinate parameters of the lower hinge point of the bucket cylinder.

[0177] Calculate the bucket connecting rod pin coordinate parameters based on the bucket pin coordinate parameters, bucket tip coordinate parameters, and bucket connecting rod pin coordinate parameters.

[0178] In this embodiment of the invention, optionally, based on the point numbering set above, the first included angle is ∠DCF, and the fourth distance parameter CD = l;

[0179] Further optionally, the coordinate parameters D(x) of the upper hinge point of the boom cylinder mentioned above... d ,y d ,z d This can be specifically calculated using the fifth formula, which is as follows:

[0180] x d =x c +l·cos(∠DCF+α)·cosθ1;

[0181] y d =y c +l·cos(∠DCF+α)·sinθ1;

[0182] z d = l·sin(∠DCF+α)+H0;

[0183] For the coordinate parameters of the lower hinge point of the boom cylinder E(x) e ,y e ,z e Based on triangle EFQ, we can solve the triangle to calculate the second included angle ∠EFQ, and then calculate the coordinates of point E in the vehicle coordinate system.

[0184] x e =x f +EF·cos∠EFQ·cosα·cosθ1;

[0185] y e =y f +EF·cos∠EFQ·cosα·sinθ1;

[0186] z e =(L1+EF·cos∠EFQ)·sinα+H0;

[0187] Similarly, the coordinates of point G can be calculated using △FGN and the coordinates of point F, the coordinates of point M can be calculated using △GMN and the coordinates of point G, and the coordinates of point K can be calculated using △QKV and the coordinates of point Q.

[0188] As can be seen, implementing this optional embodiment can cover the entire kinematic relationship of the working device by adding joint coordinate calculations for cylinder hinge points, connecting rod pins, etc., providing data support for advanced functions such as collision warning and energy consumption optimization, and expanding the dimensions of attitude monitoring; the coordinate calculation results are verified by geometric constraints (fixed angles, distance parameters), for example, the cylinder coordinates are derived by using the stick idler connecting rod hinge point as the intermediate point, reducing the impact of sensor failure or noise and improving system robustness; real-time monitoring of key hinge point coordinates can help identify mechanical structural anomalies (such as cylinder displacement deviation), facilitating early maintenance and enhancing fault diagnosis capabilities.

[0189] In an optional embodiment, the aforementioned bucket angle parameter is calculated based on the third included angle from the bucket pin to the boom idler joint hinge point to the bucket cylinder lower hinge point. The calculation method for the bucket angle parameter is as follows:

[0190] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0191] Collect the sixth distance parameter between the lower hinge point of the bucket cylinder and the hinge point of the stick idler link;

[0192] The seventh distance parameter is calculated based on the fifth distance parameter, the sixth distance parameter, and the third included angle. The seventh distance parameter is used to represent the distance between the bucket pin and the lower hinge point of the bucket cylinder.

[0193] Collect the fourth included angle from the bucket tip to the bucket pin to the bucket connecting rod pin;

[0194] Collect the fifth included angle from the bucket connecting rod pin to the hinge point of the bucket pin and the stick idler connecting rod;

[0195] Collect the eighth distance parameter between the lower hinge point of the bucket cylinder and the bucket connecting rod pin;

[0196] Collect the ninth distance parameter between the bucket pin and the bucket connecting rod pin;

[0197] Calculate the bucket turning angle parameters based on the fourth included angle, the fifth included angle, the fifth distance parameter, the sixth distance parameter, the seventh distance parameter, the eighth distance parameter, and the ninth distance parameter.

[0198] In this embodiment of the invention, optionally, the boom angle parameter α, the stick angle parameter β, and the vehicle body rotation angle parameter θ1 can all be directly acquired by controlling the corresponding tilt sensor. However, in practical applications, due to various factors, the angle acquired by the bucket tilt sensor may not be the bucket angle parameter γ, but rather ∠QNM; therefore, it is necessary to first calculate the bucket angle parameter γ, based on... We can obtain:

[0199]

[0200] As can be seen, implementing this optional embodiment can improve the accuracy of angle data and solve the difficulty of indirect measurement by indirectly deriving it through the triangular geometric relationship of multiple parameters (angle, distance) when the bucket angle is difficult to measure directly due to the complexity of the linkage mechanism. It only requires the collection of fixed included angle and distance parameters (no need for dedicated angle sensors), which simplifies system deployment and reduces hardware costs. The angle logic is verified by combining the physical characteristics of the bucket linkage mechanism (such as the fourth included angle and the fifth included angle), avoiding attitude distortion caused by virtual rendering and ensuring the authenticity of the bucket attitude.

[0201] 103. Based on attitude parameters and multi-dimensional environmental image parameters, generate a multi-dimensional display image of the construction machinery. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

[0202] In this embodiment of the invention, optionally, the multi-dimensional display image described above may include at least a projection of the bucket tip landing point, a front view of the bucket, and a cross-sectional view of the construction machinery;

[0203] In this embodiment of the invention, optionally, such as Figure 3 As shown, Figure 3 This is a projection diagram of the landing point of the bucket tip, as disclosed in an embodiment of the present invention. The projection diagram of the landing point of the bucket tip indicates the projection information of the landing point of the bucket.

[0204] Further optional, such as Figure 4 As shown, Figure 4 This is a front view of a bucket disclosed in an embodiment of the present invention. The front view of the bucket indicates the landing point projection information and the bucket elevation information.

[0205] Further optional, such as Figure 5 As shown, Figure 5 This is a cross-sectional view of an engineering machine disclosed in an embodiment of the present invention. The cross-sectional view of the engineering machine indicates the real-time highest position information and the distance information of the bucket relative to the corresponding cab of the engineering machine. Figure 5 The cross-sectional diagram of the construction machinery indicates that the real-time highest position is 3.61m, and the distance between the bucket and the corresponding cab of the construction machinery is 4.32m.

[0206] Further optionally, the multi-dimensional display image described above may also include the multi-dimensional distance parameters and / or multi-dimensional tilt angle parameters described above;

[0207] As can be seen, implementing the embodiments of the present invention enables real-time calculation of the attitude parameters of the construction machinery, including at least the bucket tip coordinate parameters and the joint position coordinate parameters of at least one mechanical joint, based on the collected multi-dimensional tilt angle parameters of the construction machinery. This achieves real-time mathematical modeling of the physical tilt angle of the construction machinery, and solves for the real-time attitude of the construction machinery, realizing the real-time positioning of each working component of the construction machinery. Compared with the convolutional neural network model used in the prior art, no model training is required, saving model training and maintenance costs. It also reduces the requirements for technical personnel and training data, and is unaffected by the working environment (such as smoke, dust, rain, snow, and other harsh weather conditions). Starting from the attitude, the system guides the generation of multi-dimensional display images of the construction machinery. By further combining the multi-dimensional environmental image parameters collected from the construction machinery within a preset range, it generates at least one or more of the following multi-dimensional display images of the construction machinery: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information. This improves the accuracy and comprehensiveness of the construction machinery image display, allowing operators to have a more intuitive and comprehensive understanding of the real-time spatial attitude of the construction machinery. While reducing human visual errors, it also further avoids collisions between the bucket and obstacles, improving the operational precision and safety of fine chemical conditions (such as trenching, grooving, crushing, etc.).

[0208] Example 2

[0209] Please see Figure 6 , Figure 6 This is a flowchart illustrating another posture-guided image fusion and display method for engineering machinery disclosed in an embodiment of the present invention. Figure 6 The described posture-guided image fusion display method for construction machinery can be applied to construction machinery, including buckets, and can also be applied to intelligent devices associated with the construction machinery. These intelligent devices include, but are not limited to, one or more of battery devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. This invention does not limit the application of these methods. Figure 6 As shown, the posture-guided image fusion and display method for engineering machinery may include the following operations:

[0210] 201. Collect multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within the preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions within the preset range of the construction machinery.

[0211] 202. Based on the multi-dimensional tilt angle parameters, calculate the attitude parameters of the construction machinery. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery.

[0212] In this embodiment of the invention, for supplementary explanations of steps 201 and 202, please refer to the supplementary explanations of steps 101 and 102 in Embodiment 1. This embodiment of the invention will not repeat these explanations.

[0213] 203. Determine the pixel matrix parameters corresponding to the multi-dimensional environmental image parameters and the image acquisition intrinsic parameter matrix parameters of the corresponding sensing components. The image acquisition intrinsic parameter matrix parameters are used to indicate at least the image acquisition focal length and the image acquisition principal point. The pixel matrix parameters correspond to the pixel coordinate system.

[0214] 204. Based on the pixel matrix parameters and the image acquisition intrinsic parameter matrix parameters, generate the perception matrix parameters of the sensing component, and the perception matrix parameters correspond to the sensing coordinate system of the sensing component;

[0215] 205. Determine the rotation matrix parameters and translation vector parameters corresponding to the sensing component. The rotation matrix parameters are calculated based on the Euler angles of the sensing component, and the translation vector parameters are used to represent the position of the optical center of the sensing component in the vehicle coordinate system of the engineering machinery.

[0216] 206. Based on the perception matrix parameters, rotation matrix parameters, and translation vector parameters, generate the vehicle matrix parameters of the construction machinery. The vehicle matrix parameters correspond to the vehicle coordinate system of the construction machinery.

[0217] 207. Based on the vehicle matrix parameters, attitude parameters and multi-dimensional environmental image parameters, generate a multi-dimensional display image of the construction machinery. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

[0218] In this embodiment of the invention, optionally, the camera coordinate system is based on the optical center of the camera as the origin, the z-axis direction as the optical axis direction, the y-axis direction as perpendicular to the image plane downwards, and the x-axis direction as perpendicular to the image plane to the right.

[0219] The aforementioned perceptual matrix parameters are generated as follows:

[0220]

[0221] in, Used to represent the aforementioned perception matrix parameters, The pixel matrix parameters mentioned above are used to represent the image acquisition intrinsic parameter matrix parameters mentioned above, f is used to represent the physical focal length, B is used to represent the camera baseline distance, and d is used to represent the parallax. x f is used to represent the focal length component of the camera in the x-direction of the pixel coordinate system. y Used to represent the focal length component of the camera in the y-direction of the pixel coordinate system, and f x It is calculated based on the ratio between the physical focal length and the physical width of the pixel, f. y It is calculated based on the ratio between the physical focal length and the pixel physical height, c x Used to represent the offset of the camera's optical axis in the x-direction of the pixel coordinate system, c y Used to represent the offset of the camera's optical axis in the y-direction of the pixel coordinate system;

[0222] Furthermore, the above-mentioned vehicle matrix parameters are generated as follows:

[0223]

[0224] in, Used to represent the above-mentioned vehicle matrix parameters, R is used to represent the aforementioned perception matrix parameters. 3×3 The parameter t is used to represent the rotation matrix parameters mentioned above. 3×1 Used to represent the translation vector parameters mentioned above; assuming the camera is equipped with Euler angle information: pitch angle is θ, yaw angle is... Roll angle φ;

[0225] but:

[0226]

[0227] As can be seen, implementing the embodiments of the present invention can align the pixels in the environmental image with the actual spatial position of the bucket through coordinate system transformation (pixel → sensing component → vehicle coordinate system), eliminate viewpoint distortion, ensure the positioning accuracy of the landing point projection, and achieve precise image fusion; based on the fusion of attitude data and environmental images using vehicle matrix parameters, virtual markers (such as elevation markers) are seamlessly matched with the actual scene, enhancing the operator's spatial perception and improving display intuitiveness; the predefined intrinsic parameter matrix (focal length, principal point) and rotation matrix simplify the real-time conversion process, improve image generation efficiency, and reduce computational redundancy.

[0228] In this embodiment of the invention, as an optional implementation, the above-mentioned generation of a multi-dimensional display image of construction machinery based on vehicle matrix parameters, attitude parameters, and multi-dimensional environmental image parameters includes:

[0229] Collect millimeter-wave radar detection parameters, which are used to represent the distribution of obstacles within a preset range of the engineering machinery;

[0230] The millimeter-wave radar detection parameters are correlated with the bucket tip coordinate parameters to generate obstacle correlation parameters, which are used to represent the distance and positional relationship between obstacles and the bucket.

[0231] Based on vehicle matrix parameters, attitude parameters, multi-dimensional environmental image parameters, and obstacle association parameters, a multi-dimensional display image of the construction machinery containing obstacle warning signs is generated.

[0232] Among them, the millimeter-wave radar detection parameters are generated by controlling the millimeter-wave radar to transmit detection signals and receive reflected signals, and are used to obtain the distribution range and shape of obstacles within a preset range in real time; the obstacle association parameters are obtained by calculating the spatial distance between the obstacle coordinates in the millimeter-wave radar detection parameters and the bucket tip coordinate parameters, and are used to determine whether the obstacle is within the movement trajectory range of the bucket; the obstacle warning sign marks the distribution of obstacles in a multi-dimensional display image through image generation technology, and dynamically adjusts the display mode of the sign based on the distance relationship.

[0233] In this embodiment of the invention, optionally, for obstacle detection, a millimeter-wave radar can transmit frequency-modulated waves and receive reflected signals to generate point cloud data and mark the distance, azimuth angle and outline of the obstacle;

[0234] For correlation analysis, the obstacle coordinates can be transformed to the vehicle coordinate system, and the Euclidean distance between the obstacle and the bucket tip can be calculated; if the distance is less than the safety threshold (e.g., 2 meters), it is determined to be a collision risk obstacle.

[0235] For dynamic warning signs, obstacle outlines (red polygons) can be overlaid on the environmental image; the signs can be dynamically adjusted according to the distance, such as high risk (distance < 0.5 meters): flashing red warning box; medium risk (0.5 meters ≤ distance < 2 meters): yellow static box; safe (distance ≥ 2 meters): green semi-transparent box.

[0236] As can be seen, implementing this optional embodiment can detect obstacle distribution through millimeter-wave radar, perform spatial correlation analysis in conjunction with the bucket tip coordinates, dynamically mark collision risk areas (such as obstacles within the movement trajectory), provide early warnings to operators to avoid danger, and achieve an upgrade in proactive safety protection; obstacle correlation parameters filter non-threat targets (such as distant static objects) through distance relationships, focus on high-risk obstacles, reduce invalid alarm interference, and lower the false alarm rate; obstacle warning signs (such as graded color boxes) are superimposed on multi-dimensional display images to uniformly present environmental, attitude, and risk information, simplify the operator's decision-making path, and enhance information integration.

[0237] Example 3

[0238] Please see Figure 7 , Figure 7 This is a schematic diagram of a posture-guided image fusion display device for construction machinery disclosed in an embodiment of the present invention. This posture-guided image fusion display device can be applied to construction machinery, including buckets, and can also be applied to intelligent devices associated with the construction machinery. These intelligent devices include, but are not limited to, one or more of battery devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices; the embodiments of the present invention do not limit the application to these devices. Figure 7 As shown, the posture-guided image fusion display device for engineering machinery may include:

[0239] The acquisition module 301 is used to acquire multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within a preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions within a preset range of the construction machinery.

[0240] The calculation module 302 is used to calculate the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery.

[0241] The generation module 303 is used to generate a multi-dimensional display image of the construction machinery based on attitude parameters and multi-dimensional environmental image parameters. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

[0242] As can be seen, implementing the embodiments of the present invention enables real-time calculation of the attitude parameters of the construction machinery, including at least the bucket tip coordinate parameters and the joint position coordinate parameters of at least one mechanical joint, based on the collected multi-dimensional tilt angle parameters of the construction machinery. This achieves real-time mathematical modeling of the physical tilt angle of the construction machinery, and solves for the real-time attitude of the construction machinery, realizing the real-time positioning of each working component of the construction machinery. Compared with the convolutional neural network model used in the prior art, no model training is required, saving model training and maintenance costs. It also reduces the requirements for technical personnel and training data, and is unaffected by the working environment (such as smoke, dust, rain, snow, and other harsh weather conditions). Starting from the attitude, the system guides the generation of multi-dimensional display images of the construction machinery. By further combining the multi-dimensional environmental image parameters collected from the construction machinery within a preset range, it generates at least one or more of the following multi-dimensional display images of the construction machinery: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information. This improves the accuracy and comprehensiveness of the construction machinery image display, allowing operators to have a more intuitive and comprehensive understanding of the real-time spatial attitude of the construction machinery. While reducing human visual errors, it also further avoids collisions between the bucket and obstacles, improving the operational precision and safety of fine chemical conditions (such as trenching, grooving, crushing, etc.).

[0243] In this embodiment of the invention, as an optional implementation, the aforementioned multi-dimensional tilt angle parameters include vehicle body rotation angle parameters, boom rotation angle parameters, stick rotation angle parameters, and bucket rotation angle parameters. The specific method by which the calculation module 302 calculates the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters includes:

[0244] Collect multi-dimensional distance parameters of the construction machinery. The multi-dimensional distance parameters include at least the first distance parameter between the upper hinge point and the lower hinge point of the boom, the second distance parameter between the upper hinge point of the boom and the bucket pin, the third distance parameter between the bucket pin and the bucket tip, the first horizontal component distance parameter between the body rotation center and the lower hinge point of the boom, and the second horizontal component distance parameter between the lower hinge point of the boom cylinder and the lower hinge point of the boom.

[0245] The attitude parameters of the engineering machinery are calculated based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters.

[0246] As can be seen, implementing this optional embodiment can eliminate the errors that may accumulate from simply relying on angle data by introducing fixed geometric parameters (such as hinge point spacing and horizontal component distance) and combining them with tilt angle parameters to construct a kinematic mathematical model, ensuring the reliability of the bucket tip coordinates and improving the accuracy of attitude calculation; by utilizing the inherent dimensions of the mechanical design (without the need for additional high-precision sensors), the requirement for LiDAR or binocular cameras is reduced, hardware costs are reduced, and hardware dependence is simplified; attitude calculation based on the physical constraints of the mechanical structure is applicable to different types of engineering machinery, improving the versatility and portability of the solution, and enhancing the universality and adaptability of the model in the application environment.

[0247] In this optional embodiment, as an optional implementation, the aforementioned joint position coordinate parameters include at least the lower hinge point coordinate parameters of the boom, the upper hinge point coordinate parameters of the boom, and the bucket pin coordinate parameters. The calculation module 302 calculates the attitude parameters of the construction machinery based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters in the following specific ways:

[0248] Calculate the bucket tip coordinate parameters based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters;

[0249] Calculate the bucket pin coordinate parameters based on the bucket tip coordinate parameters, the third distance parameter, and the multi-dimensional tilt angle parameters;

[0250] The coordinate parameters of the lower hinge point of the boom are calculated based on the distance parameters of the first horizontal component, the distance parameters of the second horizontal component, and the vehicle body rotation angle parameters.

[0251] The coordinate parameters of the upper hinge point of the boom are calculated based on the coordinate parameters of the lower hinge point of the boom, the first distance parameter, the body rotation angle parameter, and the boom rotation angle parameter.

[0252] As can be seen, implementing this optional embodiment can employ a strategy combining forward calculation (calculating the bucket tip coordinates step by step from the lower hinge point of the boom) and reverse calculation (calculating the joint coordinates from the bucket tip coordinates), shortening the calculation chain, improving real-time performance, and optimizing the efficiency of joint coordinate calculation. By verifying the logical relationship between the bucket tip coordinates and the bucket pin coordinates through fixed dimensional parameters (such as the third distance parameter), it avoids attitude deviations caused by single-point calculation errors and ensures data consistency. The accurate joint position coordinates provide reliable input for the subsequent generation of front views and sectional views, enhance the comprehensiveness of auxiliary information, and provide basic support for the generation of multiple views.

[0253] In this optional embodiment, as another optional implementation, the joint position coordinate parameters mentioned above also include at least one of the following: the upper hinge point coordinate parameters of the stick cylinder, the lower hinge point coordinate parameters of the stick cylinder, the upper hinge point coordinate parameters of the bucket cylinder, the lower hinge point coordinate parameters of the bucket cylinder, and the bucket connecting rod pin coordinate parameters.

[0254] Optionally, the calculation module 302 may calculate the attitude parameters of the engineering machinery based on multi-dimensional distance parameters and multi-dimensional tilt angle parameters in the following specific ways:

[0255] Collect the first included angle from the upper hinge point of the boom cylinder to the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder, and the fourth distance parameter between the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder.

[0256] Based on the coordinate parameters of the lower hinge point of the boom, the first included angle, the fourth distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter, calculate the coordinate parameters of the upper hinge point of the stick cylinder.

[0257] Collect the second included angle from the lower hinge point of the boom cylinder to the upper hinge point of the boom to the bucket pin;

[0258] Calculate the coordinate parameters of the lower hinge point of the boom cylinder based on the coordinate parameters of the upper hinge point of the boom, the second included angle, the body rotation angle parameters, and the boom rotation angle parameters;

[0259] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0260] Calculate the target coordinate parameters of the stick idler link hinge point based on the fifth distance parameter and the bucket pin coordinate parameters;

[0261] Calculate the coordinate parameters of the upper hinge point of the bucket cylinder based on the target coordinate parameters, the upper hinge point coordinate parameters of the boom, and the upper hinge point of the bucket cylinder.

[0262] Calculate the coordinate parameters of the lower hinge point of the bucket cylinder based on the target coordinate parameters, the coordinate parameters of the upper hinge point of the bucket cylinder, and the coordinate parameters of the lower hinge point of the bucket cylinder.

[0263] Calculate the bucket connecting rod pin coordinate parameters based on the bucket pin coordinate parameters, bucket tip coordinate parameters, and bucket connecting rod pin coordinate parameters.

[0264] As can be seen, implementing this optional embodiment can cover the entire kinematic relationship of the working device by adding joint coordinate calculations for cylinder hinge points, connecting rod pins, etc., providing data support for advanced functions such as collision warning and energy consumption optimization, and expanding the dimensions of attitude monitoring; the coordinate calculation results are verified by geometric constraints (fixed angles, distance parameters), for example, the cylinder coordinates are derived by using the stick idler connecting rod hinge point as the intermediate point, reducing the impact of sensor failure or noise and improving system robustness; real-time monitoring of key hinge point coordinates can help identify mechanical structural anomalies (such as cylinder displacement deviation), facilitating early maintenance and enhancing fault diagnosis capabilities.

[0265] In an optional embodiment, the aforementioned bucket angle parameter is calculated based on the third included angle from the bucket pin to the boom idler joint hinge point to the bucket cylinder lower hinge point. The calculation method for the bucket angle parameter is as follows:

[0266] Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link;

[0267] Collect the sixth distance parameter between the lower hinge point of the bucket cylinder and the hinge point of the stick idler link;

[0268] The seventh distance parameter is calculated based on the fifth distance parameter, the sixth distance parameter, and the third included angle. The seventh distance parameter is used to represent the distance between the bucket pin and the lower hinge point of the bucket cylinder.

[0269] Collect the fourth included angle from the bucket tip to the bucket pin to the bucket connecting rod pin;

[0270] Collect the fifth included angle from the bucket connecting rod pin to the hinge point of the bucket pin and the stick idler connecting rod;

[0271] Collect the eighth distance parameter between the lower hinge point of the bucket cylinder and the bucket connecting rod pin;

[0272] Collect the ninth distance parameter between the bucket pin and the bucket connecting rod pin;

[0273] Calculate the bucket turning angle parameters based on the fourth included angle, the fifth included angle, the fifth distance parameter, the sixth distance parameter, the seventh distance parameter, the eighth distance parameter, and the ninth distance parameter.

[0274] As can be seen, implementing this optional embodiment can improve the accuracy of angle data and solve the difficulty of indirect measurement by indirectly deriving it through the triangular geometric relationship of multiple parameters (angle, distance) when the bucket angle is difficult to measure directly due to the complexity of the linkage mechanism. It only requires the collection of fixed included angle and distance parameters (no need for dedicated angle sensors), which simplifies system deployment and reduces hardware costs. The angle logic is verified by combining the physical characteristics of the bucket linkage mechanism (such as the fourth included angle and the fifth included angle), avoiding attitude distortion caused by virtual rendering and ensuring the authenticity of the bucket attitude.

[0275] In another alternative embodiment, such as Figure 8 As shown, the device also includes:

[0276] The determination module 304 is used to determine the pixel matrix parameters corresponding to the multi-dimensional environmental image parameters and the image acquisition intrinsic parameter matrix parameters of the corresponding sensing component before the generation module 303 generates a multi-dimensional display image of the engineering machinery based on the attitude parameters and multi-dimensional environmental image parameters. The image acquisition intrinsic parameter matrix parameters are used to indicate at least the image acquisition focal length and the image acquisition principal point. The pixel matrix parameters correspond to the pixel coordinate system.

[0277] The generation module 303 is also used to generate the perception matrix parameters of the sensing component based on the pixel matrix parameters and the image acquisition intrinsic parameter matrix parameters, wherein the perception matrix parameters correspond to the perception coordinate system of the sensing component.

[0278] The determination module 304 is also used to determine the rotation matrix parameters and translation vector parameters corresponding to the sensing component. The rotation matrix parameters are calculated based on the Euler angles of the sensing component, and the translation vector parameters are used to represent the position of the optical center corresponding to the sensing component in the vehicle coordinate system of the engineering machinery.

[0279] The generation module 303 is also used to generate the vehicle matrix parameters of the construction machinery based on the perception matrix parameters, rotation matrix parameters, and translation vector parameters. The vehicle matrix parameters correspond to the vehicle coordinate system of the construction machinery.

[0280] Optionally, the generation module 303 generates multi-dimensional display images of the construction machinery based on attitude parameters and multi-dimensional environmental image parameters in the following specific ways:

[0281] Based on the vehicle matrix parameters, attitude parameters, and multi-dimensional environmental image parameters, a multi-dimensional display image of the construction machinery is generated.

[0282] As can be seen, implementing the embodiments of the present invention can align the pixels in the environmental image with the actual spatial position of the bucket through coordinate system transformation (pixel → sensing component → vehicle coordinate system), eliminate viewpoint distortion, ensure the positioning accuracy of the landing point projection, and achieve precise image fusion; based on the fusion of attitude data and environmental images using vehicle matrix parameters, virtual markers (such as elevation markers) are seamlessly matched with the actual scene, enhancing the operator's spatial perception and improving display intuitiveness; the predefined intrinsic parameter matrix (focal length, principal point) and rotation matrix simplify the real-time conversion process, improve image generation efficiency, and reduce computational redundancy.

[0283] In this optional embodiment, as an optional implementation, the generation module 303 generates a multi-dimensional display image of the construction machinery based on the vehicle matrix parameters, attitude parameters, and multi-dimensional environmental image parameters in the following specific ways:

[0284] Collect millimeter-wave radar detection parameters, which are used to represent the distribution of obstacles within a preset range of the engineering machinery;

[0285] The millimeter-wave radar detection parameters are correlated with the bucket tip coordinate parameters to generate obstacle correlation parameters, which are used to represent the distance and positional relationship between obstacles and the bucket.

[0286] Based on vehicle matrix parameters, attitude parameters, multi-dimensional environmental image parameters, and obstacle association parameters, a multi-dimensional display image of the construction machinery containing obstacle warning signs is generated.

[0287] Among them, the millimeter-wave radar detection parameters are generated by controlling the millimeter-wave radar to transmit detection signals and receive reflected signals, and are used to obtain the distribution range and shape of obstacles within a preset range in real time; the obstacle association parameters are obtained by calculating the spatial distance between the obstacle coordinates in the millimeter-wave radar detection parameters and the bucket tip coordinate parameters, and are used to determine whether the obstacle is within the movement trajectory range of the bucket; the obstacle warning sign marks the distribution of obstacles in a multi-dimensional display image through image generation technology, and dynamically adjusts the display mode of the sign based on the distance relationship.

[0288] As can be seen, implementing this optional embodiment can detect obstacle distribution through millimeter-wave radar, perform spatial correlation analysis in conjunction with the bucket tip coordinates, dynamically mark collision risk areas (such as obstacles within the movement trajectory), provide early warnings to operators to avoid danger, and achieve an upgrade in proactive safety protection; obstacle correlation parameters filter non-threat targets (such as distant static objects) through distance relationships, focus on high-risk obstacles, reduce invalid alarm interference, and lower the false alarm rate; obstacle warning signs (such as graded color boxes) are superimposed on multi-dimensional display images to uniformly present environmental, attitude, and risk information, simplify the operator's decision-making path, and enhance information integration.

[0289] Example 4

[0290] Please see Figure 9 , Figure 9 This is a schematic diagram of another posture-guided image fusion display device for construction machinery disclosed in this invention. This posture-guided image fusion display device for construction machinery can be applied to construction machinery, including buckets, and can also be applied to intelligent devices associated with the construction machinery. These intelligent devices include, but are not limited to, one or more of battery devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices; this invention does not limit the application to these devices. Figure 9 As shown, the posture-guided image fusion display device for engineering machinery may include:

[0291] Memory 401 that stores executable program code.

[0292] Processor 402 coupled to memory 401.

[0293] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the posture-guided engineering machinery image fusion display method described in Embodiment 1 or Embodiment 2 of the present invention.

[0294] Example 5

[0295] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the posture-guided engineering machinery image fusion display method described in Embodiment 1 or Embodiment 2 of this invention.

[0296] Example 6

[0297] This invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the posture-guided engineering machinery image fusion display method described in Embodiment 1 or Embodiment 2.

[0298] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0299] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0300] Finally, it should be noted that the posture-guided image fusion display method and apparatus for engineering machinery disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image fusion and display of engineering machinery based on posture guidance, characterized in that, The method is applied to construction machinery, which includes a bucket, and the method includes: The system collects multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within a preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions of the construction machinery within the preset range. Based on the multi-dimensional tilt angle parameters, the attitude parameters of the construction machinery are calculated. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery. Based on the attitude parameters and the multi-dimensional environmental image parameters, a multi-dimensional display image of the construction machinery is generated. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

2. The image fusion and display method for engineering machinery based on attitude guidance according to claim 1, characterized in that, The multi-dimensional tilt angle parameters include vehicle body rotation angle parameters, boom rotation angle parameters, stick rotation angle parameters, and bucket rotation angle parameters. The calculation of the engineering machinery's attitude parameters based on these multi-dimensional tilt angle parameters includes: The multi-dimensional distance parameters of the construction machinery are collected. The multi-dimensional distance parameters include at least a first distance parameter between the upper hinge point and the lower hinge point of the boom, a second distance parameter between the upper hinge point of the boom and the bucket pin, a third distance parameter between the bucket pin and the bucket tip, a first horizontal component distance parameter between the vehicle rotation center and the lower hinge point of the boom, and a second horizontal component distance parameter between the lower hinge point of the boom cylinder and the lower hinge point of the boom. The attitude parameters of the engineering machinery are calculated based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters.

3. The image fusion and display method for engineering machinery based on attitude guidance according to claim 2, characterized in that, The joint position coordinate parameters include at least the lower hinge point coordinate parameters of the boom, the upper hinge point coordinate parameters of the boom, and the bucket pin coordinate parameters. The calculation of the attitude parameters of the construction machinery based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters includes: Based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters, calculate the bucket tip coordinate parameters of the bucket; The bucket tip coordinate parameters are calculated based on the bucket tip coordinate parameters, the third distance parameters, and the multi-dimensional tilt angle parameters. The coordinate parameters of the lower hinge point of the boom are calculated based on the first horizontal component distance parameter, the second horizontal component distance parameter, and the vehicle body rotation angle parameter. The coordinate parameters of the upper hinge point of the boom are calculated based on the coordinate parameters of the lower hinge point of the boom, the first distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter.

4. The image fusion and display method for engineering machinery based on attitude guidance according to claim 3, characterized in that, The joint position coordinate parameters also include at least one of the following: the upper hinge point coordinate parameters of the stick cylinder, the lower hinge point coordinate parameters of the stick cylinder, the upper hinge point coordinate parameters of the bucket cylinder, the lower hinge point coordinate parameters of the bucket cylinder, and the bucket connecting rod pin coordinate parameters. Furthermore, the step of calculating the attitude parameters of the engineering machinery based on the multi-dimensional distance parameters and the multi-dimensional tilt angle parameters further includes: Collect the first included angle from the upper hinge point of the boom cylinder to the lower hinge point of the boom cylinder and the fourth distance parameter between the lower hinge point of the boom cylinder and the upper hinge point of the boom cylinder; The coordinate parameters of the upper hinge point of the boom cylinder are calculated based on the lower hinge point coordinate parameters of the boom, the first included angle, the fourth distance parameter, the vehicle body rotation angle parameter, and the boom rotation angle parameter. The second included angle from the lower hinge point of the boom cylinder to the upper hinge point of the boom to the bucket pin is measured. The coordinate parameters of the lower hinge point of the boom cylinder are calculated based on the coordinate parameters of the upper hinge point of the boom, the second included angle, the rotation angle parameters of the vehicle body, and the boom rotation angle parameters. Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link; Calculate the target coordinate parameters of the stick idler link hinge point based on the fifth distance parameter and the bucket pin coordinate parameters; Based on the target coordinate parameters, the boom upper hinge point coordinate parameters, and the bucket cylinder upper hinge point, calculate the bucket cylinder upper hinge point coordinate parameters; Based on the target coordinate parameters, the coordinate parameters of the upper hinge point of the bucket cylinder, and the lower hinge point of the bucket cylinder, calculate the coordinate parameters of the lower hinge point of the bucket cylinder. The coordinate parameters of the bucket connecting rod pin are calculated based on the coordinate parameters of the bucket pin, the coordinate parameters of the bucket tip, and the bucket connecting rod pin.

5. The image fusion and display method for engineering machinery based on attitude guidance according to claim 2, characterized in that, The bucket rotation angle parameter is calculated based on the third included angle from the bucket pin to the boom idler joint hinge point to the bucket cylinder lower hinge point. The calculation method for the bucket rotation angle parameter is as follows: Collect the fifth distance parameter between the bucket pin and the hinge point of the stick idler link; Collect the sixth distance parameter between the lower hinge point of the bucket cylinder and the hinge point of the stick idler connecting rod; Based on the fifth distance parameter, the sixth distance parameter, and the third included angle, a seventh distance parameter is calculated. The seventh distance parameter is used to represent the distance between the bucket pin and the lower hinge point of the bucket cylinder. Collect the fourth included angle from the bucket tip to the bucket pin to the bucket connecting rod pin; The fifth included angle between the bucket connecting rod pin and the hinge point of the bucket pin and the stick idler connecting rod is collected; Collect the eighth distance parameter between the lower hinge point of the bucket cylinder and the bucket connecting rod pin; Collect the ninth distance parameter between the bucket pin and the bucket connecting rod pin; The bucket turning angle parameter is calculated based on the fourth included angle, the fifth included angle, the fifth distance parameter, the sixth distance parameter, the seventh distance parameter, the eighth distance parameter, and the ninth distance parameter.

6. The posture-guided image fusion and display method for engineering machinery according to any one of claims 1-5, characterized in that, Before generating the multi-dimensional display image of the engineering machinery based on the attitude parameters and the multi-dimensional environmental image parameters, the method further includes: Determine the pixel matrix parameters corresponding to the multi-dimensional environmental image parameters and the image acquisition intrinsic parameter matrix parameters of the corresponding sensing component. The image acquisition intrinsic parameter matrix parameters are at least used to indicate the image acquisition focal length and the image acquisition principal point. The pixel matrix parameters correspond to the pixel coordinate system. Based on the pixel matrix parameters and the image acquisition intrinsic parameter matrix parameters, the perception matrix parameters of the sensing component are generated, and the perception matrix parameters correspond to the perception coordinate system of the sensing component. The rotation matrix parameters and translation vector parameters corresponding to the sensing component are determined. The rotation matrix parameters are calculated based on the Euler angles of the sensing component, and the translation vector parameters are used to represent the position of the optical center corresponding to the sensing component in the vehicle coordinate system of the engineering machinery. Based on the perception matrix parameters, the rotation matrix parameters, and the translation vector parameters, the vehicle matrix parameters of the construction machinery are generated, and the vehicle matrix parameters correspond to the vehicle coordinate system of the construction machinery. And, generating a multi-dimensional display image of the engineering machinery based on the attitude parameters and the multi-dimensional environmental image parameters includes: Based on the vehicle matrix parameters, the attitude parameters, and the multi-dimensional environmental image parameters, a multi-dimensional display image of the construction machinery is generated.

7. The image fusion and display method for engineering machinery based on attitude guidance according to claim 6, characterized in that, The step of generating a multi-dimensional display image of the construction machinery based on the vehicle matrix parameters, the attitude parameters, and the multi-dimensional environmental image parameters includes: Collect millimeter-wave radar detection parameters, which are used to represent the distribution of obstacles within the preset range of the engineering machinery; The millimeter-wave radar detection parameters are correlated with the bucket tip coordinate parameters to generate obstacle correlation parameters, which are used to represent the distance and positional relationship between the obstacle and the bucket. Based on the vehicle matrix parameters, the attitude parameters, the multi-dimensional environmental image parameters, and the obstacle association parameters, a multi-dimensional display image of the construction machinery containing obstacle warning indicators is generated; The millimeter-wave radar detection parameters are generated by controlling the millimeter-wave radar to transmit detection signals and receive reflected signals, and are used to obtain the distribution range and shape of obstacles within the preset range in real time. The obstacle association parameters are obtained by calculating the spatial distance between the obstacle coordinates in the millimeter-wave radar detection parameters and the bucket tip coordinate parameters, and are used to determine whether the obstacle is located within the movement trajectory range of the bucket. The obstacle warning markers mark the obstacle distribution in the multi-dimensional display image through image generation technology, and dynamically adjust the display mode of the markers based on the distance relationship.

8. A posture-guided image fusion display device for engineering machinery, characterized in that, The device is applied to construction machinery, which includes a bucket, and the device includes: The acquisition module is used to acquire multi-dimensional tilt angle parameters of the construction machinery and multi-dimensional environmental image parameters within a preset range of the construction machinery. The multi-dimensional tilt angle parameters are used to indicate the real-time operating tilt angle of the multi-dimensional components of the construction machinery, and the multi-dimensional environmental image parameters are used to indicate the real-time environmental conditions of the construction machinery within the preset range. The calculation module is used to calculate the attitude parameters of the construction machinery based on the multi-dimensional tilt angle parameters. The attitude parameters include at least the bucket tip coordinate parameters of the bucket and the joint position coordinate parameters of at least one mechanical joint of the construction machinery. The generation module is used to generate a multi-dimensional display image of the construction machinery based on the attitude parameters and the multi-dimensional environmental image parameters. The multi-dimensional display image is used to indicate at least one or more of the following: the landing point projection information of the bucket, the bucket elevation information, the distance information of the bucket relative to the corresponding cab of the construction machinery, and the real-time highest position information.

9. A posture-guided image fusion display device for engineering machinery, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the posture-guided image fusion display method for engineering machinery as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the posture-guided engineering machinery image fusion display method as described in any one of claims 1-7.

Images