Auxiliary control system, method and engineering machinery for engineering machinery

Abstract

The embodiments of the present application provide an auxiliary control system, method, engineering machinery and machine-readable storage medium for engineering machinery. The auxiliary control system for engineering machinery includes: a display device, configured to acquire and project a display image; a reflection device, the reflection angle of the reflection device is adjustable, and is configured to reflect the display image to the windshield of the engineering machinery; an image acquisition device, configured to acquire an image of the driver of the engineering machinery; a processing device, communicating with the image acquisition device, and configured to: determine the eye position of the driver based on the image of the driver; adjust the reflection angle of the reflection device based on the eye position to adjust the imaging position of the display image on the windshield. Through the above technical solution, the difficulty of leveling the engineering machinery is reduced and the efficiency of leveling the engineering machinery is improved.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to an auxiliary control system, method, engineering machinery and machine-readable storage medium for engineering machinery. Background Art

[0002] Ground leveling (abbreviated as ground leveling) is a common operation mode performed by construction machinery such as excavators, bulldozers, and loaders, which has high requirements on the driver's driving skills. For example, when an excavator is leveling the ground, not only does it require the left and right crawlers of the excavator to be on the same plane, which is called "sitting flat", but it also requires cyclic operation to ensure that the starting position of the working device in each cycle is on the same plane.

[0003] In the prior art, when engineering machinery equipment is performing leveling operations, the driver is often required to use the naked eye to judge whether the equipment is in a "level" state and the starting point of each cycle. Once the equipment is not in a "level" state, the ground on which the leveling operation is completed will become tilted as a whole, reducing the quality of the leveling operation. If the starting position of the working device in each cycle is not on the same straight line, repeated supplementary operations are required, which increases the overall difficulty of the leveling operation and reduces the operation efficiency. Summary of the invention

[0004] In order to at least partially solve the above-mentioned problems existing in the prior art, an object of the embodiments of the present application is to provide an auxiliary control system, method, engineering machinery and machine-readable storage medium for engineering machinery.

[0005] In order to achieve the above-mentioned object, the present application provides, in a first aspect, an auxiliary control system for an engineering machine, comprising:

[0006] A display device configured to acquire and project a display image;

[0007] A reflecting device, the reflecting angle of the reflecting device being adjustable, and configured to reflect the displayed image onto a windshield of the engineering machinery;

[0008] an image acquisition device configured to acquire an image of a driver of the construction machinery;

[0009] The processing device, in communication with the image acquisition device, is configured to:

[0010] determining the eye position of the driver based on the image of the driver;

[0011] The reflection angle of the reflection device is adjusted according to the position of the human eye to adjust the imaging position of the displayed image on the windshield.

[0012] Optionally, the processing device adjusts the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the displayed image on the windshield, including:

[0013] Obtain the height difference between the human eye position and the preset human eye position;

[0014] Determine the target reflection angle of the reflection device according to the height difference;

[0015] The reflection angle of the reflection device is adjusted to the target reflection angle to adjust the imaging position of the display image on the windshield so that the adjusted imaging position is located directly in front of the driver's line of sight.

[0016] Optionally, the processing device determines the target reflection angle of the reflection device according to the height difference, including:

[0017] Determine whether the height difference is greater than the height threshold;

[0018] When it is determined that the height difference value is greater than the height threshold, a target reflection angle of the reflection device is determined according to the height difference value.

[0019] Optionally, the reflecting device includes a plane reflecting mirror, a curved reflecting mirror and an adjusting mechanism;

[0020] The flat reflector is configured to reflect the display image to the curved reflector;

[0021] The curved reflector is configured to magnify and reflect the displayed image onto the windshield;

[0022] The adjustment mechanism communicates with the processing device and is configured to adjust the reflection angle of the plane reflector and / or the curved reflector according to the control instruction of the processing device.

[0023] Optionally, the auxiliary control system further includes:

[0024] an inclination detection device, in communication with the processing device, configured to detect an inclination angle of the construction machinery;

[0025] The processing device is further configured to:

[0026] Determine whether the construction machinery is in a level position based on the tilt angle;

[0027] When it is determined that the construction machinery is not in a level position, a prompt message is issued.

[0028] Optionally, the processing device is configured to determine whether the construction machine is in a sitting level state according to the tilt angle, including:

[0029] Get the angle difference between the tilt angle and the preset angle;

[0030] Determine whether the angle difference is greater than an angle threshold;

[0031] When the angle difference is greater than the angle threshold, it is determined that the construction machinery is not in a level state.

[0032] Optionally, the auxiliary control system further includes:

[0033] The image generating device communicates with the processing device and the display device, and is configured to generate a gravity center direction image according to the tilt angle and send the gravity center direction image to the display device.

[0034] Optionally, the auxiliary control system further includes:

[0035] an environmental sensing device, in communication with the processing device, and configured to detect environmental sensing information of the construction machinery;

[0036] The image generating device is further configured to:

[0037] An environmental perception image is generated according to the environmental perception information, and the environmental perception image is sent to a display device.

[0038] A second aspect of the present application provides an auxiliary control method for engineering machinery, which is applied to the auxiliary control device for engineering machinery. The auxiliary control method for engineering machinery includes:

[0039] Acquire and project a display image through a display device;

[0040] reflecting the display image to the windshield of the construction machinery through the reflecting device;

[0041] Acquire an image of the driver of the construction machinery by means of an image acquisition device;

[0042] determining the eye position of the driver based on the image of the driver;

[0043] The reflection angle of the reflection device is adjusted according to the position of the human eye to adjust the imaging position of the displayed image on the windshield.

[0044] Optionally, adjusting the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the displayed image on the windshield includes:

[0045] Obtain the height difference between the human eye position and the preset human eye position;

[0046] Determine the target reflection angle of the reflection device according to the height difference;

[0047] The reflection angle of the reflection device is adjusted to the target reflection angle to adjust the imaging position of the display image on the windshield so that the adjusted imaging position is located directly in front of the driver's line of sight.

[0048] Optionally, determining a target reflection angle of the reflection device according to the height difference includes:

[0049] Determine whether the height difference is greater than the height threshold;

[0050] When it is determined that the height difference value is greater than the height threshold, a target reflection angle of the reflection device is determined according to the height difference value.

[0051] Optionally, the auxiliary control method further includes:

[0052] Obtain the tilt angle of the construction machinery;

[0053] Determine whether the construction machinery is in a level position based on the tilt angle;

[0054] When it is determined that the construction machinery is not in a level position, a prompt message is issued.

[0055] Optionally, determining whether the construction machinery is in a level state according to the tilt angle includes:

[0056] Get the angle difference between the tilt angle and the preset angle;

[0057] Determine whether the angle difference is greater than an angle threshold;

[0058] When the angle difference is greater than the angle threshold, it is determined that the construction machinery is not in a level state.

[0059] A third aspect of the present application provides an engineering machinery, comprising the above-mentioned auxiliary control system for the engineering machinery.

[0060] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configures the processor to execute the above-mentioned auxiliary control method for engineering machinery.

[0061] Through the above technical scheme, that is, the display image is acquired and projected by the display device, the reflection device reflects the display image to the windshield of the engineering machinery, and the reflection angle of the reflection device is adjustable, the image acquisition device acquires the image of the driver of the engineering machinery, the processing device communicates with the image acquisition device, the driver's eye position is determined according to the driver's image, and the reflection angle of the reflection device is adjusted according to the eye position to adjust the imaging position of the display image on the windshield. In this way, the display image can be projected onto the windshield, and the imaging position of the display image on the windshield can be automatically adjusted according to the change of the driver's line of sight, thereby providing a good assistance to the driver when the engineering machinery is leveling, reducing the difficulty of leveling the engineering machinery and improving the leveling efficiency of the engineering machinery.

[0062] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0064] Figure 1 is a schematic diagram of the structure of an auxiliary control system for engineering machinery provided in an embodiment of the present application;

[0065] Figure 2 is a schematic diagram of the position relationship of an auxiliary control system for engineering machinery provided in an embodiment of the present application;

[0066] Figure 3 is a schematic diagram of an engineering machine provided in an embodiment of the present application displaying an image on a windshield in a sitting level state;

[0067] Figure 4 is a schematic diagram of an image displayed on a windshield of an engineering machine provided in an embodiment of the present application when the engineering machine is not in a level position;

[0068] Figure 5 is a flow chart of an auxiliary control method for engineering machinery provided in an embodiment of the present application;

[0069] Figure 6 is a flow chart of step S15 in the auxiliary control method for engineering machinery provided in an embodiment of the present application;

[0070] Figure 7 is a flow chart of step S152 in the auxiliary control method for engineering machinery provided in an embodiment of the present application;

[0071] Figure 8 is another flow chart of the auxiliary control method for engineering machinery provided in the embodiment of the present application;

[0072] Fig. 9 is a flow chart of step S17 in the auxiliary control method for engineering machinery provided in an embodiment of the present application;

[0073] Fig.10 It is an internal structure diagram of the computer device provided in the embodiment of the present application.

[0074] Description of Reference Numerals

[0075] 1. Display device; 2. Windshield;

[0076] 3. Image acquisition device; 4. Processing device;

[0077] 5. Plane reflector; 6. Curved reflector;

[0078] 7. Cab bracket; 8. Inclination detection device;

[0079] 9. Image generating device; 10. Environment sensing device. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0081] Figure 1 Schematic diagram of the structure of the auxiliary control system for engineering machinery provided in the embodiment of the present application. Figure 1 As shown, in one embodiment of the present application, an auxiliary control system for engineering machinery is provided, including: a display device 1, configured to acquire and project a display image; a reflection device, the reflection angle of the reflection device is adjustable, and is configured to reflect the display image to the windshield 2 of the engineering machinery; an image acquisition device 3, configured to acquire an image of the driver of the engineering machinery; a processing device 4, communicating with the image acquisition device 3, and configured to: determine the eye position of the driver based on the image of the driver; adjust the reflection angle of the reflection device according to the eye position to adjust the imaging position of the display image on the windshield 2.

[0082] Specifically, the display device 1 may include a display, and may also include a fluorescent lamp and a projector, etc. The display image may include a driving image of the engineering machinery during operation, and may also include a text image converted from instrument information, etc. The display device 1 sends the display image to the reflection device in a projected manner. The reflection device has a reflection function, and can reflect the received display image to the windshield 2 of the engineering machinery (generally the front windshield of the cab), so that the driver can see the display image, thereby providing a good assistance to the driver when the engineering machinery is leveling. The image acquisition device 3 can use a camera to obtain the image of the driver of the engineering machinery in real time and send it to the processing device 4. Among them, the reflection angle of the reflection device is adjustable, and the processing device 4 can determine the driver's eye position according to the driver's image, and then adjust the reflection angle of the reflection device according to the eye position to adjust the imaging position of the display image on the windshield 2. It can be understood that due to the difference in height of the drivers and the different sitting postures of the drivers in the driving position, different drivers will have different eye positions. The reflection angle of the reflection device is adjusted according to the position of the driver's eyes to automatically adjust the imaging position of the display image on the windshield, so that the driver can see the entire display image on the windshield 2. In this way, the display image can be projected onto the windshield, and the imaging position of the display image on the windshield can be automatically adjusted according to the change of the driver's line of sight, thereby providing a good assistance to the driver when the engineering machinery is leveling the ground, reducing the difficulty of the engineering machinery leveling the ground and improving the leveling efficiency of the engineering machinery.

[0083] In one embodiment, the processing device 4 adjusts the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the displayed image on the windshield, including: obtaining the height difference between the human eye position and the preset human eye position; determining the target reflection angle of the reflection device according to the height difference; adjusting the reflection angle of the reflection device to the target reflection angle to adjust the imaging position of the displayed image on the windshield 2, so that the adjusted imaging position is located directly in front of the driver's line of sight.

[0084] Specifically, the processing device 4 can pre-calibrate a human eye position and set it as a preset human eye position, and then collect the image of the driver of the engineering machinery through the image acquisition device 3, and determine the driver's human eye position, and then calculate the height difference between the human eye position and the preset human eye position, and then determine the target reflection angle of the reflection device according to the height difference, that is, the reflection angle to which the reflection device needs to be adjusted. After adjusting the reflection angle of the reflection device to the target reflection angle, the imaging position of the displayed image on the windshield 2 can be adjusted so that the adjusted imaging position is located directly in front of the driver's line of sight, that is, to ensure that the center position of the eye box (the range that allows the driver's eyes to see the entire displayed image) is on the same horizontal line as the driver's human eye position.

[0085] Further, in one embodiment, the processing device 4 determines the target reflection angle of the reflection device according to the height difference value, including: determining whether the height difference value is greater than the height threshold; when it is determined that the height difference value is greater than the height threshold, determining the target reflection angle of the reflection device according to the height difference value.

[0086] Specifically, since the image acquisition device 3 itself has a certain measurement tolerance, the processing device 4 can set a height threshold according to the measurement tolerance of the image acquisition device 3. After the processing device 4 calculates the height difference between the human eye position and the preset human eye position, it first determines whether the height difference is greater than the height threshold. When the height difference is greater than the height threshold, the target reflection angle of the reflection device is further determined according to the height difference, and then the reflection angle of the reflection device is adjusted to the target reflection angle to ensure that the center position of the eye box and the human eye position of the driver are on the same horizontal line. It can be understood that when the height difference is less than or equal to the height threshold, there is no need to adjust the reflection angle of the reflection device.

[0087] In one embodiment, the reflecting device includes a plane reflector 5, a curved reflector 6 and an adjusting mechanism; the plane reflector 5 is configured to reflect the displayed image to the curved reflector 6; the curved reflector 6 is configured to magnify the displayed image and reflect it to the windshield 2; the adjusting mechanism communicates with the processing device 4 and is configured to adjust the reflection angle of the plane reflector 5 and / or the curved reflector 6 according to the control instruction of the processing device 4.

[0088] Specifically, the plane reflector 5 can make the displayed image enter the curved reflector 6 after one reflection, thereby increasing the optical path of the light. The curved reflector 6 can match the shape of the windshield to eliminate the distortion of the displayed image, and amplify the displayed image to increase the observation area where the light presents the image. In order to ensure that the center position of the eye box is on the same horizontal line as the driver's eye position, the adjustment mechanism can only adjust the reflection angle of the plane reflector 5, or only adjust the reflection angle of the curved reflector 6, and can also adjust the reflection angle of the plane reflector 5 and the reflection angle of the curved reflector 6 at the same time. A preferred embodiment is that the adjustment mechanism only adjusts the reflection angle of the curved reflector 6, and the adjustment mechanism and the curved reflector 6 are integrated into an electrically controlled curved reflector, and the display device 1, the plane reflector 5 and the curved reflector 6 (an electrically controlled curved reflector integrated with an adjustment mechanism) are all fixed above the cab bracket 7.

[0089] It should be noted that the target reflection angle of the reflection device can be determined according to the positional relationship among the reflection device, the windshield 2, and the image acquisition device 3, as well as the preset human eye position, height difference, etc.

[0090] For an example, see Figure 2 , Figure 2 Schematic diagram of the position relationship of the auxiliary control system for engineering machinery provided in the embodiment of the present application. Figure 2 As shown, h 0 represents the preset eye position, h represents the actual eye position of the driver, and hx 0 represents the height of the image acquisition device 3 from the preset human eye position, hx represents the height of the image acquisition device 3 from the actual human eye position, and Ax 0 represents the distance between the image acquisition device 3 and the intersection of the preset human eye position and the windshield 2, Ax represents the distance between the image acquisition device 3 and the intersection of the actual human eye position and the windshield 2, L represents the distance between the image acquisition device 3 and the curved reflector 6, a 0 represents the reflection angle of the curved reflector 6 before adjustment, a represents the reflection angle of the curved reflector 6 after adjustment (target reflection angle), c 0 represents the angle between the reflected light of the curved reflector 6 before adjustment and the horizontal direction, c represents the angle between the reflected light of the curved reflector 6 after adjustment and the horizontal direction, and θ represents the angle between the windshield 2 and the horizontal direction. It can be seen from the geometric relationship in the figure that:

[0091]

[0092]

[0093]

[0094] Wherein, Δc represents the change in the angle between the reflected light of the curved reflector 6 and the horizontal direction before and after adjustment, Δc=c 0 -c, based on the principle of light reflection:

[0095] Δc=2Δα=2(α 0 -α) (4)

[0096] Substituting equation (2) into equation (3) yields:

[0097]

[0098] Substituting equation (1) into equation (5) yields:

[0099]

[0100] From the geometric relationship, we can get:

[0101] h 0 -h=hx 0 -hx (7)

[0102] Substituting equation (4) and equation (7) into equation (6), we can obtain:

[0103]

[0104] Among them, a 0 、h 0 、hx 0 are all known, L and θ are constants, and after determining the height hx of the image acquisition device 3 from the actual human eye position, the processing device 4 can determine the reflection angle a of the adjusted curved reflector 6, that is, the target reflection angle.

[0105] In one embodiment, the auxiliary control system also includes: an inclination detection device 8, which communicates with the processing device 4 and is configured to detect the inclination angle of the engineering machinery; the processing device 4 is also configured to: determine whether the engineering machinery is in a level state based on the inclination angle; and issue a prompt message when it is determined that the engineering machinery is not in a level state.

[0106] Specifically, the inclination detection device 8 can use a gravity sensor, and by measuring the acceleration caused by gravity, the inclination angle of the construction machinery relative to the horizontal plane can be determined, and sent to the processing device 4. The processing device 4 can determine whether the construction machinery is in a sitting level state according to the inclination angle of the construction machinery, and send a prompt message when the construction machinery is not in a sitting level state, to remind the driver to adjust the construction machinery to be in a sitting level state, to prevent the construction machinery from performing leveling operations when it is not in a sitting level state, and to ensure the leveling quality of the construction machinery.

[0107] Further, in one embodiment, the processing device 4 is configured to determine whether the construction machinery is in a level state based on the tilt angle, including: obtaining the angle difference between the tilt angle and the preset angle; determining whether the angle difference is greater than an angle threshold; and when the angle difference is greater than the angle threshold, determining that the construction machinery is not in a level state.

[0108] Specifically, the preset angle can be 90°, and the angle threshold can be set according to the requirements of level ground. After calculating the angle difference between the inclination angle and the preset angle, the processing device 4 determines whether the angle difference is greater than the angle threshold. When the angle difference is greater than the angle threshold, it is determined that the construction machinery is not in a level state, and a prompt message is issued to remind the driver to adjust the construction machinery to make it in a level state; when the angle difference is less than or equal to the angle threshold, it is determined that the construction machinery is in a level state, and the driver can start leveling operations.

[0109] Furthermore, in one embodiment, the auxiliary control system also includes: an image generating device 9, which communicates with the processing device 4 and the display device 1, and is configured to generate a center of gravity direction image according to the tilt angle, and send the center of gravity direction image to the display device 1.

[0110] Specifically, the image generating device 9 can adopt GPU (English full name: Graphics Processing Unit, Chinese full name: Graphics Processing Unit). The image generating device 9 obtains the inclination angle data of the construction machinery from the processing device 4, and generates a center of gravity direction image according to the inclination angle data, and then sends the center of gravity direction image to the display device 1, so that the display device 1 displays the center of gravity direction image, and then reflects the center of gravity direction image onto the windshield 2 through the reflection device, so that the driver has a more intuitive feeling of the current inclination state of the construction machinery, thereby better adjusting the construction machinery to a level state.

[0111] Furthermore, in one embodiment, the auxiliary control system also includes: an environmental perception device 10, which communicates with the processing device 4 and is configured to detect environmental perception information of the construction machinery; the image generation device 9 is also configured to: generate an environmental perception image based on the environmental perception information, and send the environmental perception image to the display device 1.

[0112] Specifically, the environment perception device 10 can use detection equipment such as cameras, laser radars, infrared sensors, etc. to detect the environment perception information of the construction machinery in real time when the construction machinery is leveling, and send it to the processing device 4. The image generation device 9 obtains the environment perception information of the construction machinery from the processing device 4, generates an environment perception image according to the environment perception information, and then sends the environment perception image to the display device 1, so that the display device 1 displays the environment perception image, and then reflects the environment perception image to the windshield 2 through the reflection device, so as to enhance the driver's perception of the actual working environment of the construction machinery, provide a good assistance for the driver, and further reduce the difficulty of the construction machinery leveling operation and the entry threshold of the driver.

[0113] For practical application, please refer to Figure 3 and Figure 4 , Figure 3 is a schematic diagram of an engineering machine provided in an embodiment of the present application displaying an image on a windshield in a sitting level state, Figure 4 Schematic diagram of an engineering machine provided in an embodiment of the present application showing an image on a windshield when the machine is not in a level position. Figure 3 and Figure 4 As shown in the figure, the horizontal dotted line and vertical dotted line in the upper left corner represent the horizontal direction and vertical direction of the ground respectively, and the arrow represents the direction of the center of gravity of the construction machinery. When the construction machinery is in a level state, the arrow will be perpendicular to the horizontal dotted line and coincide with the vertical dotted line, that is, Figure 3 As shown; when the construction machinery is not in a level state, the arrow will have a certain angle with the horizontal dotted line and the vertical dotted line, that is, Figure 4As shown, at this time, the driver can adjust the construction machinery to a level state according to the angle between the arrow and the horizontal dotted line or the vertical dotted line, which is convenient and intuitive, thereby shortening the driver's preparation time for leveling operations and improving the efficiency of leveling operations.

[0114] Please continue reading Figure 3 and Figure 4 , the thick solid line represents the straight line where the initial position and the end position of the working device of the engineering machinery are located. When leveling the ground, the driver only needs to move the working device to the same position of each straight line in each cycle, such as the outermost point of each straight line, to ensure that the starting point of each cycle is at the same distance. There is no need to supplement the work due to the different lengths of leveling each cycle, and a cycle of engineering machinery leveling operation can be completed along the direction of each straight line. In addition, the horizontal and vertical grids can be displayed to further assist the driver in moving the working device to the same position of each straight line. In this way, not only can the driver's repeated supplementary operations be reduced, the difficulty of leveling the engineering machinery can be reduced, the efficiency of leveling the engineering machinery can be improved, but also the leveling quality of the engineering machinery can be improved.

[0115] Through the above technical scheme, that is, the display image is acquired and projected by the display device, the reflection device reflects the display image to the windshield of the engineering machinery, and the reflection angle of the reflection device is adjustable, the image acquisition device acquires the image of the driver of the engineering machinery, the processing device communicates with the image acquisition device, the driver's eye position is determined according to the driver's image, and the reflection angle of the reflection device is adjusted according to the eye position to adjust the imaging position of the display image on the windshield. In this way, the display image can be projected onto the windshield, and the imaging position of the display image on the windshield can be automatically adjusted according to the change of the driver's line of sight, thereby providing a good assistance to the driver when the engineering machinery is leveling, reducing the difficulty of leveling the engineering machinery and improving the leveling efficiency of the engineering machinery.

[0116] See also Figure 5 , Figure 5 1 is a flow chart of the auxiliary control method for engineering machinery provided in the embodiment of the present application. Figure 5 As shown, the embodiment of the present application further provides an auxiliary control method for engineering machinery, which is applied to the above-mentioned auxiliary control device for engineering machinery. The auxiliary control method for engineering machinery includes:

[0117] Step S11: acquiring and projecting a display image through a display device;

[0118] Step S12: reflecting the display image to the windshield of the engineering machinery through a reflection device;

[0119] Step S13: acquiring an image of the driver of the construction machinery through an image acquisition device;

[0120] Step S14: determining the eye position of the driver according to the image of the driver;

[0121] Step S15: adjusting the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the displayed image on the windshield.

[0122] For further information, see Figure 6 , Figure 6 The flowchart of step S15 in the auxiliary control method for engineering machinery provided in the embodiment of the present application is shown in FIG. In step S15, adjusting the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the displayed image on the windshield may include the following steps:

[0123] Step S151: obtaining a height difference between the human eye position and a preset human eye position;

[0124] Step S152: determining the target reflection angle of the reflection device according to the height difference;

[0125] Step S153: adjusting the reflection angle of the reflection device to the target reflection angle to adjust the imaging position of the display image on the windshield so that the adjusted imaging position is located directly in front of the driver's line of sight.

[0126] For further information, see Figure 7 , Figure 7 1 is a flow chart of step S152 in the auxiliary control method for engineering machinery provided in an embodiment of the present application. Determining the target reflection angle of the reflection device according to the height difference in step S152 may include the following steps:

[0127] Step S1521: Determine whether the height difference is greater than the height threshold;

[0128] Step S1522: When it is determined that the height difference is greater than the height threshold, determine the target reflection angle of the reflection device according to the height difference.

[0129] See also Figure 8 , Figure 8 FIG. 1 is another flow chart of the auxiliary control method for engineering machinery provided in the embodiment of the present application. Figure 8 As shown, based on steps S11 to S15, the auxiliary control method for engineering machinery further includes:

[0130] Step S16: obtaining the inclination angle of the construction machinery;

[0131] Step S17: determining whether the construction machine is in a level state according to the tilt angle;

[0132] Step S18: When it is determined that the construction machinery is not in a level state, a prompt message is issued.

[0133] For further information, see Fig. 9 , Fig. 9 The flowchart of step S17 in the auxiliary control method for engineering machinery provided in the embodiment of the present application is shown in FIG. Determining whether the engineering machinery is in a level state according to the tilt angle in step S17 may include the following steps:

[0134] Step S171: obtaining the angle difference between the tilt angle and the preset angle;

[0135] Step S172: When the angle difference is greater than the angle threshold, it is determined that the construction machine is not in a level state.

[0136] For more details about the auxiliary control method for engineering machinery in the embodiment of the present application, please refer to the above description of the auxiliary control system for engineering machinery, and the same or corresponding technical effects as the above-mentioned auxiliary control system for engineering machinery can be achieved, so they will not be repeated here.

[0137] In summary, through the above method, that is, acquiring and projecting the display image through the display device, reflecting the display image to the windshield of the engineering machinery through the reflection device, acquiring the image of the driver of the engineering machinery through the image acquisition device, determining the driver's eye position according to the driver's image, adjusting the reflection angle of the reflection device according to the eye position to adjust the imaging position of the display image on the windshield, through this type of method, the display image can be projected onto the windshield, and the imaging position of the display image on the windshield can be automatically adjusted according to the change of the driver's line of sight, thereby providing a good assistance to the driver when the engineering machinery is leveling, reducing the difficulty of leveling the engineering machinery, and improving the leveling efficiency of the engineering machinery.

[0138] An embodiment of the present application also provides a construction machinery, including the above-mentioned auxiliary control system for the construction machinery.

[0139] In one embodiment, the construction machine is an excavator.

[0140] In one embodiment, the excavator may further include:

[0141] Communication interface, capable of exchanging information with other devices (such as network equipment, terminals, etc.);

[0142] Memory is used to store computer programs that can be executed on the processor.

[0143] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the method provided by one or more technical solutions above can be implemented by adjusting kernel parameters.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0145] In actual application, the various components in the excavator can be coupled together through the bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0146] The memory in the embodiment of the present application is used to store various types of data to support the operation of the excavator. Examples of such data include: any computer program used to operate on the excavator.

[0147] The method disclosed in the above embodiment of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0148] In an exemplary embodiment, the processor may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0149] It can be understood that the memory of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0150] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the above-mentioned auxiliary control method for engineering machinery is implemented.

[0151] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0152] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, a method of any one of the above embodiments is implemented.

[0154] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the method of initializing any one of the above embodiments.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0161] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0163] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An auxiliary control system for engineering machinery, characterized in that: include: A display device configured to acquire and project a display image; a reflecting device, wherein the reflecting angle of the reflecting device is adjustable and the reflecting device is configured to reflect the displayed image onto the windshield of the engineering machinery; an image acquisition device configured to acquire an image of a driver of the engineering machine; A processing device, in communication with the image acquisition device, is configured to: determining the eye position of the driver according to the image of the driver; adjusting the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the display image on the windshield; The processing device adjusts the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the display image on the windshield, including: Obtaining a height difference between the human eye position and a preset human eye position; Determining a target reflection angle of the reflection device according to the height difference; Adjusting the reflection angle of the reflection device to the target reflection angle to adjust the imaging position of the display image on the windshield so that the adjusted imaging position is located directly in front of the driver's line of sight; an inclination detection device, in communication with the processing device and configured to detect an inclination angle of the engineering machine; The image generating device communicates with the processing device and the display device, and is configured to generate a gravity center direction image according to the tilt angle and send the gravity center direction image to the display device.

2. The auxiliary control system according to claim 1, characterized in that: The processing device determines the target reflection angle of the reflection device according to the height difference, including: Determining whether the height difference is greater than a height threshold; In a case where it is determined that the height difference is greater than the height threshold, a target reflection angle of the reflection device is determined according to the height difference.

3. The auxiliary control system according to claim 1, characterized in that: The reflecting device comprises a plane reflecting mirror, a curved reflecting mirror and an adjusting mechanism; The flat reflector is configured to reflect the display image to the curved reflector; The curved reflector is configured to magnify the display image and reflect it to the windshield; The adjustment mechanism communicates with the processing device and is configured to adjust the reflection angle of the plane reflector and / or the curved reflector according to the control instruction of the processing device.

4. The auxiliary control system according to claim 1, characterized in that: The processing device is further configured to: determining whether the engineering machine is in a sitting level state according to the tilt angle; When it is determined that the engineering machinery is not in the sitting level state, a prompt message is issued.

5. The auxiliary control system according to claim 4, characterized in that: The processing device is configured to determine whether the engineering machine is in a sitting level state according to the tilt angle, including: Obtaining an angle difference between the tilt angle and a preset angle; Determining whether the angle difference is greater than an angle threshold; When the angle difference is greater than the angle threshold, it is determined that the engineering machine is not in a level state.

6. The auxiliary control system according to claim 1, characterized in that: The auxiliary control system further includes: an environmental sensing device, in communication with the processing device and configured to detect environmental sensing information of the engineering machinery; The image generating device is further configured to: An environmental perception image is generated according to the environmental perception information, and the environmental perception image is sent to the display device.

7. An auxiliary control method for engineering machinery, characterized in that: Applied to the auxiliary control system for engineering machinery according to any one of claims 1 to 6, the auxiliary control method comprises: Acquiring and projecting a display image through the display device; Reflecting the display image onto the windshield of the engineering machinery through the reflection device; Acquire an image of the driver of the engineering machinery by means of the image acquisition device; determining the eye position of the driver according to the image of the driver; adjusting the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the display image on the windshield; The step of adjusting the reflection angle of the reflection device according to the position of the human eye to adjust the imaging position of the displayed image on the windshield includes: Obtaining a height difference between the human eye position and a preset human eye position; Determining a target reflection angle of the reflection device according to the height difference; Adjusting the reflection angle of the reflection device to the target reflection angle to adjust the imaging position of the display image on the windshield so that the adjusted imaging position is located directly in front of the driver's line of sight; Obtaining the inclination angle of the engineering machinery; The image generating device generates a gravity center direction image according to the tilt angle, and sends the gravity center direction image to the display device.

8. The auxiliary control method according to claim 7, characterized in that: The step of determining a target reflection angle of the reflection device according to the height difference comprises: Determining whether the height difference is greater than a height threshold; In a case where it is determined that the height difference is greater than the height threshold, a target reflection angle of the reflection device is determined according to the height difference.

9. The auxiliary control method according to claim 7, characterized in that: The auxiliary control method further includes: determining whether the engineering machine is in a sitting level state according to the tilt angle; When it is determined that the engineering machinery is not in the sitting level state, a prompt message is issued.

10. The auxiliary control method according to claim 9, characterized in that: The step of determining whether the engineering machine is in a sitting level state according to the tilt angle includes: Obtaining an angle difference between the tilt angle and a preset angle; Determining whether the angle difference is greater than an angle threshold; When the angle difference is greater than the angle threshold, it is determined that the engineering machine is not in a level state.

11. An engineering machine, characterized in that: The invention comprises an auxiliary control system for a construction machine according to any one of claims 1 to 6.

12. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to execute the auxiliary control method for a construction machine according to any one of claims 7 to 10.

Images