Remote auxiliary control method for engineering machinery and engineering machinery system

Abstract

The embodiment of the invention relates to the technical field of engineering machinery, and discloses a remote auxiliary control method for engineering machinery, which comprises the following steps: receiving task planning information configured by a user through a remote cabin, and determining skill types and task parameters of each stage according to the task planning information; on-site environment data are collected through a sensor assembly, and environment vector information is formed according to the on-site environment data; according to the skill type and the task parameter of each stage and the environment vector information, generating a track sequence of each joint of the engineering machinery working device according to a set period, and sending the track sequence of each joint of the engineering machinery working device to an engineering machinery end; and according to the trajectory control vector and the obtained state monitoring information of each joint of the engineering machinery working device, trajectory control is performed on the working state of the engineering machinery. According to the remote auxiliary control method for the engineering machinery, the controllability of the whole system is guaranteed, and the operation efficiency in a complex task scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of construction machinery, and particularly relates to a remote auxiliary control method for construction machinery and a construction machinery system. Background Art

[0002] At present, during the operation of traditional excavators, it mainly relies on the experience and visual judgment of operators, and there are problems such as inaccurate positioning, low operation efficiency, and high safety hazards. With the progress of technology, intelligence and automation have become the development trend of construction machinery.

[0003] Moreover, in specific scenarios such as mines and ports, the operation areas are often remote and have a high degree of danger. Considering the safety of personnel, remote excavators have become a common operation option. Operators can operate the excavator at a control center far from the site. In this way, there is no need for personnel to be present at the operation site, fundamentally eliminating the problem of personnel facing safety hazards. However, current remote excavators have a low operation efficiency during actual operation, and their operation level is difficult to reach the efficiency standard when operated by local drivers on-site. This has greatly affected aspects such as the overall project progress and economic benefits. Therefore, designing a solution that can improve the intelligence of excavators has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0004] In view of the above-mentioned defects, an embodiment of the present invention discloses a remote auxiliary control method for construction machinery, which can improve the safety and efficiency in complex task scenarios and has better system adaptability.

[0005] A first aspect of an embodiment of the present invention discloses a remote auxiliary control method for construction machinery, including:

[0006] Receiving task planning information configured by a user through a remote cabin, and determining the skill types and task parameters of each stage according to the task planning information;

[0007] Collecting on-site environmental data through a sensor assembly disposed on the construction machinery, and forming environmental vector information according to the on-site environmental data;

[0008] Generating a trajectory sequence of each joint of the working device of the construction machinery according to the skill types and task parameters of each stage and the environmental vector information at a set period, and sending the trajectory sequence of each joint of the working device of the construction machinery to the construction machinery end;

[0009] Converting the trajectory sequence of each joint of the working device of the construction machinery into a trajectory control vector, and performing trajectory control on the working state of the construction machinery according to the trajectory control vector and the obtained status monitoring information of each joint of the working device of the construction machinery.

[0010] As an alternative embodiment, in the first aspect of the embodiments of the present invention, before receiving the task planning information configured by the user through the remote cabin, it further includes:

[0011] Receiving the operation mode information determined by the user through the remote cabin. If the operation mode information is the manual operation mode, receiving the operation instruction information sent by the user through the remote cabin in real time, and controlling the working state of the construction machinery working device according to the operation instruction information;

[0012] If the operation mode information is the automatic operation mode, execute receiving the task planning information configured by the user through the remote cabin.

[0013] As an alternative embodiment, in the first aspect of the embodiments of the present invention, the auxiliary control method further includes:

[0014] During the operation process, receiving the mode conversion instruction of the user in real time. If the mode conversion instruction is to switch to the manual mode, receiving the operation instruction information sent by the user through the remote cabin in real time to intervene in the action state of the construction machinery through manual control.

[0015] As an alternative embodiment, in the first aspect of the embodiments of the present invention, after collecting the on-site environmental data through the sensor assembly provided on the construction machinery, it further includes:

[0016] If the on-site environmental data does not meet the automatic operation conditions, sending a prompt message to the remote cabin to remind the user to perform manual operation.

[0017] As an alternative embodiment, in the first aspect of the embodiments of the present invention, the skill types include the excavation task type, the loading task type, and the leveling task type; the task parameters include the excavation depth value, the loading quantity, and the land flatness;

[0018] The forming of the environmental vector information according to the on-site environmental data includes:

[0019] Identifying the on-site environmental data to determine the position information, shape information, and volume information of the corresponding target objects and obstacles, and generating the corresponding environmental vector information according to the position information, shape information, and volume information of the target objects and obstacles;

[0020] If the number of the obstacles is multiple, determining the optimal excavation sequence according to the volume position quantity of the intermediate obstacles.

[0021] The second aspect of the embodiments of the present invention discloses a construction machinery control system, including a construction machinery end, the main controller, communication module, positioning module, camera module, lidar, and inclination sensor of the construction machinery end;

[0022] The number of the inclination sensors is multiple, and the multiple inclination sensors are respectively arranged at the body, the boom, the arm and the bucket of the engineering construction machinery, and are used for monitoring the inclination angles of each part in real time and transmitting the monitored inclination angles to the main controller;

[0023] The camera module is used for collecting the surrounding environment images of the intelligent excavator in real time and transmitting the environment images to the main controller; the lidar is used for detecting the obstacles in the surrounding environment and transmitting the detected signals to the main controller to determine the shape, volume and distance from the bucket of the object;

[0024] The positioning module is used for determining the position information of the intelligent excavator and the bucket;

[0025] The communication module is used for communicating with the background server;

[0026] The main controller is used for receiving the sensor data and performing analysis and processing to execute corresponding operation commands.

[0027] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the main controller is built with a PID algorithm, and the actions of each part of the excavator are accurately controlled through the PID algorithm.

[0028] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the control system further includes a background server and a remote control terminal; the background server is communicatively connected to the engineering machinery terminal, and the remote control terminal is communicatively connected to the background server;

[0029] The background server is used for receiving various operation data transmitted by the engineering machinery terminal in real time, and the operation data includes the state of the excavator, the position of the bucket and the operation progress; the background server performs two-way data interaction with the main controller of the engineering machinery terminal, and is used for sending operation instructions, receiving the data fed back by the excavator and monitoring and scheduling the operation;

[0030] The remote server includes a remote industrial computer, and the remote industrial computer includes a display screen, control buttons and an operating handle. The display screen is used for providing an intuitive operation interface for the operator. The operator sends operation instructions through the remote control cabin industrial computer and transmits them to the main controller of the engineering machinery terminal through the network to realize the remote control of the excavator.

[0031] The third aspect of the embodiments of the present invention discloses an electronic device, including: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory for executing the remote auxiliary control method of the engineering machinery disclosed in the first aspect of the embodiments of the present invention.

[0032] A fourth aspect of the embodiments of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the remote auxiliary control method for construction machinery disclosed in the first aspect of the embodiments of the present invention.

[0033] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0034] In the remote auxiliary control method for construction machinery in the embodiments of the present invention, through the integration of manual operation and intelligent automatic planning, on the one hand, the subjective judgment ability of the operator is fully exerted, and on the other hand, the operation efficiency is improved by means of the accuracy of the algorithm, and the controllability of the entire system is ensured, and the operation efficiency in complex task scenarios is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 is a schematic flow chart of the remote auxiliary control method for construction machinery disclosed in the embodiments of the present invention;

[0037] Figure 2 is a schematic hardware structure diagram of the remote auxiliary control device for construction machinery disclosed in the embodiments of the present invention;

[0038] Figure 3 is a schematic specific flow chart of the remote auxiliary control for construction machinery disclosed in the embodiments of the present invention;

[0039] Figure 4 is another schematic specific flow chart of the remote auxiliary control for construction machinery disclosed in the embodiments of the present invention;

[0040] Figure 5 is still another schematic specific flow chart of the remote auxiliary control for construction machinery disclosed in the embodiments of the present invention;

[0041] Figure 6 is a schematic structure diagram of the construction machinery system disclosed in the embodiments of the present invention;

[0042] Figure 7 is a schematic structure diagram of the construction engineering equipment disclosed in the embodiments of the present invention;

[0043] Figure 8 is a schematic structure diagram of an electronic device disclosed in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] It should be noted that the terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusion. Exemplarily, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0046] In specific scenarios such as mines and ports, the operation areas are often remote and have a high degree of danger. Considering the safety of personnel, remote excavators have become a common operation option. The operator can operate the excavator at a control center far from the site. In this way, there is no need for personnel to be present at the operation site, fundamentally eliminating the problem of personnel facing safety hazards. However, current remote excavators have a low operation efficiency during actual operation, and their operation level is difficult to reach the efficiency standard when operated by a local driver on-site. Based on this, the embodiments of the present invention disclose a remote auxiliary control method, system, electronic device and storage medium for construction machinery, which combines manual operation with intelligent automatic planning. On the one hand, it gives full play to the subjective judgment ability of the operator, and on the other hand, it improves the operation efficiency by means of the accuracy of the algorithm, and ensures the controllability of the entire system, improving the operation efficiency in complex task scenarios.

[0047] Embodiment 1

[0048] Please refer to Figure 1 , Figure 1It is a schematic flowchart of a remote assistance control method for construction machinery disclosed in an embodiment of the present invention. Among them, the execution subject of the method described in the embodiment of the present invention is an execution subject composed of software or / and hardware. This execution subject can receive relevant information through wired or / and wireless means and can send certain instructions. Of course, it can also have certain processing functions and storage functions. This execution subject can control multiple devices, such as remote physical servers or cloud servers and related software, or it can also be a local host or server and related software that performs relevant operations on devices placed somewhere. In some scenarios, it can also control multiple storage devices, and the storage devices can be placed in the same place or different places as the devices. As Figures 1 to 5 shown, the remote assistance control method based on construction machinery includes the following steps:

[0049] S101: Receive the task planning information configured by the user through the remote cabin, and determine the skill types and task parameters for each stage according to the task planning information;

[0050] S102: Collect on-site environmental data through the sensor assembly provided on the construction machinery, and form environmental vector information according to the on-site environmental data;

[0051] S103: Generate the trajectory sequences of each joint of the working device of the construction machinery according to the skill types and task parameters for each stage and the environmental vector information at a set period, and send the trajectory sequences of each joint of the working device of the construction machinery to the construction machinery side;

[0052] S104: Convert the trajectory sequences of each joint of the working device of the construction machinery into trajectory control vectors, and perform trajectory control on the working state of the construction machinery according to the trajectory control vectors and the obtained status monitoring information of each joint of the working device of the construction machinery.

[0053] Through the solution of the embodiment of the present invention, the user can perform task planning in the remote cabin, away from the operation site, avoiding direct exposure of the operator to a dangerous environment. By collecting on-site environmental data through the sensor assembly and forming environmental vector information, the construction machinery can perceive the changes in the operation environment in real time. In this way, in different terrain, geological conditions, and complex construction site environments, it can adjust the trajectories of each joint of the working device according to the actual situation, better adapt to various operation environments, and improve the accuracy and efficiency of the operation.

[0054] The solution of the embodiment of the present invention generates the trajectory sequences of each joint of the working device of the construction machinery according to the task planning information and the environmental vector information, and performs precise trajectory control, which helps to achieve more precise operations. For example, in some scenarios with high requirements for construction accuracy, such as excavation, lifting and other operations, the operation accuracy and quality can be improved, and the errors can be reduced. The entire control process integrates multiple types of information to generate the trajectory sequences and perform trajectory control, enabling the construction machinery to develop in the direction of intelligence and automation. The intervention of manual operation is reduced, the labor intensity of the operator is lowered, the operation stability and reliability are improved, which helps to achieve unmanned or less-manned operations and enhance the overall efficiency of the engineering operations.

[0055] The solution of the embodiment of the present invention mainly improves the problems existing in the existing remote excavator technology, aiming to improve the operation efficiency of the remote excavator. In the existing technology, although the remote excavator has solved the problem of ensuring the safety of personnel in remote and dangerous operation areas such as mines and ports, that is, by allowing the operator to operate in a control center far from the site, so that there is no need for personnel on the operation site, eliminating the potential safety hazards of personnel. However, its operation efficiency is low and it is difficult to reach the efficiency level of local drivers' on-site operation, which seriously affects the overall project progress and economic benefits.

[0056] The features that distinguish the present invention from other existing technologies: In terms of the operation mode. An innovative operation mode that combines manual remote control and machine autonomous operation is realized. The operator can flexibly select the manual remote control mode according to the actual operation situation and directly control the excavator to complete the corresponding tasks; or the operator can also select the machine autonomous operation mode, allowing the system to perform autonomous operations according to preset automatic tasks, such as automatic loading, automatic excavation, etc., relying on algorithms. This flexible switching and organic combination operation mode can not only give play to the advantages of the operator in scenarios that require precise manual operation, but also improve the efficiency in some regular and repetitive operation links by means of machine autonomous operation. Most of the existing other technologies focus on a single operation mode and cannot flexibly balance different operation requirements, with poor adaptability.

[0057] In terms of task planning and execution mechanism, a unique mode is adopted where task planning is manually completed by remote operators, and machine trajectory planning and motion control are completed by algorithm control. Based on their own experience and understanding of the overall operation requirements, remote operators perform manual task planning at the cabin end to determine the skill types, task parameters, etc., providing a general direction for the operation. On this basis, the system uses advanced algorithms to be responsible for machine trajectory planning and motion control. For example, it automatically plans the motion trajectory according to the task type and target, generates joint space trajectory sequence information, and precisely controls the motion trajectory of the excavator to achieve efficient operation. This combination gives full play to the subjective judgment ability of the operator on the one hand, and improves the operation efficiency with the precision of the algorithm on the other hand, while ensuring the controllability of the entire system. Different from the past, the existing technologies either rely too much on manual operation and are inefficient in complex task scenarios, or completely focus on machine autonomous operation and lack flexible manual intervention, resulting in poor controllability.

[0058] More preferably, before receiving the task planning information configured by the user through the remote cabin, it further includes:

[0059] Receiving the operation mode information determined by the user through the remote cabin. If the operation mode information is the manual operation mode, the operation instruction information sent by the user through the remote cabin is received in real time, and the working state of the construction machinery working device is controlled according to the operation instruction information;

[0060] If the operation mode information is the automatic operation mode, execute receiving the task planning information configured by the user through the remote cabin.

[0061] In the actual implementation process, users can freely choose the manual operation mode or the automatic operation mode according to actual needs and operation scenarios. In some complex and special working conditions, such as when fine operation is required, emergencies need to be handled, or equipment debugging is carried out, the manual operation mode can be selected. Experienced operators can control the working state of the construction machinery working device in real time, giving full play to human subjective initiative and judgment to ensure the accuracy and safety of the operation. In some conventional and highly repetitive operation scenarios, such as large-area excavation and site leveling, the automatic operation mode can be selected. By receiving the task planning information configured by the user, the construction machinery can automatically complete the operation according to the preset procedures and parameters, improving the operation efficiency and reducing the labor cost. For the automatic operation mode, by pre-configuring the task planning information, the construction machinery can operate according to the optimized trajectory and parameters, avoiding the mistakes and non-standard actions that may occur in manual operation, thus improving the operation efficiency and quality. For example, in road construction, an automatically operated paver can accurately pave according to parameters such as the set speed and thickness, reducing the time and error of manual adjustment. At the same time, in the manual operation mode, by receiving the operation instruction information in real time and controlling the working device, the operator can also adjust the operation method in a timely manner according to the actual situation on site, improving the operation efficiency in case of flexible response. The manual mode and the automatic mode in the embodiments of the present invention are not independent, but are interrelated. By manually performing task planning to provide more comprehensive data to be processed to the control system, the system can further analyze and process the data combined with the manually input data to improve the effect of automated processing.

[0062] Through the solution of the embodiments of the present invention, the labor intensity of the operator can be greatly reduced. The automatic operation mode undertakes most of the conventional operation tasks. The operator does not need to be in a highly concentrated state for a long time, nor does he need to perform repetitive physical labor, thus greatly reducing the labor intensity. Even in the manual operation mode, since the operation is carried out through the remote cabin and the operator is away from the operation site, avoiding the harsh working environment and dangerous operation conditions, the labor intensity can also be reduced to a certain extent.

[0063] The two operation modes in the embodiments of the present invention are independent of each other and complementary to each other. When one mode fails or malfunctions, it can be quickly switched to the other mode to continue the operation, improving the reliability and stability of the entire system. For example, if there is an error in the task planning information in the automatic operation mode or the equipment sensor fails, it can be quickly switched to the manual operation mode, and the operator can manually control the equipment to avoid operation interruption or safety accidents. The embodiments of the present invention also organically unify the two through task planning, greatly improving the overall operation efficiency.

[0064] More preferably, the auxiliary control method further includes:

[0065] During the operation process, the system receives the mode conversion instruction from the user in real time. If the mode conversion instruction is to switch to the manual mode, it will receive the operation instruction information sent by the user through the remote cabin in real time to intervene in the action state of the construction machinery through manual control.

[0066] During the operation of construction machinery, various emergencies may occur, such as sensor failures, sudden changes in the on-site environment leading to abnormalities in automatic operations, etc. At this time, the user can switch to the manual mode by sending a mode conversion instruction to intervene in the action state of the construction machinery in a timely manner, avoiding accidents or reducing losses. For example, during automatic excavation, if an unknown underground obstacle is encountered, the manual mode allows the operator to flexibly adjust the excavation action according to the actual situation to avoid the obstacle.

[0067] Some special operation tasks or working conditions may require more precise and flexible operations, which are difficult to fully meet by the automatic operation mode. By switching to the manual mode, experienced operators can perform more precise control with their professional skills and judgment to achieve better operation effects. For example, when performing some delicate building demolition work, manual operation can better protect the surrounding environment. When some links of the automatic operation mode have errors or failures, the manual mode, as a backup control method, can ensure that the construction machinery continues to operate and does not completely stop working, thus improving the fault tolerance and reliability of the entire system. Even after the automatic operation mode returns to normal, it is possible to choose whether to switch back to the automatic mode according to the actual situation, increasing the flexibility of the operation.

[0068] More preferably, after collecting the on-site environment data through the sensor assembly provided on the construction machinery, it further includes:

[0069] If the on-site environment data does not meet the automatic operation conditions, a prompt message is sent to the remote cabin to remind the user to perform manual operation.

[0070] When the on-site environment data does not meet the automatic operation conditions, such as in the case of complex terrain, bad weather or sensor failures, etc., automatic operation may not guarantee the accuracy and safety of the operation. At this time, sending a prompt message to let the user switch to the manual operation mode in a timely manner can utilize the experience and judgment of the operator to flexibly handle various complex situations and ensure the smooth continuation of the operation.

[0071] Through the real-time monitoring and judgment of the on-site environment data, the system can automatically adjust the operation mode according to different environmental conditions, improving the adaptability of the construction machinery to various complex environments. Whether in a conventional operation environment or in some extreme or special situations, the most suitable operation method can be selected to give full play to the advantages of automatic operation and manual operation.

[0072] In an environment not suitable for automatic operation, if the device continues to run in the automatic mode, it may lead to problems such as the device colliding with obstacles and overloading, thus increasing the risk of device damage. By promptly switching to the manual mode, the operator can prevent the device from being damaged in a dangerous or adverse environment, extend the service life of the device, and reduce the maintenance cost.

[0073] More preferably, the skill types include an excavation task type, a loading task type, and a leveling task type; the task parameters include the excavation depth value, the loading quantity, and the land flatness.

[0074] The forming of the environmental vector information based on the on-site environmental data includes:

[0075] Identifying the on-site environmental data to determine the position information, shape information, and volume information of the corresponding target objects and obstacles, and generating the corresponding environmental vector information according to the position information, shape information, and volume information of the target objects and obstacles.

[0076] If the number of the obstacles is multiple, determine the optimal excavation sequence according to the volume position quantity of the intermediate obstacles.

[0077] In the embodiments of the present invention, the clear regulations of the skill types and task parameters enable the construction machinery to clearly know the specific operation tasks and objectives. For example, the excavation task type clearly indicates that an excavation operation is to be carried out, and the excavation depth value specifically stipulates the depth requirement for excavation, which enables the construction machinery to operate more accurately according to the requirements when performing the task, improves the pertinence and accuracy of the operation, and avoids operation deviation. Similarly, the corresponding relationships between the loading task type and the loading quantity, and between the leveling task type and the land flatness can also enable the construction machinery to have clear objectives and measurement criteria in different operation scenarios, so as to better complete the tasks.

[0078] By identifying and processing the on-site environmental data to generate environmental vector information, the construction machinery can real-time sense the situation of target objects and obstacles in the operation environment. Determining the position information, shape information, and volume information of the target objects and obstacles helps the construction machinery plan a reasonable operation path, avoid obstacles, and improve the safety and efficiency of the operation. For example, in an excavation operation, if the position and shape of underground obstacles are known in advance, the excavation equipment can be prevented from colliding with the obstacles, reducing the risk of equipment damage and improving the excavation efficiency at the same time.

[0079] When there are multiple obstacles, determining the optimal excavation sequence based on the volume, position, and quantity of the middle obstacle can further optimize the operation process of construction machinery. This method can avoid frequently adjusting the excavation direction and position during excavation, reduce unnecessary actions and time waste, and improve the operation efficiency under complex working conditions. For example, when conducting excavation operations in a site with multiple obstacles, arranging the excavation sequence reasonably can enable the excavation equipment to operate more smoothly, reducing equipment wear and energy consumption.

[0080] As Figures 2 to 4 shown, the solution of the embodiment of the present invention is specifically implemented as follows:

[0081] Manual task planning: The operator conducts task planning at the cabin end to determine the skill type and task parameters. This provides a macroscopic operation guide for the subsequent operation of the excavator, ensuring that the excavator operates according to the predetermined goals.

[0082] Perception network: Collect on-site data through sensors to form vector information. The perception network can perceive and analyze the complex operation environment, providing accurate data support for motion planning.

[0083] Motion planning: Receive the data from task planning and the perception network, and generate trajectory sequences (including position and angular velocity) of each joint of the excavator working device at a certain cycle (e.g., 10 Hz). This module can plan the optimal motion path of the excavator according to the on-site environment and task requirements.

[0084] Trajectory control: Convert the trajectory information generated by motion planning into control vectors at a certain cycle (e.g., 50 Hz), and perform precise control of the trajectory in combination with joint state observables (position, angular velocity) to ensure the accurate operation of the excavator.

[0085] Wired control chassis: With the assistance of angle sensors, execute the operations determined by trajectory control, and drive the actuators of the excavator to complete the corresponding operation tasks.

[0086] The system operation process of the embodiment of the present invention is as follows: The operator first views the operation view at the cabin end and conducts manual task planning to determine the tasks to be completed in this operation; the operator can select the automatic or manual operation mode. If the manual mode is selected, it enters the manual control process, and the operator directly controls the excavator to complete the operation task and then ends the operation. When the operator selects the automatic mode, the task parameters are set manually, and the system can execute corresponding operations according to the preset task types, such as automatic loading, automatic excavation, automatic leveling, etc. After each automatic task is completed, the operation process ends.

[0087] During the entire operation process, the system allows manual control intervention. If the operator finds that the automatically planned trajectory or control instructions do not conform to the actual situation, or some special operations are required, the operator can directly intervene in the actions of the excavator through manual control. This combination of manual control and automatic control not only ensures the intelligence and automation level of the system, but also retains the operator's control right in key operations, improving the flexibility and reliability of the system.

[0088] Through the above-mentioned significant features that are different from the prior art in aspects such as sensor application, task planning and execution, and operation mode, the embodiments of the present invention effectively solve the problem of low operation efficiency of remote excavators, improve the overall operation level, and at the same time enhance the adaptability and controllability of the system.

[0089] In the remote auxiliary control method of the construction machinery in the embodiments of the present invention, through the integration of manual operation and intelligent automatic planning, on the one hand, the subjective judgment ability of the operator is fully utilized, and on the other hand, the operation efficiency is improved by means of the accuracy of the algorithm, and the controllability of the entire system is ensured, and the operation efficiency in complex task scenarios is improved.

[0090] Embodiment 2

[0091] As Figure 6 and Figure 7 shown, the embodiments of the present invention disclose a construction machinery control system, including a construction machinery end, the main controller, communication module, positioning module, camera module, lidar, and inclination sensor of the construction machinery end;

[0092] The number of the inclination sensors is multiple, and the multiple inclination sensors are respectively arranged at the body, boom, arm, and bucket of the construction machinery equipment, and are used for real-time monitoring of the inclination angles of each part and transmitting the monitored inclination angles to the main controller;

[0093] The camera module is used for real-time collecting the surrounding environment images of the intelligent excavator and transmitting the environment images to the main controller; the lidar is used for detecting the obstacles in the surrounding environment and transmitting the detected signals to the main controller to determine the shape, volume, and distance from the bucket of the object;

[0094] The positioning module is used for determining the position information of the intelligent excavator and the bucket;

[0095] The communication module is used for communicating with the background server;

[0096] The main controller is used for receiving the sensor data and performing analysis and processing to execute corresponding operation commands.

[0097] More preferably, the main controller is built with a PID algorithm to precisely control the movements of various parts of the excavator. By building in the PID algorithm, it can precisely adjust the movements of various parts of the excavator according to the sensor data, ensuring that the operation is completed according to the predetermined path and target, and improving the operation accuracy and quality. For example, when excavating a foundation pit with a specific shape, it can precisely control the movement trajectory of the bucket to ensure that the size and shape of the foundation pit meet the requirements.

[0098] More preferably, the control system further includes a background server and a remote control terminal; the background server is communicatively connected to the construction machinery terminal, and the remote control terminal is communicatively connected to the background server;

[0099] The background server is used to receive various operation data transmitted by the construction machinery terminal in real time. The operation data includes the status of the excavator, the position of the bucket, and the operation progress; the background server conducts two-way data interaction with the main controller of the construction machinery terminal, and it is used to send operation instructions, receive the data fed back by the excavator, and monitor and schedule the operation;

[0100] The remote server includes a remote industrial computer, and the remote industrial computer includes a display screen, control buttons, and an operating handle. The display screen is used to provide an intuitive operation interface for the operator. The operator sends operation instructions through the remote control cabin industrial computer and transmits them to the main controller of the construction machinery terminal through the network to achieve remote control of the excavator.

[0101] The intelligent excavator according to the embodiment of the present invention includes the following components: a main controller: used to control the various movements of the excavator, receive sensor data, analyze and process it, and execute corresponding operation commands. A 5G communication device: realizes real-time data transmission with the background management system to ensure instant feedback of data. An RTK positioning device: used to accurately locate the position of the excavator and the bucket, and provide high-precision real-time position data. A camera: used to collect images of the surrounding environment of the excavator in real time to assist in positioning and target recognition. A lidar: used to detect obstacles in the surrounding environment, and obtain the shape, volume of the object, and the distance from the bucket. An inclination sensor: respectively installed on the vehicle body, boom, arm, and bucket, used to monitor the inclination angles of various parts in real time, and obtain the precise postures of various parts of the excavator.

[0102] Through the RTK positioning system and four inclination sensors in the embodiment of the present invention, the intelligent excavator can calculate the status of various parts of the excavator in real time. The position of the bucket not only depends on the positioning of the RTK device, but can also be accurately calibrated by combining the lidar data. The results of the lidar scan are combined with the data of the inclination sensor to correct the position of the bucket in real time, thereby ensuring that the operation of the excavator is more precise.

[0103] Obstacle Detection and Target Location of LiDAR. The LiDAR can accurately scan the front area, identify the shape, volume of the target object, and the distance from the bucket. Through the data of the LiDAR, the excavator can accurately know the position and characteristics of the target object, avoid collisions with surrounding obstacles, and thus improve the operation safety.

[0104] Backstage Management System. The backstage management system is responsible for real-time reading of various operation data of the excavator, including the status of the excavator, the position of the bucket, the operation progress, etc. The backstage system can conduct two-way data interaction with the main controller, send operation instructions, receive the data fed back by the excavator, and monitor and schedule the operation. The system can also analyze the operation situation and provide key data such as operation efficiency and operation progress.

[0105] PID Algorithm Control of the Main Controller. The main controller of the excavator is built-in with the PID algorithm for accurately controlling the actions of various parts of the excavator. By receiving data from sensors, the main controller can adjust the posture of the excavator to ensure that each action is smoothly completed according to the predetermined path and target.

[0106] Implementation of the Automated Operation Process: Taking the target object A and the target area B as examples, assume that the target object A is located on the left side of the excavator. Through LiDAR scanning, the specific position, volume, and shape of the target object A can be accurately known. While the target area B is located on the right side of the excavator. The main controller of the excavator calculates through the data of the RTK device and the LiDAR to automatically generate the operation trajectory. The main controller of the excavator will control the excavator to move the target object A to the target area B according to the position information of the target area B. The system will also real-time feedback information such as the volume of the object in area A, the number of excavations, the excavation sequence, and the time required for the operation, and at the same time synchronize the real-time data to the backstage management system to display the operation progress and various data.

[0107] Intelligent Operation Planning and Optimization. The system can automatically plan the excavation path and operation sequence according to information such as the position, shape, and volume of the target object A. Through the real-time feedback operation data, the backstage management system can optimize and adjust the operation, improve the operation efficiency, and reduce the ineffective operation time. At the same time, the system can also real-time estimate the time required for the operation and generate a detailed operation plan and progress report.

[0108] In the embodiments of the present invention, the data obtained by sensors such as LiDAR, camera, IMU, and RTK are processed, and a convolutional neural network (CNN) or an instance segmentation model such as Mask R-CNN and a point cloud segmentation algorithm (such as PointNet++ or VoxelNet) are used to identify the contour of the target object and extract key information to determine the three-dimensional shape and volume of the object.

[0109] Then, determine the optimal excavation sequence based on the volume and position of the target object and the volume, position, and quantity of the intermediate obstacles to reduce the moving distance or time. In the case of multiple obstacles, a task scheduling algorithm is required. Through reinforcement learning in machine learning, the system learns the optimal strategy in simulation.

[0110] Finally, the mechanical properties of the soil or materials can also be simulated through a physics engine, so that the path planning can be more accurate, such as finite element analysis or discrete element method.

[0111] In the embodiments of the present invention, sensors such as lidar, cameras, IMUs, and RTKs each play a unique role. The lidar can detect obstacles and obtain information such as the distance of objects, the cameras collect visual images, the IMUs provide attitude information, and the RTKs achieve precise positioning. After the fusion of these sensor data, the environment and target objects around the excavator can be comprehensively and accurately perceived.

[0112] The solution of the embodiments of the present invention uses a convolutional neural network (CNN), an instance segmentation model (such as Mask R-CNN), and a point cloud segmentation algorithm (such as PointNet++ or VoxelNet) to process sensor data, which can accurately identify the contour of the target object, extract key information, and then determine the three-dimensional shape and volume of the object, providing accurate basic data for subsequent operations.

[0113] Intelligent excavation sequence planning in the embodiments of the present invention: Determine the optimal excavation sequence according to the volume, position of the target object, and the volume, position, and quantity of the intermediate obstacles, effectively reducing the moving distance or time of the excavator, improving the operation efficiency, and reducing energy consumption.

[0114] Reinforcement learning optimization strategy: For the complex situation of multiple obstacles, the reinforcement learning algorithm in machine learning is adopted to enable the system to continuously learn and optimize the task scheduling strategy in simulation, so that the excavator can more intelligently handle various complex working conditions, further improving the operation efficiency and accuracy.

[0115] Precise path planning: Simulate the mechanical properties of the soil or materials through a physics engine. For example, using finite element analysis or discrete element method, consider the mechanical properties of the soil during path planning, make the operation path of the excavator more in line with the actual situation, avoid operation deviations caused by factors such as soil resistance, and improve the excavation accuracy and quality.

[0116] Through the combination of multiple technologies in the embodiments of the present invention, the intelligent excavator realizes the precise perception of the target object, the optimization of the excavation sequence, and the precision of path planning, thereby significantly improving the operation efficiency of the excavator, reducing energy consumption, and at the same time enhancing the operation safety and accuracy, having high practical value and economic benefits.

[0117] Technical effects corresponding to the embodiments of the present invention: 1. High-precision operation: By combining the RTK positioning system, tilt sensor, and lidar, the position of the bucket can be accurately located, achieving high-precision operation. 2. Automated operation: The excavator can automatically generate an operation trajectory based on the data provided by the background management system and automatically complete the excavation task, greatly improving the operation efficiency and reducing the error of manual operation. 3. Real-time monitoring and feedback: The background management system monitors the operation status of the excavator in real time and promptly feeds back the operation data to ensure the smooth progress of the operation process. 4. High safety: The lidar can accurately detect target objects and obstacles, avoiding collision accidents and improving the operation safety. 5. Intelligent optimization of the operation plan: Through intelligent planning and real-time data feedback, the system can optimize the operation path and sequence, improving the overall operation efficiency.

[0118] An intelligent excavator and its automatic operation system provided by the present invention adopt a variety of advanced technologies, can accurately control the operation actions of the excavator, automatically execute the excavation task, and perform real-time data interaction with the background management system. The intelligent excavator of the present invention integrates hardware such as an RTK positioning system, 5G communication equipment, lidar, and tilt sensor to achieve precise operation path planning and dynamic monitoring, improving the operation efficiency and accuracy.

[0119] Embodiment III

[0120] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of an electronic device disclosed in an embodiment of the present invention. The electronic device can be a computer, a server, etc. Of course, in certain cases, it can also be an intelligent device such as a mobile phone, a tablet computer, and a monitoring terminal, as well as an image acquisition device with processing functions. As Figure 8 shown, the electronic device may include:

[0121] A memory 510 storing executable program code;

[0122] A processor 520 coupled to the memory 510;

[0123] Wherein, the processor 520 calls the executable program code stored in the memory 510 and executes some or all of the steps in the remote auxiliary control method of the construction machinery in Embodiment I.

[0124] An embodiment of the present invention discloses a computer-readable storage medium that stores a computer program, wherein the computer program enables a computer to execute some or all of the steps in the remote auxiliary control method of the construction machinery in Embodiment I.

[0125] An embodiment of the present invention also discloses a computer program product. When the computer program product runs on a computer, the computer is caused to execute some or all of the steps in the remote auxiliary control method of the construction machinery in the first embodiment.

[0126] An embodiment of the present invention also discloses an application publishing platform. The application publishing platform is used to publish a computer program product. When the computer program product runs on a computer, the computer is caused to execute some or all of the steps in the remote auxiliary control method of the construction machinery in the first embodiment.

[0127] In various embodiments of the present invention, it should be understood that the magnitude of the sequence numbers of the various processes does not necessarily mean the order of execution. The order of execution of the various processes should be determined according to their functions and internal logics, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0128] The unit described as a separated component may or may not be physically separated, and the component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, in each embodiment of the present invention, the various functional units may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for causing a computer device (which may be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0131] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0132] Those of ordinary skill in the art can understand that some or all of the steps in the various methods of the embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disc memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0133] The remote auxiliary control method, system, electronic device, and storage medium of the construction machinery disclosed in the embodiments of the present invention have been introduced in detail above. Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A remote auxiliary control method for engineering machinery, characterized in that: include: Receive the task planning information configured by the user through the remote cabin, and determine the skill type and task parameters of each stage according to the task planning information; Collecting on-site environmental data through a sensor assembly disposed on the engineering machinery, and forming environmental vector information according to the on-site environmental data; Generate a trajectory sequence of each joint of the engineering machinery working device according to the skill type and task parameters of each stage and the environmental vector information according to a set period, and send the trajectory sequence of each joint of the engineering machinery working device to the engineering machinery end; The trajectory sequence of each joint of the engineering machinery working device is converted into a trajectory control vector, and the working state of the engineering machinery is trajectory controlled according to the trajectory control vector and the acquired state monitoring information of each joint of the engineering machinery working device.

2. The remote auxiliary control method for engineering machinery according to claim 1, characterized in that: Before receiving the mission planning information configured by the user through the remote module, the method further includes: Receiving operation mode information determined by the user through the remote cabin, if the operation mode information is a manual operation mode, receiving operation instruction information sent by the user through the remote cabin in real time, and controlling the working state of the engineering machinery working device according to the operation instruction information; If the operation mode information is the automatic operation mode, the mission planning information configured by the user through the remote cabin is received.

3. The remote auxiliary control method for engineering machinery according to claim 1, characterized in that: The auxiliary control method further includes: During the operation, the user's mode conversion instruction is received in real time. If the mode conversion instruction is to switch to manual mode, the operation instruction information sent by the user through the remote cabin is received in real time to intervene in the action state of the engineering machinery through manual control.

4. The remote auxiliary control method for engineering machinery according to claim 1, characterized in that: After collecting the on-site environmental data by means of the sensor assembly provided on the engineering machinery, the method further includes: If the on-site environmental data does not meet the automatic operation conditions, a prompt message is sent to the remote cabin to remind the user to perform manual operation.

5. The remote auxiliary control method for engineering machinery according to claim 1, characterized in that: The skill types include excavation task type, loading task type and leveling task type; The task parameters include excavation depth value, loading quantity and land flatness; The forming of environmental vector information according to the on-site environmental data comprises: Identify the on-site environmental data to determine the position information, shape information and volume information of the corresponding target and obstacle, and generate corresponding environmental vector information according to the position information, shape information and volume information of the target and obstacle; If there are multiple obstacles, the optimal excavation sequence is determined according to the volume position number of the intermediate obstacles.

6. A control system for construction machinery, characterized in that: It includes an engineering machinery end, a main controller of the engineering machinery end, a communication module, a positioning module, a camera module, a laser radar and a tilt sensor; There are multiple inclination sensors, which are respectively arranged on the body, the boom, the forearm and the bucket of the engineering machinery equipment, for real-time monitoring of the inclination angle of each part, and transmitting the monitored inclination angle to the main controller; The camera module is used to collect images of the surrounding environment of the intelligent excavator in real time and transmit the images to the main controller; the laser radar is used to detect obstacles in the surrounding environment and transmit the detected signals to the main controller to determine the shape, volume and distance of the object from the bucket; The positioning module is used to determine the position information of the intelligent excavator and the bucket; The communication module is used to communicate with the backend server; The main controller is used to receive sensor data and perform analysis and processing to execute corresponding operation commands.

7. The construction machinery control system according to claim 6, characterized in that: The main controller has a built-in PID algorithm, through which the actions of various parts of the excavator are accurately controlled.

8. The construction machinery control system according to claim 6, characterized in that: The control system further comprises a background service end and a remote control end; the background service end is in communication connection with the engineering machinery end, and the remote control end is in communication connection with the background service end; The backend server is used to receive various operation data transmitted by the engineering machinery end in real time, and the operation data includes the state of the excavator, the position of the bucket and the operation progress; the backend server performs two-way data interaction with the main controller of the engineering machinery end, and is used to send operation instructions, receive data fed back by the excavator, and monitor and schedule the operation; The remote service end includes a remote industrial computer, which includes a display screen, control buttons and an operating handle. The display screen is used to provide an intuitive operating interface for the operator. The operator sends operating instructions through the industrial computer in the remote control cabin, which are transmitted to the main controller at the engineering machinery end via the network to realize remote control of the excavator.

9. An electronic device, characterized in that: include: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the remote auxiliary control method for engineering machinery according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the remote auxiliary control method for an engineering machine according to any one of claims 1 to 5.

Images