Vision-based crane control system and method

Abstract

The invention provides a crane control system and method based on vision. The system comprises an equipment end and an operation end, the equipment end comprises an equipment end visual module and an equipment end control module, and the operation end comprises an operation end control module and an operation end display module; the equipment end visual module is used for collecting video stream data and real-time state data of the crane in the working process and sending the video stream data and the real-time state data to the operation end display module; the operation end display module is used for receiving and displaying the video stream data and the real-time state data; the operation end control module is used for sending a crane control instruction to the equipment end control module, and the equipment end control module is used for receiving and analyzing the crane control instruction and executing a corresponding action of the crane control instruction according to an analysis result. According to the invention, remote control and real-time monitoring are realized, and the real-time performance, safety and precision of remote operation of the crane are improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial control technology, and in particular to a vision-based crane control system and method. Background Art

[0002] Existing crane remote control systems often use industrial remote controls, Wi-Fi, LoRa, and other communication methods for communication. End-to-end latency generally ranges from 50-200ms, resulting in high control latency. Furthermore, cranes operate in complex environments, and the single communication link used by industrial remote controls, Wi-Fi, LoRa, and other methods to transmit remote control signals is prone to signal loss and has weak anti-interference capabilities.

[0003] Currently, only 20% of high-end cranes are equipped with industrial image acquisition equipment, and these typically use 720P resolution lenses with a narrow 60° viewing angle. This can result in blind spots of up to 30° when hoisting large steel structures, forcing operators to frequently switch viewing angles and resulting in insufficient visual feedback. Existing crane remote control systems struggle to meet the demands for precise remote control and visual operations in complex environments.

[0004] In summary, the existing technology has problems such as high latency, unstable signal, and insufficient visual feedback. Summary of the Invention

[0005] The present invention provides a vision-based crane control system and method to address the defects of high delay, unstable signal and insufficient visual feedback in the prior art, and to achieve vision-based crane control with good real-time performance, high safety and high precision.

[0006] The present invention provides a vision-based crane control system, comprising an equipment end and an operation end;

[0007] The device end includes a device end visual module and a device end control module, and the operation end includes an operation end control module and an operation end display module;

[0008] The device-side visual module is used to collect video stream data and real-time status data of the crane during operation, and send the video stream data and the real-time status data to the operation-side display module;

[0009] The operation terminal display module is used to receive and display the video stream data and the real-time status data;

[0010] The operation-end control module is used to send a crane control instruction to the device-end control module, wherein the crane control instruction is generated based on the video stream data and the real-time status data;

[0011] The device-side control module is used to receive and analyze the crane control instruction, and execute the corresponding action of the crane control instruction according to the analysis result;

[0012] Among them, the device-side control module and the operation-side control module are connected through a first communication link; the device-side visual module and the operation-side display module are connected through a second communication link.

[0013] According to a vision-based crane control system provided by the present invention, the first communication link is an ExpressLRS wireless link; and / or

[0014] The second communication link is a wireless link based on Wi-Fi 6 or a wireless link based on a 5G network.

[0015] According to a vision-based crane control system provided by the present invention, the device-side vision module includes a first preset number of image acquisition devices and / or a second preset number of sensors;

[0016] The image acquisition device is used to acquire video stream data of the crane during operation, and the sensor is used to acquire real-time status data of the crane during operation.

[0017] According to a vision-based crane control system provided by the present invention, the real-time status data includes at least one of the real-time position of the equipment, equipment operating status information, equipment motor current, equipment power supply voltage and limit status information.

[0018] According to a vision-based crane control system provided by the present invention, the real-time status data also includes fault alarm information.

[0019] According to a vision-based crane control system provided by the present invention, the operating end also includes a communication detection module, which is used to detect first link quality information of the first communication link and second link quality information of the second communication link, and send the first link quality information and the second link quality information to the operating end display module.

[0020] The present invention also provides a vision-based crane control method, comprising:

[0021] Collecting video stream data and real-time status data of the crane during operation based on the device-side vision module, and sending the video stream data and the real-time status data to the operation-side display module;

[0022] Receive and display the video stream data and the real-time status data based on the operation terminal display module;

[0023] Based on the operation end control module, a crane control instruction is sent to the device end control module, wherein the crane control instruction is generated according to the video stream data and the real-time status data;

[0024] receiving and parsing the crane control instruction based on the device-side control module, and executing a corresponding action of the crane control instruction according to the parsing result;

[0025] Among them, the device-side control module and the operation-side control module are connected through a first communication link; the device-side visual module and the operation-side display module are connected through a second communication link.

[0026] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the vision-based crane control system described above is implemented.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-described vision-based crane control systems.

[0028] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-described vision-based crane control systems.

[0029] The vision-based crane control system and method provided by the present invention obtains video stream data and real-time status data through the device-side vision module, and displays them on the operation-side display module. Crane control instructions are generated based on the displayed content, and the crane control instructions are sent to the device-side control module based on the operation-side control module. The device control-side module controls the crane according to the crane control instructions. The present invention configures a first communication link and a second communication link to connect the device side and the operation side to achieve remote control and real-time monitoring, thereby improving the real-time, safety and accuracy of remote crane operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 Schematic diagram of the structure of the vision-based crane control system provided by the present invention;

[0032] Figure 21 is a flow chart of a vision-based crane control method provided by the present invention;

[0033] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0035] The following combination Figure 1 Describe the vision-based crane control system of the present invention, Figure 1 Schematic diagram of the structure of the crane control system based on vision provided by the present invention; Figure 1 As shown, the present invention provides a vision-based crane control system, including an equipment end 110 and an operation end 120 .

[0036] The equipment end 110 refers to mechanical equipment including a crane, and in particular refers to a crane end, and the operation end 120 refers to an industrial control console or controller for operating the crane.

[0037] The device end 110 includes a device end visual module 112 and a device end control module 114 , and the operation end 120 includes an operation end control module 122 and an operation end display module 124 .

[0038] The device-side vision module 112 is used to collect video stream data and real-time status data of the crane during operation, and send the video stream data and the real-time status data to the operation-side display module 124 .

[0039] The device-side visual module 112 is used to collect all status data of the crane during the working process, including video stream data and real-time status data. The relevant data is packaged and sent to the operator-side display module 124 for viewing and analysis by the operator 120.

[0040] The operation terminal display module 124 is used to receive and display the video stream data and the real-time status data.

[0041] The operation terminal display module 124 displays the received video stream data and the real-time status data on a display interface of the operation terminal 120 .

[0042] The operation-end control module 122 is configured to send a crane control instruction to the device-end control module 114 , wherein the crane control instruction is generated based on the video stream data and the real-time status data.

[0043] It should be explained that the crane control instructions can be automatically generated after analyzing the video stream data and real-time status data, or can be manually input by the operator after analyzing the video stream data and real-time status data. The present invention does not impose any restrictions on this.

[0044] Crane control commands include automatic crane control commands and starter crane manual control commands, which enable the crane to move smoothly, operate, stop, and limit its range of motion. Commands include controlling the crane's movement, lifting, rotation, and grabbing.

[0045] The device-side control module 114 is configured to receive and analyze the crane control instruction, and execute an action corresponding to the crane control instruction according to the analysis result.

[0046] After receiving the control instruction, the device-side control module 114 analyzes and executes the corresponding action to drive the crane to move until the corresponding operation is completed.

[0047] The device-side control module 114 and the operation-side control module 122 are connected via a first communication link; the device-side visual module 112 and the operation-side display module 124 are connected via a second communication link.

[0048] That is, the information exchange between the device-side control module 114 and the operation-side control module 122 is performed through the first communication link, and the information exchange between the device-side control module 114 and the operation-side control module 122 is performed through the second communication link.

[0049] By transmitting crane control instructions and status data using different communication links, the communication links can be set up according to the characteristics of each data, minimizing transmission delays and ensuring signal stability.

[0050] The vision-based crane control system provided by the present invention obtains video stream data and real-time status data through the device-side vision module, and displays them on the operation-side display module. Crane control instructions are generated according to the displayed content, and the crane control instructions are sent to the device-side control module based on the operation-side control module. The device control-side module controls the crane according to the crane control instructions. The present invention configures a first communication link and a second communication link to connect the device side and the operation side to realize remote control and real-time monitoring, thereby improving the real-time, safety and accuracy of remote crane operations.

[0051] The first communication link and the second communication link are further described below. In some embodiments, the first communication link is an ExpressLRS wireless link; and / or

[0052] The second communication link is a wireless link based on Wi-Fi 6 or a wireless link based on a 5G network.

[0053] Specifically, ExpressLRS, as an open-source, high-performance, low-latency, long-distance wireless control protocol, has a high refresh rate and anti-interference capability, and has the potential to be applied to remote control of cranes. The present invention sets the first communication link to the ExpressLRS wireless link to achieve dynamic frequency hopping and enhance anti-interference capability; achieves a low latency of 4ms and a refresh rate of 500Hz (500Hz requires a 2.4GHz main frequency) for control instructions, and completes precise control of cranes.

[0054] Furthermore, in some embodiments, the present invention sets the second communication link to a wireless link based on Wi-Fi 6 or 5G network to achieve high-definition and high-speed video transmission.

[0055] In the normal communication state, the present invention completes instruction transmission and action execution through the first communication link, and the second communication link returns the operation screen in real time.

[0056] This invention utilizes a low-latency wireless control link and a high-definition video return link to enable remote control and real-time monitoring, improving the real-time, safety, and precision of remote crane operations. The system features anti-interference links, low wireless communication latency, and high-frequency control. This invention enables precise remote control and high-definition visual monitoring of cranes, enhancing real-time, safety, and intelligent operation. It is particularly suitable for remote crane operations in smart construction sites, ports, and high-risk environments.

[0057] The device-side vision module 112 will be further described below. In some embodiments, the device-side vision module 112 includes a first preset number of image acquisition devices and / or a second preset number of sensors;

[0058] The image acquisition device is used to acquire video stream data of the crane during operation, and the sensor is used to acquire real-time status data of the crane during operation.

[0059] Specifically, in this embodiment, image acquisition equipment and sensors are used to collect all crane status data during operation. The image acquisition equipment is used to collect video stream data, and the sensors are used to collect real-time status data. It should be noted that the sensors utilize media such as lasers and microwaves and have functions such as distance measurement, speed measurement, and obstacle avoidance.

[0060] Furthermore, the number of image acquisition devices is recorded as a first preset number, and the number of sensors is recorded as a second preset number. It should be noted that the first preset number and the second preset number can be set according to circumstances, and the present invention does not impose any limitation on this.

[0061] In some embodiments, cameras are used as image acquisition devices and are placed at important locations such as the front, back, top, bottom, left, and right of the crane, as well as at key locations such as the traveling mechanism, operating mechanism, reel, and wheel rails.

[0062] Furthermore, in some embodiments, the real-time status data includes at least one of the real-time position of the device, device operating status information, device motor current, device power supply voltage, and limit status information.

[0063] In some embodiments, the real-time status data also includes fault alarm information.

[0064] That is to say, the fault alarm information is alarm information generated by a fault during the operation of the crane, and the fault includes but is not limited to a fault in the electrical, mechanical, hydraulic, control system, etc.

[0065] Furthermore, in some embodiments, the operation end 120 also includes a communication detection module 126, which is used to detect the first link quality information of the first communication link and detect the second link quality information of the second communication link, and send the first link quality information and the second link quality information to the operation end display module 124.

[0066] In this embodiment, the operation terminal 120 has a built-in communication detection module 126, which is used to detect the quality of the first communication link and the second communication link, obtain the first link quality information and the second link quality information, and send them to the operation terminal display module 124 for visual display.

[0067] Furthermore, in some embodiments, if a communication anomaly is detected, a slow-down procedure is initiated. In some embodiments, the slow-down procedure controls the crane to stop smoothly, thereby improving operational safety.

[0068] The link quality can be detected by receiving and parsing instructions and data from the link to determine the connectivity and stability of the link.

[0069] The vision-based crane control method provided by the present invention is described below. The vision-based crane control method described below and the vision-based crane control system described above can refer to each other. Figure 2 This is a flow chart of the vision-based crane control method provided by the present invention, as shown in FIG. Figure 2 As shown, the method includes the following steps:

[0070] Step 210: Collecting video stream data and real-time status data of the crane during operation based on the device-side vision module, and sending the video stream data and the real-time status data to the operation-side display module;

[0071] Step 220: receiving and displaying the video stream data and the real-time status data based on the operation terminal display module;

[0072] Step 230: Based on the operation-side control module, sending a crane control instruction to the equipment-side control module, wherein the crane control instruction is generated according to the video stream data and the real-time status data;

[0073] Step 240: receiving and parsing the crane control instruction based on the device-side control module, and executing a corresponding action of the crane control instruction according to the parsing result;

[0074] Among them, the device-side control module and the operation-side control module are connected through a first communication link; the device-side visual module and the operation-side display module are connected through a second communication link.

[0075] According to a vision-based crane control method provided by the present invention, the first communication link is an ExpressLRS wireless link; and / or

[0076] The second communication link is a wireless link based on Wi-Fi 6 or a wireless link based on a 5G network.

[0077] According to a vision-based crane control method provided by the present invention, the device-side vision module includes a first preset number of image acquisition devices and / or a second preset number of sensors;

[0078] The image acquisition device is used to acquire video stream data of the crane during operation, and the sensor is used to acquire real-time status data of the crane during operation.

[0079] According to a vision-based crane control method provided by the present invention, the real-time status data includes at least one of the real-time position of the equipment, equipment operating status information, equipment motor current, equipment power supply voltage and limit status information.

[0080] According to a vision-based crane control method provided by the present invention, the real-time status data also includes fault alarm information.

[0081] According to a vision-based crane control method provided by the present invention, the operating end also includes a communication detection module, which is used to detect first link quality information of the first communication link and second link quality information of the second communication link, and send the first link quality information and the second link quality information to the operating end display module.

[0082] The vision-based crane control method provided by the present invention obtains video stream data and real-time status data through a device-side vision module, and displays them on an operator-side display module. Crane control instructions are generated based on the displayed content, and the operator-side control module sends the crane control instructions to the device-side control module. The device control-side module controls the crane according to the crane control instructions. The present invention configures a first communication link and a second communication link to connect the device side and the operator side, thereby realizing remote control and real-time monitoring, thereby improving the real-time, safety and accuracy of remote crane operations.

[0083] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor (processor) 310, a communication interface (Communications Interface) 320, a memory (memory) 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330 to execute a vision-based crane control system, which includes: based on the device-side vision module, collecting video stream data and real-time status data of the crane during operation, and sending the video stream data and the real-time status data to the operation-side display module; based on the operation-side display module, receiving and displaying the video stream data and the real-time status data; based on the operation-side control module, sending the crane control instruction to the device-side control module, wherein the crane control instruction is generated based on the video stream data and the real-time status data; based on the device-side control module, receiving and parsing the crane control instruction, and executing the corresponding action of the crane control instruction according to the parsing result; wherein, the device-side control module and the operation-side control module are connected via a first communication link; the device-side vision module and the operation-side display module are connected via a second communication link.

[0084] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vision-based crane control system provided by the above-mentioned methods. The method includes: based on the device-side vision module, collecting video stream data and real-time status data of the crane during operation, and sending the video stream data and the real-time status data to the operation-side display module; based on the operation-side display module, receiving and displaying the video stream data and the real-time status data; based on the operation-side control module, sending a crane control instruction to the device-side control module, wherein the crane control instruction is generated based on the video stream data and the real-time status data; based on the device-side control module, receiving and parsing the crane control instruction, and executing the corresponding action of the crane control instruction according to the parsing result; wherein, the device-side control module and the operation-side control module are connected via a first communication link; the device-side vision module and the operation-side display module are connected via a second communication link.

[0086] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vision-based crane control system provided by the above-mentioned methods, the method comprising: collecting video stream data and real-time status data of the crane during operation based on a device-side vision module, and sending the video stream data and the real-time status data to an operation-side display module; receiving and displaying the video stream data and the real-time status data based on the operation-side display module; sending a crane control instruction to the device-side control module based on the operation-side control module, wherein the crane control instruction is generated based on the video stream data and the real-time status data; receiving and parsing the crane control instruction based on the device-side control module, and executing the corresponding action of the crane control instruction according to the parsing result; wherein the device-side control module and the operation-side control module are connected via a first communication link; and the device-side vision module and the operation-side display module are connected via a second communication link.

[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A vision-based crane control system, characterized in that: Including device side and operation side; The device end includes a device end visual module and a device end control module, and the operation end includes an operation end control module and an operation end display module; The device-side visual module is used to collect video stream data and real-time status data of the crane during operation, and send the video stream data and the real-time status data to the operation-side display module; The operation terminal display module is used to receive and display the video stream data and the real-time status data; The operation-end control module is used to send a crane control instruction to the device-end control module, wherein the crane control instruction is generated based on the video stream data and the real-time status data; The device-side control module is used to receive and analyze the crane control instruction, and execute the corresponding action of the crane control instruction according to the analysis result; Among them, the device-side control module and the operation-side control module are connected through a first communication link; the device-side visual module and the operation-side display module are connected through a second communication link.

2. The vision-based crane control system according to claim 1, characterized in that The first communication link is an ExpressLRS wireless link; and / or The second communication link is a wireless link based on Wi-Fi 6 or a wireless link based on a 5G network.

3. The vision-based crane control system according to claim 1, characterized in that The device-side vision module includes a first preset number of image acquisition devices and / or a second preset number of sensors; The image acquisition device is used to acquire video stream data of the crane during operation, and the sensor is used to acquire real-time status data of the crane during operation.

4. The vision-based crane control system according to claim 1, characterized in that The real-time status data includes at least one of the real-time position of the device, the operating status information of the device, the motor current of the device, the power supply voltage of the device, and the limit status information.

5. The vision-based crane control system according to claim 1, characterized in that: The real-time status data also includes fault alarm information.

6. The vision-based crane control system according to claim 1, characterized in that The operation terminal further includes a communication detection module, which is used to detect first link quality information of the first communication link and second link quality information of the second communication link, and send the first link quality information and the second link quality information to the operation terminal display module.

7. A vision-based crane control method, characterized in that: include: Collecting video stream data and real-time status data of the crane during operation based on the device-side vision module, and sending the video stream data and the real-time status data to the operation-side display module; Receive and display the video stream data and the real-time status data based on the operation terminal display module; Based on the operation end control module, a crane control instruction is sent to the device end control module, wherein the crane control instruction is generated according to the video stream data and the real-time status data; receiving and parsing the crane control instruction based on the device-side control module, and executing a corresponding action of the crane control instruction according to the parsing result; Among them, the device-side control module and the operation-side control module are connected through a first communication link; the device-side visual module and the operation-side display module are connected through a second communication link.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the vision-based crane control method according to claim 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vision-based crane control method according to claim 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the vision-based crane control method according to claim 7 is implemented.

Images