Crane operation monitoring method and system based on digital twinning

Abstract

This invention discloses a crane operation monitoring method and system based on digital twins. This solution ingeniously utilizes digital twin technology to construct a visualized backend monitoring system, providing remote monitoring services for the on-site operation of gantry cranes. Simultaneously, it records the gantry crane's operation execution logs to determine the equipment's health status (i.e., indirectly judged through command response sensitivity or drive effect). Furthermore, it determines whether foreign objects are interfering with the crane's movement by analyzing images of a preset range along the crane track. Moreover, it indirectly monitors the gantry crane's vibration by combining differences in image frames within the images, providing a sound technical monitoring strategy for the safe operation of gantry cranes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crane operation monitoring, in particular to a crane operation monitoring method and system based on digital twinning. BACKGROUND

[0002] As a crane with a high lifting load, the gantry crane is widely used for the transfer of large building materials, containers and other goods indoors and outdoors. In outdoor environments, the track and power equipment of the gantry crane may have problems such as device failure or poor operation due to outdoor environmental erosion and foreign object interference. The objects transferred by the gantry crane mostly have a high weight, and if an accident occurs, it may cause an incalculable risk. Therefore, it is of great practical significance to monitor the operation of the outdoor gantry crane.

[0003] Although the present technical personnel in the field has set up sensors and other devices based on the Internet of Things on the gantry crane to assist in collecting device state information during the operation of the gantry crane, for background monitoring, digital feedback is not intuitive, so it has high requirements for the experience and knowledge accumulation of background monitoring personnel. If the digital twinning technology is combined to display the working site in a visual form in the background, it will greatly improve the intuitiveness and work efficiency of the background monitoring. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a crane operation monitoring method and system based on digital twinning, which is responsive, intuitive, and reliable and convenient to implement.

[0005] In order to achieve the above technical purpose, the technical solution adopted by the present application is:

[0006] A crane operation monitoring method based on digital twinning for monitoring the operation of a gantry crane set up outdoors, the gantry crane has a positioning unit on both sides of the gantry, the load cart and the hook for positioning its position, the operation monitoring method comprises:

[0007] A reference coordinate system is established for the working site of the gantry crane according to the preset conditions, and the terrain and fixed facilities in the preset area range below and on both sides of the gantry of the gantry crane are profile scanned to obtain a digital model of the construction site;

[0008] The gantry crane is profile scanned according to the preset conditions to form a crane digital model;

[0009] Real-time positioning data of positioning units on the gantry, the trolley and the hook of the gantry crane are acquired, the real-time positioning data is combined with the digital model of the crane and mapped into the digital model of the construction site for visual display.

[0010] In response to a work start signal of the gantry crane, work state information and operation records of the gantry crane are acquired according to preset conditions, and an operation execution log is generated.

[0011] The operation execution log is acquired, data judgment is performed thereon according to preset conditions, a warning information is generated and output.

[0012] As a possible implementation, further, the operation execution log of the present scheme includes a time period in which the gantry, the trolley or the hook of the gantry crane changes from a first state to a second state and keeps constant speed or changes less than a preset amplitude within a preset time length after the operation instruction of the gantry crane is issued.

[0013] As a preferred selection implementation, preferably, the first state is moving and the second state is stopping.

[0014] The first state is stopping and the second state is moving.

[0015] As a possible implementation, further, the present scheme further includes a pair of image acquisition devices arranged at both ends of the gantry of the gantry crane, the image taking end of the pair of image acquisition devices faces downward and is used for capturing images of the tracks on both sides of the gantry, wherein when the gantry crane is stopped, the tracks are located in the middle of the captured image of the image acquisition device.

[0016] The operation monitoring method further includes:

[0017] In response to a work start signal of the gantry crane, real-time image acquisition is performed on the tracks on both sides of the gantry, track images are generated and introduced into a detection neural network, foreign matters in the images are detected and identified by the detection neural network, and a detection result is output.

[0018] The detection result is obtained, and a warning information is output according to preset conditions.

[0019] As a preferred selection implementation, preferably, the training method of the detection neural network includes:

[0020] Images of the tracks connected by the gantry are acquired, foreign matters in the images are labeled and marked with their danger levels to form a training data set;

[0021] A preset amount of data in the training data set is selected as training data, and the training data is introduced into the neural network for training.

[0022] selecting a preset amount of data from the remaining data in the training data set as verification data, and importing the verification data into the trained neural network for verification,

[0023] When the verification result does not meet the preset requirement, the training data is reimported into the trained neural network for further training for a preset number of times until the verification result meets the preset requirement, and a detection neural network is obtained.

[0024] As a preferred selection implementation, preferably, the scheme further comprises:

[0025] In response to a signal that the portal frame of the portal crane moves on the track, real-time track images are acquired, and the track images are segmented into image frames at a preset time interval;

[0026] The image frames are acquired, a reference line is established at the center of the image frame, and the track is positioned and profiled in the image frame to obtain a track profile;

[0027] With the two side profiles in the length direction of the track as a reference, a center reference line parallel to the length direction of the track is established, and then the distance AL between the reference line and the center reference line is calculated in real time and output, wherein the center reference line coincides with the reference line of the image frame when the portal crane is stopped, that is, the distance AL between the reference line and the center reference line is 0;

[0028] The AL is acquired and compared with a preset threshold to generate a first comparison result, and the difference between the AL on both sides of the portal frame is also calculated and compared with a preset threshold to generate a second comparison result;

[0029] The first comparison result and / or the second comparison result are acquired, and a warning information and / or an operation intervention instruction are output according to a preset condition.

[0030] As a preferred selection implementation, preferably, the track image of the scheme is a video, and the frame rate is 25, 30, 40, 50 or 60 frames per second;

[0031] In the image frames segmented at a preset time interval, the time difference between adjacent image frames is not more than 0.2-0.5 seconds.

[0032] As a preferred selection implementation, preferably, the scheme acquires an operation execution log, and according to a preset condition, the operation intervention instruction is output according to a preset condition, the operation intervention instruction includes a stop, brake or pause instruction.

[0033] As a preferred selection implementation, preferably, the warning information of the scheme is also mapped to the visualization output in the digital model of the construction site.

[0034] Based on the above, the application also provides a crane operation monitoring system based on digital twinning, which comprises:

[0035] The positioning units are multiple and arranged on the gantry, the trolley and the hook of the gantry crane;

[0036] The model construction unit is used for establishing a reference coordinate system for the working site of the gantry crane according to a preset condition, and is also used for obtaining the topography and fixed facilities in the preset area range below and on both sides of the gantry of the gantry crane through contour scanning to establish a construction site digital model, and is further used for forming a crane digital model through contour scanning of the gantry crane according to a preset condition;

[0037] The digital twinning model construction unit is used for obtaining real-time positioning data of the positioning units on the gantry, the trolley and the hook of the gantry crane, combining the real-time positioning data with the crane digital model and mapping the crane digital model into the construction site digital model for visual display;

[0038] The construction monitoring unit is used for obtaining working state information and operation records of the gantry crane according to a preset condition in response to a working start signal of the gantry crane, and generating an operation execution log;

[0039] The early warning processing unit is used for obtaining the operation execution log, judging the operation execution log according to a preset condition, generating early warning information and outputting the early warning information.

[0040] Based on the above, the application also provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to realize the crane operation monitoring method based on digital twinning.

[0041] Compared with the prior art, the application has the beneficial effects that: the scheme ingeniously constructs a visual background monitoring through digital twinning technology to provide remote monitoring services for the on-site work of the gantry crane, and at the same time, the operation execution log of the gantry crane is recorded to judge the health degree of the equipment (i.e., indirectly judging through instruction response sensitivity or driving effect), in addition, whether there is foreign matter interference with the movement of the crane track is judged through the image of the preset range of the crane track, the scheme further indirectly monitors the vibration problem of the gantry crane by combining the differences of the image frames in the image, and provides a good technical monitoring strategy for the safe work of the gantry crane. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 is one of the schematic diagrams of the brief implementation process of the monitoring method of the present application;

[0044] Figure 2 is the second schematic diagram of the brief implementation process of the monitoring method of the present application;

[0045] Figure 3 is the third schematic diagram of the brief implementation process of the monitoring method of the present application;

[0046] Figure 4 is the schematic diagram of the center reference line and the reference line in the monitoring method of the present application;

[0047] Figure 5 is the schematic diagram of the brief unit connection of the system of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in combination with the drawings and embodiments. It is particularly pointed out that the following embodiments are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following embodiments are only some embodiments of the present application, not all embodiments, and all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] As shown in Figure 1 , the present embodiment is a crane operation monitoring method based on digital twinning, which is used for work operation monitoring of a gantry crane set outdoors. The gantry crane is provided with a positioning unit for positioning its position on both sides of the gantry, the load trolley and the hook. The operation monitoring method comprises:

[0050] S01, establishing a reference coordinate system for the work site of the gantry crane according to the preset conditions, and simultaneously, contour scanning is performed on the terrain and fixed facilities in the preset area range below and on both sides of the gantry of the gantry crane to obtain a digital model of the construction site;

[0051] S02, contour scanning the gantry crane according to the preset conditions to form a crane digital model;

[0052] S03, obtaining real-time positioning data of the portal frame, load trolley and hook positioning unit of the portal crane, combining the real-time positioning data with the crane digital model and mapping them into the construction site digital model for visual display;

[0053] S04, in response to a work start signal of the portal crane, obtaining work state information and operation records of the portal crane according to preset conditions, and generating an operation execution log;

[0054] S05, obtaining the operation execution log, judging the data according to preset conditions, generating warning information and outputting.

[0055] In order to facilitate the recording of operation nodes and the formation of detection tasks, in the scheme, the operation execution log includes the time when the portal frame, load trolley or hook of the portal crane changes from a first state to a second state and keeps constant speed or changes less than a preset amplitude within a preset time length after the operation instruction of the portal crane is issued; as a preferred selection implementation manner, preferably, the first state is moving and the second state is stopping (i.e. the process of changing from dynamic to static); when the first state is stopping, the second state is moving (i.e. the process of changing from static to dynamic).

[0056] Under the above scheme, the acceleration time of the portal frame from static to uniform speed can be recorded, and the health degree of the driving device can be judged by combining the preset query table; the braking time of the portal frame from moving to static can also be recorded to judge the health degree of the braking device, and the time difference between instruction issuance and formal execution can also be used to judge whether there is an electrical fault causing the equipment to delay the instruction.

[0057] In combination with Figure 2 As another possible implementation manner, further, the portal crane of the scheme is further provided with a pair of image acquisition devices at both ends of the portal frame, the image taking end of the pair of image acquisition devices faces downward and is used to capture the image of the track on both sides of the portal frame, wherein when the portal crane stops, the track is located in the middle of the captured image of the image acquisition device.

[0058] The operation monitoring method further includes:

[0059] S06, in response to a work start signal of the portal crane, performing real-time image acquisition on the tracks on both sides of the portal frame, generating track images and importing them into a detection neural network, and outputting detection results by the detection neural network.

[0060] S07, obtaining the detection results and outputting warning information according to preset conditions.

[0061] The training method of the detection neural network of the scheme includes:

[0062] A01, acquire the image of the track connected by the gantry, label the foreign matter in the image and mark its danger level to form a training data set;

[0063] A02, select a preset amount of data in the training data set as training data and import it into the neural network for training;

[0064] A03, select a preset amount of data from the remaining data in the training data set as validation data and import it into the trained neural network for validation,

[0065] When the validation result does not meet the preset requirement, the training data is reimported into the trained neural network for further training for a preset number of times until the validation result meets the preset requirement, and the detection neural network is obtained.

[0066] In combination Figure 3 and Figure 4 As a preferred selection implementation, preferably, the scheme further comprises:

[0067] S08, in response to the signal of the gantry moving on the track, real-time acquisition of the track image, and segmentation into image frames at a preset time interval;

[0068] S09, acquiring the image frame, establishing a reference line at the center of the image frame, and positioning and contour recognizing the track in the image frame to obtain the track contour;

[0069] S10, taking the two side contours of the track length direction as a reference, establishing a center reference line parallel to the track length direction, and then real-time calculating and outputting the distance AL between the reference line and the center reference line, wherein the center reference line coincides with the reference line of the image frame when the gantry crane is stopped, that is, the distance AL between the reference line and the center reference line is 0;

[0070] S11, acquiring AL and comparing it with a preset threshold to generate a first comparison result, and also calculating the difference value of AL on both sides of the gantry and comparing the difference value with a preset threshold to generate a second comparison result;

[0071] S12, acquiring the first comparison result and / or the second comparison result, and presetting the output of the warning information and / or the operation intervention instruction.

[0072] Through the above scheme, the vibration conditions of the two sides of the gantry can be roughly judged, so as to roughly evaluate the cooperation of the track and the gantry and the working state of the driving device below the gantry (that is, to evaluate the health condition of the equipment through the vibration amplitude). Since the difference between the image frames is used for judgment, the image frame acquisition frequency has certain requirements. In the present scheme, as a relatively optimal selection implementation manner, preferably, the track image is a video, and the frame number is 25, 30, 40, 50 or 60 frames per second; meanwhile, in the image frames segmented according to the preset time interval, the time difference between adjacent image frames is not more than 0.2-0.5 seconds.

[0073] In addition, in the present scheme, the operation execution log is acquired, and the operation intervention instruction is outputted according to the preset condition while the data of the operation execution log is judged and the warning information is generated and outputted according to the preset condition. The operation intervention instruction includes shutdown, braking or suspension instruction.

[0074] In order to facilitate intuitive monitoring, the warning information in the present scheme is also mapped into the visual output of the construction site digital model.

[0075] In combination with Figure 5 Based on the above, the present embodiment also provides a crane operation monitoring system based on digital twinning, which comprises:

[0076] The positioning unit is arranged on the two sides of the gantry of the gantry crane, the load trolley and the hook.

[0077] The model construction unit is used for establishing a reference coordinate system for the working site of the gantry crane according to the preset condition, and is also used for obtaining the contour scanning of the terrain and fixed facilities in the preset area range below and on the two sides of the gantry of the gantry crane to establish a construction site digital model, and is also used for forming a crane digital model by contour scanning the gantry crane according to the preset condition.

[0078] The digital twinning model construction unit is used for obtaining the real-time positioning data of the positioning unit on the gantry, the load trolley and the hook of the gantry crane, combining the real-time positioning data with the crane digital model and mapping it into the construction site digital model for visual display.

[0079] The construction monitoring unit is used for obtaining the working state information and operation record of the gantry crane according to the preset condition in response to the working start signal of the gantry crane, and generating an operation execution log.

[0080] The warning processing unit is used for obtaining the operation execution log, judging the data of the operation execution log according to the preset condition, generating and outputting warning information.

[0081] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0082] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk.

[0083] The above only describes some embodiments of the present application, and does not limit the protection scope of the present application. Any equivalent device or equivalent flow transformation, or direct or indirect application in other related technical fields, which is made by using the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A crane operation monitoring method based on digital twin, used for monitoring the operation of an outdoor gantry crane, wherein positioning units for locating the position are provided on both sides of the gantry frame, the trolley, and the hook of the gantry crane, characterized in that, The operation monitoring method includes: A reference coordinate system is established for the working site of the gantry crane according to the preset conditions. At the same time, the terrain and fixed facilities under and on both sides of the gantry crane are obtained by contour scanning to establish a digital model of the construction site. A digital model of the gantry crane is generated by contour scanning according to preset conditions. Acquire real-time positioning data of the gantry crane's gantry, trolley, and hook positioning unit, combine the real-time positioning data with the crane's digital model, and map it onto the construction site's digital model for visualization. In response to the start signal of the gantry crane, the system acquires the working status information and operation records of the gantry crane according to preset conditions, and generates an operation execution log. Obtain operation execution logs, perform data analysis based on preset conditions, generate and output early warning information; The gantry crane is equipped with a pair of image acquisition devices at both ends of the gantry frame. The image capturing ends of the pair of image acquisition devices face downward and are used to capture images of the tracks on both sides of the gantry frame. When the gantry crane stops, the tracks are located in the middle of the image captured by the image acquisition devices. The operation monitoring method also includes: In response to the start signal of the gantry crane, real-time image acquisition is performed on the tracks on both sides of the gantry, generating track images and importing them into the detection neural network. The detection neural network then detects and identifies foreign objects in the images and outputs the detection results. Once the test results are obtained, warning information will be output according to preset conditions. The operation monitoring method also includes: In response to the signal that the gantry crane’s gantry frame is moving on the track, the track image is acquired in real time and divided into image frames at preset time intervals; Acquire image frames, establish a baseline at the center of the image frame, and simultaneously locate and identify the track in the image frame to obtain the track contour. Using the two side profiles along the track length as a reference, a center reference line parallel to the track length direction is established. Then, the distance △L between the baseline and the center reference line is calculated in real time and output. The center reference line coincides with the baseline of the screen frame when the gantry crane stops, that is, the distance △L between the baseline and the center reference line is 0. The first comparison result is generated by obtaining △L and comparing it with a preset threshold. At the same time, the difference between △L on both sides of the gantry is calculated and compared with the preset threshold to generate a second comparison result. Obtain the first comparison result and / or the second comparison result, and output early warning information and / or operation intervention instructions based on preset conditions.

2. The crane operation monitoring method based on digital twin as described in claim 1, characterized in that, The operation execution log includes the time during which the gantry crane's gantry frame, trolley, or hook changes from a first state to a second state after the gantry crane's operation command is issued, and the speed remains constant or changes less than a preset amplitude within a preset duration.

3. The crane operation monitoring method based on digital twin as described in claim 2, characterized in that, The first state is when moving, and the second state is when stopped; When the first state is stopped, the second state is moving.

4. The crane operation monitoring method based on digital twin as described in claim 1, characterized in that, The training method for the detection neural network includes: Acquire images of the tracks connected to the gantry, label the images with foreign objects and mark their hazard levels to form a training dataset; Select a predetermined amount of data from the training dataset as training data and import it into the neural network for training; A predetermined amount of data is selected from the remaining data in the training dataset as validation data, and then imported into the trained neural network for validation. If the verification result does not meet the preset requirements, the training data is re-imported into the trained neural network and trained for a preset number of times until the verification result meets the preset requirements, thus obtaining the detection neural network.

5. The crane operation monitoring method based on digital twin as described in claim 4, characterized in that, The orbital image is a video, with a frame rate of 25, 30, 40, 50 or 60 frames per second; In the image frames divided according to a preset time interval, the time difference between adjacent image frames does not exceed 0.2 to 0.5 seconds.

6. The crane operation monitoring method based on digital twin as described in any one of claims 1 to 5, characterized in that, The system acquires operation execution logs, performs data judgments on them according to preset conditions, generates and outputs early warning information, and also outputs operation intervention instructions according to preset conditions. The operation intervention instructions include shutdown, braking, or suspension instructions. The warning information is also mapped to the digital model of the construction site for visualization output.

7. A crane operation monitoring system based on digital twins, wherein the crane operation monitoring method based on digital twins as described in any one of claims 1 to 5 is characterized in that, It includes: The positioning units are multiple and are arranged on both sides of the gantry frame, the trolley and the hook of the gantry crane; The model building unit is used to establish a reference coordinate system for the working site of the gantry crane according to preset conditions. At the same time, it performs contour scanning to obtain the terrain and fixed facilities within the preset area below and on both sides of the gantry crane to build a digital model of the construction site. It is also used to perform contour scanning of the gantry crane according to preset conditions to form a digital model of the crane. The digital twin model building unit is used to acquire real-time positioning data of the gantry crane's gantry, trolley, and hook positioning unit, and to combine the real-time positioning data with the crane's digital model and map it onto the construction site's digital model for visualization. The construction monitoring unit is used to respond to the start signal of the gantry crane, acquire the working status information and operation records of the gantry crane according to preset conditions, and generate an operation execution log. The early warning processing unit is used to acquire operation execution logs, perform data judgment on them according to preset conditions, generate early warning information, and output it.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the crane operation monitoring method based on digital twin as described in any one of claims 1 to 6.

Images