Tracking method, device and equipment of portal crane lifting appliance, storage medium and program product

Abstract

The invention provides a portal crane lifting appliance tracking method, device and equipment, a storage medium and a program product, and is applied to a system comprising a camera, and the portal crane lifting appliance tracking method comprises the steps that the camera collects video data of a portal crane lifting appliance and the surrounding environment of the portal crane lifting appliance; determining a corresponding template frame based on the video data; determining a similarity value between a video frame in the video data and the corresponding template frame; determining positioning data of the portal crane lifting appliance based on the template frame, the corresponding video frame and the similarity value; and based on the positioning data, the camera is controlled to move so as to track the portal crane lifting appliance. According to the method, the positioning data of the portal crane lifting appliance is calculated by determining the similarity value of the video frame and the template frame in the video data, so that the tracking control of the portal crane lifting appliance is realized, and the working efficiency of the portal crane lifting appliance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of spreader tracking, and in particular to a method, device, equipment, storage medium and program product for tracking a gantry crane spreader. Background Art

[0002] Gantry cranes are widely used in various industries such as construction, logistics, ports, and manufacturing, and are indispensable equipment in modern industrial and infrastructure construction. The main function of gantry cranes is to carry and stack heavy objects through slings. The accuracy and safety of their operation are directly related to the operating efficiency and the safety of personnel and equipment.

[0003] However, the traditional gantry crane operation method has the following major problems: low operating accuracy. Traditional gantry crane operation mainly relies on the operator's experience and visual control, which is prone to inaccurate positioning, especially when operating in large spans or complex environments; there are great safety hazards. During the crane operation process, due to the lack of real-time monitoring and intelligent feedback, safety accidents such as hoist collision and heavy object falling are prone to occur, posing serious safety threats to personnel and equipment. Summary of the invention

[0004] In view of this, the present invention provides a tracking method, device, equipment, storage medium and program product for a gantry crane hoist, which can solve the problem that operators position the gantry crane hoist by visual inspection, which may lead to inaccurate positioning and safety accidents such as hoist collision and heavy object falling.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for tracking a gantry crane hoist, which is applied to a system including a camera, and comprises:

[0007] The camera collects video data of the gantry crane spreader and its surroundings;

[0008] Determining a corresponding template frame based on the video data;

[0009] Determining a similarity value between a video frame in the video data and a corresponding template frame;

[0010] Determine the positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value;

[0011] Based on the positioning data, the camera is controlled to move so as to track the gantry crane hoist.

[0012] In one embodiment of the present invention, the camera includes a main hook camera and an auxiliary hook camera, the main hook camera is used to monitor the main hook of the gantry crane spreader, and the auxiliary hook camera is used to monitor the auxiliary hook of the gantry crane spreader;

[0013] The video data includes main hook video data from the main hook camera and auxiliary hook video data from the auxiliary hook camera.

[0014] In one embodiment of the present invention, determining a similarity value between a video frame in video data and a corresponding template frame includes:

[0015] Determine a plurality of sample points on a video frame;

[0016] Based on multiple sample points, video frames and corresponding template frames, the similarity value between each sample point and the template frame is calculated through a convolutional neural network.

[0017] In one embodiment of the present invention, based on multiple sample points, video frames and corresponding template frames, a similarity value between each sample point and the template frame is calculated by a convolutional neural network, including:

[0018] Determine a first feature map of the video frame and a corresponding second feature map of the template frame through a convolutional neural network;

[0019] The second feature map is used as a convolution kernel, and a convolution operation is performed on the corresponding first feature map to obtain a similarity map, wherein the similarity map is a similarity value between multiple sample points on the video frame and the template frame.

[0020] In one embodiment of the present invention, determining the positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value includes:

[0021] Determine the loss function based on the sample points and the similarity map;

[0022] The positioning data of the gantry crane spreader in the video frame is determined based on the loss function.

[0023] In one embodiment of the present invention, determining a loss function based on sample points and a similarity map includes:

[0024] Determine the label value corresponding to each sample point based on the position of the sample point and the gantry crane spreader, wherein the closer the sample point is to the gantry crane spreader, the label value of the sample point is a positive number, and the farther the sample point is from the gantry crane spreader, the label value of the sample point is a negative number;

[0025] The loss function is determined based on the label value and the similarity value, wherein the label value of the sample point is positive and the larger the similarity value is, the more correct the tracking is and the smaller the value of the loss function is.

[0026] In one embodiment of the present invention, determining the positioning data of the gantry crane spreader in the video frame based on the loss function includes:

[0027] Determine the center point based on the sample point corresponding to the maximum similarity value in the loss function;

[0028] Determine the position information of the center point in the video frame;

[0029] The positioning data of the gantry crane spreader is determined based on the position information.

[0030] In one embodiment of the present invention, controlling the movement of a camera based on positioning data includes:

[0031] Determine the movement data of the camera based on the positioning data, the movement data including the movement direction, the movement angle and the change of focal length;

[0032] The camera is controlled to move based on the movement data to ensure that the gantry crane spreader is in the center of the video frame.

[0033] The present invention also provides a tracking device for a gantry crane hoist, which is applied to a system including a camera, and comprises:

[0034] The acquisition module is used to collect video data of the gantry crane and its surrounding environment;

[0035] A processing module, used for determining a corresponding template frame based on the video data;

[0036] Determining a similarity value between a video frame in the video data and a corresponding template frame;

[0037] Determine the positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value;

[0038] The control module is used to control the movement of the camera based on the positioning data to track the gantry crane spreader.

[0039] In one embodiment of the present invention, the control module includes:

[0040] A processing unit, for calculating movement data of the camera based on the positioning data;

[0041] An automatic control unit for automatically adjusting the position of the camera based on the movement data;

[0042] Manual control interface, used for operators to manually control the camera.

[0043] In one embodiment of the present invention, it also includes:

[0044] The user interface module is used to provide a graphical operation interface to display the implementation status, position and posture data of the gantry crane spreader.

[0045] The present invention also provides an electronic device, comprising:

[0046] Memory for storing computer programs;

[0047] The processor is used to execute the computer program in the memory to implement the above tracking method.

[0048] The present invention also provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned tracking method.

[0049] The present invention also provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned tracking method.

[0050] The above technical solution of the present invention has at least the following beneficial effects:

[0051] The tracking method of the gantry crane spreader of the embodiment of the present invention determines the similarity value between the video frame in the video data and the template frame, thereby calculating the positioning data of the gantry crane spreader, and realizing the tracking control of the gantry crane spreader. Compared with the traditional solution in which the gantry crane spreader mainly relies on the operator to locate based on experience or visual observation, which is easy to cause safety accidents such as collision of the gantry crane spreader and falling of the objects on the spreader, the present invention collects the video data of the gantry crane through the camera and calculates the positioning data of the gantry crane spreader in real time, which can reduce the probability of safety accidents such as collision of the gantry crane spreader and is conducive to improving the efficiency of the gantry crane spreader operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A scene diagram of a method for tracking a gantry crane hoist according to an embodiment of the present invention;

[0053] Figure 2 A flow chart of a method for tracking a gantry crane hoist according to an embodiment of the present invention;

[0054] Figure 3 A flowchart of determining a similarity value between a video frame in video data and a corresponding template frame according to an embodiment of the present invention;

[0055] Figure 4 A flowchart of calculating the similarity value between each sample point and a template frame by a convolutional neural network according to an embodiment of the present invention;

[0056] Figure 5 A flow chart of determining positioning data of a gantry crane spreader according to an embodiment of the present invention;

[0057] Figure 6 A flowchart of determining a loss function based on sample points and a similarity map according to an embodiment of the present invention;

[0058] Figure 7 A flowchart of determining positioning data of a gantry crane spreader in a video frame according to an embodiment of the present invention;

[0059] Figure 8A flowchart of controlling camera movement based on positioning data according to an embodiment of the present invention;

[0060] Fig. 9 It is a structural schematic diagram of a tracking device for a gantry crane hoist according to an embodiment of the present invention;

[0061] Fig.10 FIG. 4 is a block diagram of a SoC according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0063] The following first describes in detail the application scenario of the gantry crane hoist tracking method according to the embodiment of the present invention with reference to the accompanying drawings.

[0064] refer to Figure 1 , Figure 1 FIG. 1 shows a scene diagram of a tracking method for a gantry crane hoist 100 according to an embodiment of the present invention. Figure 1 As shown, the sling 100 on the gantry crane can carry and stack heavy objects. The tracking method of the gantry crane sling 100 can be implemented by a tracking system. The tracking system includes a camera 110 and a control center 200. The control center 200 is connected to the camera 110 for communication. The camera 110 is arranged at the top of the gantry crane boom and is used to collect video stream information of the sling 100 and its surrounding environment in real time. The control center 200 can obtain the video stream information, and by calculating the adjustment position information of the camera 110 and controlling the camera 110 to move, the gantry crane sling 100 can be tracked, and the safety accidents such as collision of the gantry crane sling 100 can be reduced, thereby improving the practicality and reliability of the gantry crane operation.

[0065] A tracking method for a gantry crane hoist according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0066] refer to Figure 2 , Figure 2 FIG. 1 is a flow chart showing a method for tracking a gantry crane hoist according to an embodiment of the present invention. Figure 2 As shown, the gantry crane hoist tracking method includes steps S201-S205, and the method is applied to a system including a camera. The camera can be set on the gantry crane hoist, preferably, the camera can be set on the front end or the end of the head of the arm of the gantry crane hoist.

[0067] In S201 , the camera collects video data of the gantry crane hoist and its surroundings.

[0068] For example, a gantry crane can carry and stack objects through a spreader. The camera collects real-time video stream data (i.e., video data) of the gantry crane spreader and the objects to be carried and stacked by the spreader.

[0069] In S202, a corresponding template frame is determined based on the video data. The template frame may be a video frame of a gantry crane hoist in a specific scenario, for example, a video frame of a gantry crane hoist lifting a load when the gantry crane performs a handling operation. When there are multiple cameras, the video data acquired by each camera will correspond to its own template frame, for example, the video data acquired by the main hook camera corresponds to the main hook template frame, and the video data acquired by the auxiliary hook camera corresponds to the auxiliary hook template frame.

[0070] In S203, a similarity value between a video frame in the video data and a corresponding template frame is determined.

[0071] For example, a video frame in the video data, i.e., the real-time video stream data, is selected to determine the similarity value between the video frame and the template frame. After a period of time (e.g., 1 second or 2 seconds), the next video frame is selected, and the similarity value between the next video frame and the template frame is determined, and step S203 is repeated to achieve real-time calculation of the similarity value between the video data and the template frame.

[0072] In S204, the positioning data of the gantry crane spreader is determined based on the video frame, the corresponding template frame and the similarity value.

[0073] For example, the position of the gantry crane spreader is calculated through the similarity value between the real-time video stream data (ie, the video frame in the video data) and the template frame to obtain the positioning data of the gantry crane spreader.

[0074] In S205, based on the positioning data, the camera is controlled to move so as to track the gantry crane hoist.

[0075] For example, the control center of the system can control the movement or rotation of the camera based on the positioning data so that the gantry crane hoist is located at the center of the camera's real-time video stream data (i.e., video data), which helps to improve the efficiency of the gantry crane hoist tracking and facilitates the gantry crane hoist to carry or stack the objects.

[0076] In an embodiment of the present invention, by determining the similarity value between the video frame in the video data and the template frame, the positioning data of the gantry crane spreader is calculated to achieve tracking control of the gantry crane spreader. Compared with the prior art where the gantry crane spreader mainly relies on the operator's experience or visual inspection for positioning, which is prone to inaccurate positioning leading to safety accidents such as collision of the gantry crane spreader and falling of the objects on the spreader, the present invention collects the video data of the gantry crane in real time through a camera, and calculates the positioning data of the gantry crane spreader in real time, which can reduce the probability of safety accidents such as collision of the gantry crane spreader and is conducive to improving the efficiency of the gantry crane spreader operation.

[0077] In one embodiment of the present invention, Figure 1 As shown, the cameras in the system include a main hook camera and an auxiliary hook camera. The main hook camera is set on the main hook of the gantry crane hoist, and the auxiliary hook camera is set on the auxiliary hook of the gantry crane hoist. The main hook camera is used to monitor the working condition of the main hook of the gantry crane hoist. The auxiliary hook camera is used to monitor the working condition of the auxiliary hook of the gantry crane hoist.

[0078] Correspondingly, in S201, the video data includes the main hook video data from the main hook camera and the auxiliary hook video data from the auxiliary hook camera. By setting the main hook camera and the auxiliary hook camera, the working condition of the gantry crane spreader can be more accurately monitored, further improving the efficiency of gantry crane spreader tracking.

[0079] Next, combine Figure 2 , and refer to Figure 3 , to illustrate a specific embodiment of the present invention. Figure 3 FIG. 1 is a flow chart showing a method of determining a similarity value between a video frame in video data and a corresponding template frame according to an embodiment of the present invention. Figure 2 and Figure 3 As shown, in S203, determining the similarity value between the video frame in the video data and the corresponding template frame may include the following steps S301-S302.

[0080] Step S301: determine a plurality of sample points on a video frame.

[0081] Specifically, a region to be trained is determined on a video frame. For example, the region to be trained is a specific circular region centered on the gantry crane hoist, and a plurality of sample points are selected in the region to be trained.

[0082] Step S302: Based on multiple sample points, video frames and corresponding template frames, a similarity value between each sample point and the template frame is calculated by a convolutional neural network.

[0083] Among them, Convolutional Neural Networks (CNN) is a type of feedforward neural network with deep structure and convolutional calculation, and is one of the representative algorithms of deep learning. Convolutional neural networks can be constructed by imitating the visual perception mechanism of organisms, and can perform supervised learning and unsupervised learning. The convolution kernel parameter sharing in the hidden layer and the sparsity of the inter-layer connection enable convolutional neural networks to learn grid features such as pixels and audio with a small amount of calculation, with stable results and no additional feature engineering requirements for data.

[0084] For example, the positions of multiple sample points on the corresponding template frame are determined, and the similarity value between each sample point on the video frame and the template frame is calculated through the convolutional neural network model, which facilitates the judgment of the posture of the gantry crane hoist.

[0085] Combine the following Figure 3 , and refer to Figure 4 , to illustrate a specific embodiment of the present invention. Figure 4 A flow chart of calculating the similarity value between each sample point and a template frame through a convolutional neural network according to an embodiment of the present invention is shown.

[0086] like Figure 3 and Figure 4 As shown, in S302, based on multiple sample points, video frames and corresponding template frames, a similarity value between each sample point and the template frame is calculated by a convolutional neural network, which may include the following steps S401-S402.

[0087] Step S401: Determine a first feature map of a video frame and a second feature map of a corresponding template frame through a convolutional neural network.

[0088] The first feature map and the second feature map are information extracted from an input image (e.g., a video frame or a template frame) through a convolutional layer, and the information represents different features, such as edges, textures, shapes, etc.

[0089] Step S402: Using the second feature map as a convolution kernel, perform a convolution operation on the corresponding first feature map to obtain a similarity map.

[0090] The similarity map is the similarity value between multiple sample points on the video frame and the template frame. The convolution kernel is the internal component of the convolution layer, which is used to further extract features from the input image. Each element in the convolution kernel corresponds to a weight coefficient and a bias, similar to a neuron in a feedforward neural network. The second feature map extracted from the template frame is used as the convolution kernel, and a convolution operation is performed on the first feature map of the video frame to obtain the similarity values ​​between multiple sample points in the area to be trained and the corresponding template frame.

[0091] For example, the main hook video frame x1 in the main hook video data and the auxiliary hook video frame x2 in the auxiliary hook video data are calculated through a full convolutional network to obtain two first feature maps. The corresponding main hook template frame z1 and auxiliary hook template frame z2 are calculated through a full convolutional network to obtain two second feature maps. The second feature maps of z1 and z2 are used as convolution kernels to perform convolution operations on the first feature maps of x1 and x2 respectively to obtain two similarity maps (score maps). The two similarity maps represent the similarity values ​​of each sample point in the to-be-trained area in the main hook video frame x1 and the auxiliary hook video frame x2 with the template frames z1 and z2, respectively, so as to further improve the tracking efficiency of the gantry crane hoist.

[0092] Next, combine Figure 2 , and refer to Figure 5 , to illustrate a specific embodiment of the present invention. Figure 5 FIG. 1 is a flow chart showing a method for determining the positioning data of a gantry crane hoist according to an embodiment of the present invention. Figure 2 and Figure 5 As shown, in S204, determining the positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value may include the following steps S501-S502.

[0093] Step S501: Determine a loss function based on sample points and a similarity map.

[0094] For example, a loss function is determined based on a plurality of sample points and similarity values ​​corresponding to the sample points. The specific process of determining the loss function is shown in the following steps S601-S602.

[0095] Step S502: determining the positioning data of the gantry crane spreader in the video frame based on the loss function.

[0096] According to the value of the loss function, the positioning data of the gantry crane spreader in the video frame is determined. For example, the smaller the value of the loss function, the more accurate the positioning data. By constructing the loss function and selecting the value of the loss function according to the position and similarity value of the sample point, it is helpful to improve the accuracy of the positioning data of the gantry crane spreader.

[0097] Next, combine Figure 5 , and refer to Figure 6 , to illustrate an embodiment of the present invention. Figure 6 The flowchart of determining the loss function based on the sample points and the similarity map according to one embodiment of the present invention is shown. In S501, determining the loss function based on the sample points and the similarity map may include the following steps S601-S602.

[0098] Step S601: Determine the label value corresponding to each sample point based on the position of the sample point and the gantry crane hoist.

[0099] The closer the sample point is to the gantry crane hoist, the label value of the sample point is a positive number, and the farther the sample point is from the gantry crane hoist, the label value of the sample point is a negative number.

[0100] Step S602: Determine a loss function based on the tag value and the similarity value.

[0101] Among them, the label value of the sample point is positive and the larger the similarity value is, the more correct the tracking is, and the smaller the value of the loss function is.

[0102] For example, the loss function for a single sample point is constructed as follows:

[0103] l(y,v)=log(1+e -yv )

[0104] Among them, v is the similarity value of this sample point in the similarity map score map. The larger the value, the more similar it is. y∈{+1,-1} is the label of this point, which means that this sample point is near the target hanger, then the label value is 1, and the sample point is in other locations, then the label value is -1. Therefore, when v is large and y is 1, it is considered that the tracking is correct and the value of the loss function is small, otherwise the value of the loss function is large.

[0105] The overall loss function composed of multiple sample points (corresponding to the loss function in step S602) is:

[0106]

[0107] Where u is a certain position in the score map (i.e., a sample point), and the overall loss function is evaluated by taking the average value. By distinguishing between positive and negative samples at the sample points, it is helpful to improve the practicality and accuracy of the loss function construction.

[0108] Next, combine Figure 5 , and refer to Figure 7 , to illustrate an embodiment of the present invention. Figure 7 FIG. 1 is a flow chart showing a method of determining the positioning data of a gantry crane hoist in a video frame according to an embodiment of the present invention. Figure 5 and Figure 7 As shown, in S502, determining the positioning data of the gantry crane hoist in the video frame based on the loss function may include the following steps S701-S703.

[0109] Step S701: Determine the center point based on the sample point corresponding to the maximum similarity value in the loss function.

[0110] For example, the sample point with the largest similarity value and label value 1 in the loss function is taken as the center point.

[0111] Step S702: Determine the position information of the center point in the video frame.

[0112] For example, the position information of the center point in the main hook video frame and the auxiliary hook video frame is calculated.

[0113] Step S703: Determine the positioning data of the gantry crane hoist based on the position information.

[0114] According to the positions of the center point in the main hook video frame and the auxiliary hook video frame, the positioning data of the gantry crane spreader is determined, which can further improve the tracking accuracy of the gantry crane spreader.

[0115] Next, combine Figure 2 , and refer to Figure 8 , to illustrate a specific embodiment of the present invention. Figure 8 FIG. 1 is a flow chart showing a control of camera movement based on positioning data according to an embodiment of the present invention. Figure 2 and Figure 8 As shown, in S205, controlling the movement of the camera based on the positioning data may include the following steps S801-S802.

[0116] Step S801: Determine the movement data of the camera based on the positioning data.

[0117] For example, the movement data may include movement direction, movement angle, and change focus.

[0118] Step S802: Control the camera to move based on the movement data to ensure that the gantry crane hoist is at the center of the video frame.

[0119] For example, the control center can control the movement of the camera according to the mobile data so that the camera automatically zooms and tracks the spreader, so that the spreader is always in the center position of the video frame of the real-time video stream data (i.e., video data), which is beneficial to improve the accuracy and reliability of the tracking of the gantry crane spreader.

[0120] refer to Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram showing the structure of a tracking device for a gantry crane hoist according to an embodiment of the present invention. Fig. 9 As shown, the present invention also provides a tracking device for a gantry crane hoist, which is applied to a system including a camera, including a collection module, a processing module and a control module.

[0121] The acquisition module is used to collect video data of the gantry crane hoist and its surrounding environment.

[0122] The processing module is used to determine the corresponding template frame based on the video data, determine the similarity value between the video frame in the video data and the corresponding template frame, and determine the positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value. In other words, the processing module can use computer vision technology to process the video data to identify the position, posture and relative position of the spreader to the target object.

[0123] Specifically, the processing module may further include an input terminal, a training terminal and an output terminal.

[0124] The input end transmits video stream information via wired or wireless means.

[0125] The training end uses convolutional neural networks and twin network structure methods to process the sling information in the video frames and classify and label them.

[0126] The output end outputs the position, posture and other information of the spreader.

[0127] The control module is used to control the movement of the camera based on the positioning data to track the gantry crane spreader. In other words, the control module can receive the position, posture and other information provided by the processing module, so that the camera can automatically zoom and track the spreader target, keeping the spreader target always in the center of the video.

[0128] In addition, in some other embodiments of the present invention, the tracking device of the gantry crane hoist may also include a data transmission module for transmitting the collected video stream data to the central processing unit through a wired or wireless network. Specifically, the data transmission module may also include wireless transmission and wired transmission. Wireless transmission uses 5G network transmission, which has the advantages of ultra-low latency and ultra-high bandwidth. Wired transmission uses optical fiber communication, which has the characteristics of long transmission distance, high bandwidth, and anti-electromagnetic interference.

[0129] In one embodiment of the present invention, the control module includes a processing unit, an automatic control unit and a manual control interface.

[0130] The processing unit is used to calculate the movement data of the camera based on the positioning data. That is, the processing unit can receive the information such as the position and posture of the spreader output by the output end of the processing module, and calculate the direction and focal length of the camera based on the information.

[0131] The automatic control unit is used to automatically adjust the position of the camera based on the movement data. That is, the automatic control unit can automatically adjust the direction and focal length of the camera according to the direction and focal length of the camera calculated by the processing unit to ensure that the hoist is always in the center of the video.

[0132] The manual control interface is used by operators to manually control the camera. For example, when the automatic control fails, the operator can manually intervene and control.

[0133] In one embodiment of the present invention, the tracking device for a gantry crane spreader further comprises a user interface module.

[0134] The user interface module is used to provide a graphical operation interface to display the implementation status, position and posture data of the gantry crane spreader, so that the data related to the gantry crane spreader can be displayed to the operator more intuitively, which is conducive to improving the operator's work efficiency.

[0135] The present invention also provides an electronic device, comprising a camera, a memory and a processor.

[0136] The memory is used to store computer programs.

[0137] The processor is used to execute the computer program in the memory to achieve the above Figure 2-Figure 8 The tracking method shown.

[0138] The present invention also provides a gantry crane hoist tracking system, comprising a camera and a control center. The camera is arranged on the top of the gantry crane boom, and is used to collect video stream information of the hoist and its surrounding environment in real time. There can be multiple cameras. The control center can obtain the video stream information, and by calculating the adjustment position information of the camera, control the camera to move, and realize the tracking of the gantry crane hoist.

[0139] The present invention also provides a computer storage medium, the computer storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the above Figure 2-Figure 8 The tracking method shown.

[0140] The present invention also provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the above Figure 2-Figure 8 The tracking method shown.

[0141] In summary, the tracking method of the gantry crane spreader of the embodiment of the present invention determines the similarity value between the video frame in the video data and the template frame, thereby calculating the positioning data of the gantry crane spreader, and realizing the tracking control of the gantry crane spreader. The present invention collects the video data of the gantry crane through the camera, and calculates the positioning data of the gantry crane spreader in real time, which can reduce the probability of safety accidents such as collision of the gantry crane spreader, and is conducive to improving the efficiency of the gantry crane spreader operation.

[0142] Reference now Fig.10 , which is a block diagram of a SoC (System on Chip) 1300 according to an embodiment of the present application. Fig.10 In the FIG, similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Fig.10In the embodiment, SoC 1300 includes: an interconnect unit 1350, which is coupled to an application processor 1310; a system agent unit 1380; a bus controller unit 1390; an integrated memory controller unit 1340; a group or one or more coprocessors 1320, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the coprocessor 1320 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high throughput MIC processor, or an embedded processor.

[0143] The static random access memory (SRAM) unit 1330 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions. The instructions may include: when executed by at least one unit in the processor, causing the Soc 1300 to execute the instructions according to the above embodiments. Figure 2-Figure 8 The methods explained will not be repeated here.

[0144] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0145] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0146] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0147] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0148] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0149] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0150] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0151] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0152] The above are preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for tracking a gantry crane spreader, applied to a system including a camera, characterized in that: include: The camera collects video data of the gantry crane hoist and its surrounding environment; Determining a corresponding template frame based on the video data; Determining a similarity value between a video frame in the video data and a corresponding template frame; Determining positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value; Based on the positioning data, the camera is controlled to move so as to track the gantry crane hoist.

2. The method for tracking a gantry crane spreader according to claim 1, characterized in that: The camera includes a main hook camera and an auxiliary hook camera, wherein the main hook camera is used to monitor the main hook of the gantry crane spreader, and the auxiliary hook camera is used to monitor the auxiliary hook of the gantry crane spreader; The video data includes main hook video data from the main hook camera and auxiliary hook video data from the auxiliary hook camera.

3. The tracking method of the gantry crane spreader according to claim 2, characterized in that: The determining of the similarity value between the video frame in the video data and the corresponding template frame includes: Determining a plurality of sample points on the video frame; Based on the plurality of sample points, the video frames and the corresponding template frames, the similarity value between each sample point and the template frame is calculated by a convolutional neural network.

4. The tracking method of the gantry crane spreader according to claim 3, characterized in that: The calculating the similarity value between each sample point and the template frame by a convolutional neural network based on the plurality of sample points, the video frames and the corresponding template frames includes: Determine a first feature map of the video frame and a corresponding second feature map of the template frame by using the convolutional neural network; The second feature map is used as a convolution kernel, and a convolution operation is performed on the corresponding first feature map to obtain a similarity map, wherein the similarity map is the similarity values ​​between the multiple sample points on the video frame and the template frame.

5. The tracking method of the gantry crane spreader according to claim 4, characterized in that: The determining the positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value includes: Determining a loss function based on the sample points and the similarity map; The positioning data of the gantry crane hoist in the video frame is determined based on the loss function.

6. The tracking method of the gantry crane spreader according to claim 5, characterized in that: The determining of the loss function based on the sample points and the similarity map includes: Determine a label value corresponding to each sample point based on the position of the sample point and the gantry crane spreader, wherein the closer the sample point is to the gantry crane spreader, the label value of the sample point is a positive number, and the farther the sample point is from the gantry crane spreader, the label value of the sample point is a negative number; The loss function is determined based on the label value and the similarity value, wherein the label value of the sample point is a positive number and the larger the similarity value is, the more correct the tracking is, and the smaller the value of the loss function is.

7. The tracking method of the gantry crane spreader according to claim 6, characterized in that: The determining the positioning data of the gantry crane hoist in the video frame based on the loss function includes: Determine a center point based on the sample point corresponding to the maximum similarity value in the loss function; Determine the position information of the center point in the video frame; The positioning data of the gantry crane spreader is determined based on the position information.

8. The method for tracking a gantry crane spreader according to claim 1, characterized in that: The step of controlling the camera movement based on the positioning data includes: Determine movement data of the camera based on the positioning data, the movement data including movement direction, movement angle and change focal length; The camera is controlled to move based on the movement data to ensure that the gantry crane hoist is located at the center of the video frame.

9. A tracking device for a gantry crane hoist, applied to a system including a camera, characterized in that: include: The acquisition module is used to collect video data of the gantry crane and its surrounding environment; A processing module, configured to determine a corresponding template frame based on the video data; Determining a similarity value between a video frame in the video data and a corresponding template frame; Determining positioning data of the gantry crane spreader based on the video frame, the corresponding template frame and the similarity value; A control module is used to control the movement of the camera based on the positioning data to track the gantry crane hoist.

10. The tracking device for a gantry crane spreader according to claim 9, characterized in that: The control module comprises: A processing unit, configured to calculate movement data of the camera based on the positioning data; An automatic control unit, configured to automatically adjust the position of the camera based on the movement data; The manual control interface is used for operators to manually control the camera.

11. The tracking device for a gantry crane spreader according to claim 9, characterized in that: Also includes: The user interface module is used to provide a graphical operation interface to display the implementation status, position and posture data of the gantry crane hoist.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program in the memory to implement the tracking method according to any one of claims 1 to 8.

13. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the tracking method according to any one of claims 1 to 8.

14. A computer program product, characterized in that When the computer program product runs on an electronic device, the electronic device executes the tracking method described in any one of claims 1 to 8.

Images