Container Truck Anti-Lifting Method, Device, Equipment and Storage Medium

Abstract

The present invention provides a method, device, equipment and storage medium for preventing a container truck from being lifted. The method includes: acquiring multiple frames of target images collected by a camera, where the target images include a container area and a container truck body area; using a preset deep learning model to perform target area detection on each frame of the target images to obtain the container area and the container truck body area in each frame of the target images; extracting a first oriented FAST and rotated BRIEF (ORB) feature point group in the container area of each frame of the target images and a second oriented FAST and rotated BRIEF (ORB) feature point group in the container truck body area; determining the motion states of the container and the container truck body according to the first oriented FAST and rotated BRIEF (ORB) feature point group and the second oriented FAST and rotated BRIEF (ORB) feature point group corresponding to each frame of the target images; and determining whether the container truck is lifted according to the motion states of the container and the container truck body. It can ensure the stability and accuracy of tracking, and effectively improve the accuracy of the discrimination result of whether the container truck is lifted.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and particularly to a method, device, equipment and storage medium for preventing a container truck from being lifted. Background Art

[0002] When a quay crane or gantry crane unloads a container from a container truck during port operations, it is necessary to unlock the lock connecting the container and the container truck body. However, in actual operations, the lock may not be unlocked or may not be fully unlocked. In this case, the quay crane or gantry crane will lift the container and the container truck together, which will not only cause serious damage to the port container equipment, but may also lead to serious accidents involving personnel and vehicles. This is the accident of lifting the container truck. Therefore, with the increasing demand for automation in port operations, the automated technology for preventing the container truck from being lifted is particularly important.

[0003] Currently, there are generally three types of methods for preventing a container truck from being lifted. One is to use lidar ranging to determine whether the container truck is lifted. The second is to use sensors to capture the opening and closing states of each lock of the container truck, and then determine whether the container truck is lifted. The third is to use a camera to capture an image and then use a vision algorithm to analyze whether the container truck is lifted.

[0004] However, the first and second methods need to determine different installation strategies for lidar and sensors according to different ports, cranes and container trucks, and the versatility is poor. In the third method, the currently used vision algorithm only tracks the area of the components related to the container truck, and then determines whether the container truck is lifted. As a result, the accuracy of the determination result of whether the container truck is lifted is relatively low. Summary of the Invention

[0005] The present invention provides a method, device, equipment and storage medium for preventing a container truck from being lifted, so as to solve the technical problems of poor versatility and low accuracy of the determination result in the prior art method for preventing a container truck from being lifted.

[0006] In a first aspect, the present invention provides a method for preventing a container truck from being lifted, including:

[0007] Obtain multiple frames of target images collected by a camera, where the target images include a container area and a container truck body area;

[0008] Use a preset deep learning model to perform target area detection on each frame of the target image to obtain the container area and the container truck body area in each frame of the target image;

[0009] Extract a first group of oriented FAST and rotated BRIEF (ORB) feature points in the container area of each frame of the target image and a second group of oriented FAST and rotated BRIEF (ORB) feature points in the container truck body area;

[0010] Determine the motion states of the container and the truck body according to the first-oriented fast-rotating orb feature point groups and the second-oriented fast-rotating orb feature point groups corresponding to each frame of target image;

[0011] Determine whether the truck is lifted according to the motion states of the container and the truck body.

[0012] In a second aspect, the present invention provides a truck anti-lifting device, including:

[0013] An image acquisition module, configured to acquire multiple frames of target images collected by a camera, where the target images include a container area and a truck body area;

[0014] An area detection module, configured to perform target area detection on each frame of target image by using a preset deep learning model to obtain the container area and the truck body area in each frame of target image;

[0015] A feature point group extraction module, configured to extract a first-oriented fast-rotating orb feature point group in the container area of each frame of target image and a second-oriented fast-rotating orb feature point group in the truck body area;

[0016] A motion state determination module, configured to determine the motion states of the container and the truck body according to the first-oriented fast-rotating orb feature point groups and the second-oriented fast-rotating orb feature point groups corresponding to each frame of target image;

[0017] A lifting judgment module, configured to determine whether the truck is lifted according to the motion states of the container and the truck body.

[0018] In a third aspect, the present invention provides an electronic device, including: at least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0022] The container truck anti-lifting method, device, equipment and storage medium provided by the present invention obtain multiple frames of target images collected by a camera, where the target images include a container area and a container truck body area; use a preset deep learning model to perform target area detection on each frame of the target image to obtain the container area and the container truck body area in each frame of the target image; extract the first oriented fast rotated orb feature point group in the container area of each frame of the target image and the second oriented fast rotated orb feature point group in the container truck body area; determine the motion states of the container and the container truck body according to the first oriented fast rotated orb feature point group and the second oriented fast rotated orb feature point group corresponding to each frame of the target image; determine whether the container truck is lifted according to the motion states of the container and the container truck body. Since both the container area and the container truck body area are detected, and whether the container truck is lifted is determined by determining the motion states of the container and the container truck body areas, the error caused by only detecting the area of the components related to the container truck is reduced. And after both the container truck body area and the container area are detected, by tracking the oriented fast rotated orb feature point group in the area, the error of tracking the entire area can be effectively reduced, ensuring the stability and accuracy of the tracking, and thus effectively improving the accuracy of the discrimination result of whether the container truck is lifted. Description of the Drawings

[0023] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0024] Figure 1 It is a network architecture diagram of the container truck anti-lifting method provided by an embodiment of the present invention;

[0025] Figure 2 It is a flowchart of the container truck anti-lifting method provided by an embodiment of the present invention;

[0026] Figure 3 It is a flowchart of the container truck anti-lifting method provided by another embodiment of the present invention;

[0027] Figure 4 It is a schematic structural diagram of the improved CenterNet model in the container truck anti-lifting method provided by another embodiment of the present invention;

[0028] Figure 5 It is a flowchart of step 203 in the container truck anti-lifting method provided by another embodiment of the present invention;

[0029] Figure 6 It is a flowchart of step 206 in the container truck anti-lifting method provided by another embodiment of the present invention;

[0030] Figure 7Flowchart of step 207 in the container truck anti-lifting method provided by another embodiment of the present invention;

[0031] Figure 8 Flowchart of step 208 in the container truck anti-lifting method provided by another embodiment of the present invention;

[0032] Figure 9 Flowchart of step 209 in the container truck anti-lifting method provided by another embodiment of the present invention;

[0033] Figure 10 Signaling flowchart of the container truck anti-lifting method provided by another embodiment of the present invention;

[0034] Figure 11 Structural schematic diagram of the container truck anti-lifting device provided by an embodiment of the present invention;

[0035] Figure 12 Block diagram of an electronic device provided by an embodiment of the present invention.

[0036] Through the above-mentioned drawings, specific embodiments of the present disclosure have been shown, and more detailed descriptions will be provided hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present disclosure in any way, but to illustrate the concept of the present disclosure to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0037] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0038] First, the prior art related to the embodiments of the present invention will be described and analyzed in detail.

[0039] Currently, there are generally three types of methods for preventing container trucks from being lifted. One is to use lidar ranging to determine whether the container truck is lifted. The second is to use sensors to capture the opening and closing states of each lock of the container truck, and then determine whether the container truck is lifted. The third is to use a camera to capture an image and then use a vision algorithm to analyze whether the container truck is lifted.

[0040] In the first method, the lidar is installed on the crane, specifically at a height between the container and the truck body. After the crane starts lifting the container, the lidar performs periodic scanning and ranging. If the lidar does not detect the truck body after a certain scan, it is determined that the truck body and the container are completely separated. If the truck body and the container are detected and the distance between the container and the truck body remains unchanged for a long time after ranging, it indicates that the truck body and the container are lifted together.

[0041] In the second method, sensors are installed on the locks, and the opening and closing states of each lock can be collected. If all the locks are in the open state, it is determined that the container and the truck body are completely separated.

[0042] However, in the first method, the installation height of the lidar needs to be determined according to different ports, cranes and trucks. In the second method, corresponding sensors need to be installed according to the positions and numbers of the locks on the truck body. Therefore, the generality of these two methods is relatively poor.

[0043] In the third method, there are mainly two vision algorithms. The first one is to track the truck body area after determining the truck body area, and then determine the motion state of the truck to judge whether the truck is lifted. The second one is to determine the area of the truck wheels, track the area of the truck wheels, and then determine the motion state of the truck to judge whether the truck is lifted. However, when tracking the truck body area, the Kernel Correlation Filter (KCF) algorithm is generally used. The overall similarity of each part of the truck frame area itself is relatively high, which is exactly the scenario where the method with a rectangular frame as the tracking area in the KCF algorithm is difficult to operate stably, resulting in inaccurate tracking of the truck body area, low accuracy in determining the motion state of the truck body area, and thus low accuracy in judging whether the truck is lifted. When tracking the truck wheels, the method of detecting ellipses using the Hough transform algorithm is adopted, and the stability and accuracy of the Hough transform algorithm are both relatively low, so the accuracy of the discrimination result of whether the truck is lifted is low.

[0044] In the face of the technical problems existing in the first and second methods, in order to improve generality, it is still necessary to analyze whether the container truck is lifted by means of a vision algorithm. In order to solve the problem of relatively low accuracy of the discrimination result of whether the container truck is lifted in the third method. The inventor found through research that whether the container truck is lifted is not only related to the motion state of the container truck body area, but also related to the motion state of the container area. And after detecting both the container truck body area and the container area, by tracking the oriented FAST and rotated (abbreviation: orb) feature point groups in the area, the tracking error can be effectively reduced, thereby ensuring the accuracy of the discrimination result for determining whether the container truck is lifted.

[0045] Therefore, based on the above creative discovery, the inventor proposed the technical solution of the embodiment of the present invention. The following introduces the network structure and application scenario of the container truck anti-lifting method provided by the embodiment of the present invention.

[0046] Figure 1 It is a network architecture diagram of the container truck anti-lifting method provided by an embodiment of the present invention. As Figure 1 shown, in the network architecture of the container truck anti-lifting method provided by the embodiment of the present invention, it includes: a control system 11 of the crane 1, a camera 12 installed on the crane 1, an electronic device 2, a container truck 3, and a container 4 placed on the body of the container truck 3. Among them, the control system is communicatively connected to the camera 12 and the electronic device 2 respectively, and the camera 12 is communicatively connected to the electronic device 2. In a specific application scenario, after the control system 11 determines that the locking signal is sent by the lock connecting the crane 1 and the container, it determines that the crane is fully locked. The container 4 is lifted by the operation of the driver. At the same time, a shooting start instruction is sent to the camera 12, and a container truck anti-lifting request is sent to the electronic device 2. The camera 12 acquires multiple frames of target images including the container and the container truck, and sends the multiple frames of target images to the electronic device 2. The electronic device 2 uses the container truck anti-lifting method provided by the embodiment of the present invention to determine whether the container truck is lifted. If it is determined that the container truck is lifted, an alarm message is sent to the control system 11 of the crane, so that the driver of the crane can pause the lifting according to the alarm message. If it is determined that the container truck is not lifted, the lifting stroke height of the crane lifting the container is obtained. When the stroke height reaches the safe operation height, the electronic device 2 stops using the container truck anti-lifting method provided by the embodiment of the present invention to determine whether the container truck is lifted. The crane switches the working mode from the manual operation mode to the automatic operation mode and continues to lift the container until the target position is reached.

[0047] The following uses specific embodiments to elaborate in detail on the technical solution of the present invention and how the technical solution of the present invention solves the above technical problems. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following will describe the embodiments of the present invention in conjunction with the accompanying drawings.

[0048] Embodiment 1

[0049] Figure 2 The figure is a flowchart of a method for preventing a container truck from being lifted provided by an embodiment of the present invention. As Figure 2 shown, the execution subject of the method for preventing a container truck from being lifted provided in this embodiment is a container truck anti-lifting device. The container truck anti-lifting device is located in an electronic device. Then the method for preventing a container truck from being lifted provided in this embodiment includes the following steps:

[0050] Step 101: Obtain multiple frames of target images collected by a camera. The target images include a container area and a container truck body area.

[0051] Among them, the target image is an image for which region detection and feature point extraction are required.

[0052] In this embodiment, the camera can be installed on the crane. When receiving a shooting start instruction sent by the crane control system, the camera can periodically shoot at least one side of the container and the container truck body to obtain multiple frames of target images. Among them, each frame of target image includes a container area and a container truck body area.

[0053] Step 102: Use a preset deep learning model to perform target area detection on each frame of target image to obtain the container area and the container truck body area in each frame of target image.

[0054] Among them, the preset deep learning model can be a convolutional neural network model, a CenterNet model, or an improved CenterNet model, etc. This embodiment does not limit this.

[0055] In this embodiment, each frame of target image is input into a preset deep learning model. The preset deep learning model performs feature extraction and detection on the container area and the container truck body area, and outputs the container area and the container truck body area.

[0056] Among them, the detected container area and container truck body area can both be represented in the form of being framed by a rectangular box.

[0057] Step 103: Extract a first oriented FAST and rotated BRIEF (ORB) feature point group in the container area of each frame of target image and a second oriented FAST and rotated BRIEF (ORB) feature point group in the container truck body area.

[0058] In this embodiment, the Oriented FAST and Rotated BRIEF (ORB) algorithm is used to extract the feature point groups in the container area of each frame of the target image. This feature point group is the first oriented FAST and rotated ORB feature point group. The ORB algorithm is also used to extract the feature point groups in the container truck body area of each frame of the target image. This feature point group is the second oriented FAST and rotated ORB feature point group.

[0059] Step 104: Determine the motion states of the container and the container truck body according to the first oriented FAST and rotated ORB feature point group and the second oriented FAST and rotated ORB feature point group corresponding to each frame of the target image.

[0060] In this embodiment, the first oriented FAST and rotated ORB feature point group corresponding to each frame of the target image can be tracked to determine the motion state of the first oriented FAST and rotated ORB feature point group, and then the motion state of the container can be determined according to the motion state of the first oriented FAST and rotated ORB feature point group. Similarly, the second oriented FAST and rotated ORB feature point group corresponding to each frame of the target image can be tracked to determine the motion state of the second oriented FAST and rotated ORB feature point group, and then the motion state of the container truck body can be determined according to the motion state of the second oriented FAST and rotated ORB feature point group.

[0061] Among them, the motion state of the container can be any one of a stationary state, an upward movement state, a downward movement state, and a left-right movement state. Similarly, the motion state of the container truck body can be any one of a stationary state, an upward movement state, a downward movement state, and a left-right movement state.

[0062] Step 105: Determine whether the container truck is lifted according to the motion states of the container and the container truck body.

[0063] Specifically, in this embodiment, if it is determined that the motion states of both the container and the container truck body are in the upward movement state, it means that the lock on the side closer to the camera is not unlocked, and it is determined that the container truck is lifted. If it is determined that the motion state of the container is in the upward movement state and the motion state of the container truck body is in the downward movement state, it means that the lock on the side farther from the camera is not unlocked, and it is determined that the container truck is lifted.

[0064] The container truck anti-lifting method provided in this embodiment obtains multiple frames of target images collected by a camera, where the target images include a container area and a container truck body area; uses a preset deep learning model to perform target area detection on each frame of target image to obtain the container area and the container truck body area in each frame of target image; extracts the first oriented fast rotation orb feature point group in the container area of each frame of target image and the second oriented fast rotation orb feature point group in the container truck body area; determines the motion states of the container and the container truck body according to the first oriented fast rotation orb feature point group and the second oriented fast rotation orb feature point group corresponding to each frame of target image; and determines whether the container truck is lifted according to the motion states of the container and the container truck body. Since both the container area and the container truck body area are detected, and the motion states of the container and the container truck are determined to determine whether the container truck is lifted, the error caused by only detecting the area of the components related to the container truck is reduced. And after detecting both the container truck body area and the container area, by tracking the oriented fast rotation orb feature point group in the area, the error of tracking the entire area can be effectively reduced, ensuring the stability and accuracy of the tracking, and thus effectively improving the accuracy of the discrimination result of whether the container truck is lifted.

[0065] Embodiment 2

[0066] Figure 3 The flowchart of the container truck anti-lifting method provided in another embodiment of the present invention is as Figure 3 shown. The container truck anti-lifting method provided in this embodiment further refines steps 102 to 105 on the basis of the container truck anti-lifting method provided in Embodiment 1. The container truck anti-lifting method provided in this embodiment includes the following steps:

[0067] Step 201, obtain multiple frames of target images collected by a camera, where the target images include a container area and a container truck body area.

[0068] In this embodiment, the implementation manner of step 201 is similar to that of step 101 in Embodiment 1, and will not be elaborated here one by one.

[0069] Optionally, in this embodiment, the preset deep learning model is an improved center network model, also known as an improved CenterNet model. Correspondingly, as an optional manner of step 102 in Embodiment 1, it includes steps 202 to 204.

[0070] Step 202, input each frame of target image into the improved CenterNet model.

[0071] Optionally, as Figure 4As shown, the improved CenterNet model includes two convolutional layer branches. Before the two convolutional layer branches, there is at least one convolutional layer, such as in Figure 4 which includes two convolutional layers. After the two convolutional layer branches, there are an activation layer, a fully connected layer, and an output layer.

[0072] Among them, after inputting each frame of the target image into the improved CenterNet model, first, at least one convolutional layer extracts the feature map of each frame of the target image, and then the feature maps of each frame of the target image are respectively input into the two convolutional layer branches.

[0073] Step 203: Detect the container area, the first significant feature area in the container area, the truck body area, and the second significant feature area in the truck body area of each frame of the target image through the improved CenterNet model.

[0074] As an alternative implementation, as Figure 5 shown, Step 203 includes the following steps:

[0075] Step 2031: Extract the features of the overall area of each frame of the target image through the first convolutional layer branch to detect the container area and the truck body area.

[0076] Among them, the first convolutional layer branch includes at least one convolutional layer, such as [[ID=ZO]] Figure 4 shown, which includes three convolutional layers. If there are multiple convolutional layers, the multiple convolutional layers are connected in series in sequence. Each convolutional layer has a convolutional kernel.

[0077] In this embodiment, when inputting the feature map of each frame of the target image into the first convolutional layer branch, the convolutional kernel in at least one convolutional layer in the first convolutional layer branch performs a convolutional operation on the feature map of the target image output by the previous convolutional layer in sequence to extract the container area feature and the truck body area feature, and input the extracted container area feature and truck body area feature into the activation layer. After the activation layer performs a non - linear transformation on the container area feature and the truck body area feature, it is input into the fully connected layer. The fully connected layer detects the container area and the truck body area according to the container area feature and the truck body area feature after non - linear transformation, and outputs the container area and the truck body area of each frame of the target image through the output layer.

[0078] Step 2032: Extract the features of the significant feature area of each frame of the target image through the second convolutional layer branch to detect the first significant feature area and the second significant feature area.

[0079] Among them, the second convolutional layer branch includes at least one convolutional layer, such as in Figure 4It includes three convolutional layers. If there are multiple convolutional layers, the multiple convolutional layers are connected in series in sequence. Each convolutional layer has a convolutional kernel.

[0080] In this embodiment, when the feature maps of each frame of target images are input into the second convolutional layer branch, the convolutional kernels in at least one convolutional layer in the second convolutional layer branch perform convolutional operations on the feature maps of the target images output by the previous convolutional layer in sequence, so as to extract the first significant feature in the container area and the second significant feature in the container truck body area, and input the extracted first significant feature and second significant feature into the excitation layer. After the excitation layer performs non-linear transformation on the first significant feature and the second significant feature, it is input into the fully connected layer. The fully connected layer detects the first significant feature area and the second significant feature area according to the first significant feature and the second significant feature after non-linear transformation, and outputs the first significant feature area and the second significant feature area of each frame of target images through the output layer.

[0081] Among them, the significant feature area in the container area is the first significant feature area. The significant feature area of the container truck body area is the second significant feature area.

[0082] Step 204, output the container area, the first significant feature area, the container truck body area and the second significant feature area through the improved CenterNet model.

[0083] In this embodiment, the container area, the first significant feature area, the container truck body area and the second significant feature area are output through the output layer of the improved CenterNet model. Among them, the container area, the first significant feature area, the container truck body area and the second significant feature area can be framed by a rectangular box to represent the detected results.

[0084] It should be noted that in this embodiment, the improved CenterNet model is the improved CenterNet model trained to convergence. When training the improved CenterNet model, the training samples are image samples with regional annotations. The annotated areas include the container area, the first significant feature area in the container area, the container truck body area and the second significant feature area in the container truck body area. When training the improved CenterNet model, the first convolutional branch is used to extract features of the container area and the container truck body area, and the second convolutional branch is used to extract features of the container significant features and the container truck body significant features. Among them, the loss function is the sum of the loss function for overall area detection and the loss function for significant area detection. When the loss function reaches the minimum, it can indicate that the improved CenterNet model has been trained to convergence.

[0085] The container crane anti-lifting method provided in this embodiment inputs each frame of target image into an improved CenterNet model when performing target area detection on each frame of target image by using a preset deep learning model to obtain the container area and the container truck body area in each frame of target image; the improved CenterNet model detects the container area in each frame of target image, the first significant feature area in the container area, the container truck body area, and the second significant feature area in the container truck body area; the improved CenterNet model outputs the container area, the first significant feature area, the container truck body area, and the second significant feature area, which can not only detect the container area and the container truck body area, but also detect the first significant feature area in the container area and the second significant feature area in the container truck body area, and then can track the feature points around the significant feature area. Ensure the stability and accuracy of tracking the container area and the container truck body area.

[0086] Step 205: Extract the first oriented FAST and rotated BRIEF (ORB) feature point group in the container area of each frame of target image and the second oriented FAST and rotated BRIEF (ORB) feature point group in the container truck body area.

[0087] In this embodiment, the implementation manner of step 205 is similar to that of step 103 in Embodiment 1, and will not be elaborated here one by one.

[0088] Step 206: Screen the feature points in the first oriented FAST and rotated BRIEF (ORB) feature point group according to the first significant feature area to obtain the first target feature points.

[0089] As an optional implementation manner, as Figure 6 shown, step 206 includes the following steps:

[0090] Step 2061: Screen out the ORB feature points in the first oriented FAST and rotated BRIEF (ORB) feature point group whose distance from the feature points in the first significant feature area is less than a preset distance threshold.

[0091] Step 2062: Determine the ORB feature points whose distance from the feature points in the first significant feature area is less than the preset distance threshold as the first target feature points.

[0092] Specifically, in this embodiment, the distances between each ORB feature point in the first ORB feature point group and each feature point in the first significant feature region are calculated. If the distance between a certain ORB feature point and a certain feature point in the first significant feature region is less than the preset distance threshold, then this ORB feature point is retained. If the distances between a certain ORB feature point and all feature points in the first significant feature region are greater than or equal to the preset distance threshold, then this ORB feature point is deleted. Since the distances between the retained ORB feature points and the feature points in the first significant feature region are all relatively close, they can better represent the features in the container region. Therefore, the retained ORB feature points are determined as target feature points. The target feature points screened out from the first ORB feature point group are the first target feature points.

[0093] Step 207: Screen the feature points in the second oriented FAST-rotated ORB feature point group according to the second significant feature region to obtain the second target feature points.

[0094] As an alternative implementation, as Figure 7 shown, step 207 includes the following steps:

[0095] Step 2071: Screen out the ORB feature points in the second oriented FAST-rotated ORB feature point group whose distances from the feature points in the second significant feature region are less than the preset distance threshold.

[0096] Step 2072: Determine the ORB feature points whose distances from the feature points in the second significant feature region are less than the preset distance threshold as the second target feature points.

[0097] In this embodiment, the implementation manners of step 2071 - step 2072 are similar to those of step 2061 - step 2062. Specifically, the distances between each ORB feature point in the second ORB feature point group and each feature point in the second significant feature region are calculated. If the distance between a certain ORB feature point and a certain feature point in the second significant feature region is less than the preset distance threshold, then this ORB feature point is retained. If the distances between a certain ORB feature point and all feature points in the second significant feature region are greater than or equal to the preset distance threshold, then this ORB feature point is deleted. Since the distances between the retained ORB feature points and the feature points in the second significant feature region are all relatively close, they can better represent the features in the container region. Therefore, the retained ORB feature points are determined as target feature points. The target feature points screened out from the second ORB feature point group are the second target feature points.

[0098] Among them, for farming, the value of the preset distance threshold is not limited. For example, it can be a distance of 10 pixels, or other more appropriate values.

[0099] The container anti-lifting method provided in this embodiment, after extracting the first oriented fast rotating ORB feature point group in the container area of each frame of target image and the second oriented fast rotating ORB feature point group in the container truck body area, filters the feature points in the first oriented fast rotating ORB feature point group according to the first significant feature area to obtain the first target feature points; filters the feature points in the second oriented fast rotating ORB feature point group according to the second significant feature area to obtain the second target feature points. Since the first target feature points are the feature points closer to the first significant feature area, the first target feature points are the feature points that can better represent the significant features of the container area. Since the second target feature points are the feature points closer to the second significant feature area, the second target feature points are the feature points that can better represent the significant features of the container truck body area. Furthermore, when tracking the target feature points subsequently, the motion states of the tracked target feature points can more accurately and reliably represent the motion states of the container and the container truck body.

[0100] Step 208, determine the motion state of the container according to the first target feature points corresponding to each frame of target image.

[0101] As an optional implementation manner, in this embodiment, as Figure 8 shown, step 208 includes the following steps:

[0102] Step 2081, use the optical flow algorithm to track the first target feature points corresponding to each frame of target image, and determine the motion state of the first target feature points.

[0103] Specifically, in this embodiment, each frame of target image is input into the optical flow algorithm. And the first target feature points are marked in each frame of target image. Therefore, the optical flow algorithm determines the movement trajectory of each first target feature point according to the positions of the same first target feature points in two adjacent frames of target images. The motion state of each first target feature point is determined according to the movement trajectory of each first target feature point.

[0104] Among them, the motion state of each first target feature point is any one of the following states: stationary state, upward movement state, downward movement state, left and right movement state.

[0105] Step 2082, determine the motion state of the container according to the motion state of the first target feature points.

[0106] Specifically, in this embodiment, after determining the motion state of each first target feature point, the proportion of the first target feature points in each motion state is determined. The motion state of the first target feature point with the largest proportion is determined as the motion state of the container. For example, if the proportion of the first target feature points in the upward movement state is 85%, the proportion of the first target feature points in the stationary state is 15%, and the proportion of the first target feature points in the left-right movement state is 10%, then the motion state of the container is determined to be the upward movement state.

[0107] Step 209: Determine the motion state of the container truck body according to the second target feature points corresponding to each frame of the target image.

[0108] As an alternative implementation, in this embodiment, as Figure 9 shown, step 209 includes the following steps:

[0109] Step 2091: Use the optical flow algorithm to track the second target feature points corresponding to each frame of the target image, and determine the motion state of the second target feature points.

[0110] Step 2092: Determine the motion state of the container truck body according to the motion state of the second target feature points.

[0111] In this embodiment, the implementation manners of steps 2091 - 2092 are similar to those of steps 2081 - 2082. Specifically, each frame of the target image is input into the optical flow algorithm. And the second target feature points are marked in each frame of the target image. Therefore, the optical flow algorithm determines the movement trajectory of each second target feature point according to the positions of the same second target feature points in two adjacent frames of the target image. The motion state of each second target feature point is determined according to the movement trajectory of each second target feature point. The proportion of the second target feature points in each motion state is determined. The motion state of the second target feature point with the largest proportion is determined as the motion state of the container.

[0112] It should be noted that steps 208 - 209 are an alternative implementation manner of step 104 in Embodiment 1.

[0113] Step 210: Determine whether the container truck is lifted according to the motion states of the container and the container truck body.

[0114] In this embodiment, since the motion state of the container is any one of the stationary state, the upward movement state, the downward movement state, and the left - right movement state. The motion state of the container truck body is any one of the stationary state, the upward movement state, the downward movement state, and the left - right movement state. Therefore, there are 16 combinations of the motion states of the container and the container truck body.

[0115] Among them, if it is determined that the motion states of both the container and the tractor body are in the upward movement state, it indicates that the lock on the side close to the camera is not unlocked, and it is determined that the tractor is lifted. Or if it is determined that the motion state of the container is in the upward movement state and the motion state of the tractor body is in the downward movement state, it indicates that the lock on the side far from the camera is not unlocked, and it is determined that the tractor is lifted. The motion states of the container and the tractor body in other cases are all determined to be safe situations, that is, the tractor is not lifted. For example, if the motion state of the container is in the downward movement state and the motion state of the tractor body is in the static state, it indicates that it is a packing operation, and there is no danger of lifting the tractor.

[0116] In the tractor anti-lifting method provided in this embodiment, when determining the motion state of the container according to the first target feature points corresponding to each frame of the target image, the optical flow algorithm is used to track the first target feature points corresponding to each frame of the target image to determine the motion state of the first target feature points; the motion state of the container is determined according to the motion state of the first target feature points. When determining the motion state of the tractor body according to the second target feature points corresponding to each frame of the target image, the optical flow algorithm is used to track the second target feature points corresponding to each frame of the target image to determine the motion state of the second target feature points; the motion state of the tractor body is determined according to the motion state of the second target feature points. Since the optical flow algorithm can accurately track the target feature points, the motion state of the target feature points can be accurately determined. Compared with the prior art that uses the KCF algorithm and the Hough transform algorithm to track the overall area, the stability and accuracy of the tracking can be effectively improved, and thus the determined container area and tractor body state are more accurate and stable.

[0117] Embodiment III

[0118] Figure 10 It is a signaling flowchart of the tractor anti-lifting method provided in another embodiment of the present invention. As Figure 10 shown, the execution subject of the tractor anti-lifting method provided in this embodiment is the tractor anti-lifting system. The tractor anti-lifting system includes: the control system of the crane, a camera, and an electronic device. The tractor anti-lifting method provided in this embodiment includes the following steps:

[0119] Step 301, the control system of the crane monitors that the lock connecting the crane and the container sends a locking signal.

[0120] In this embodiment, after the crane uses the lock to hook the container, it monitors whether the lock sends a locking signal. If a locking signal is sent, it is determined that the container lifting operation can be performed. The driver can be notified to perform the manual lifting operation.

[0121] Step 302, the control system of the crane sends a shooting start instruction to the camera.

[0122] Step 303, the control system of the crane sends a request for preventing the container truck from being lifted to the electronic device.

[0123] Step 304, the camera collects multiple frames of target images according to the shooting start instruction.

[0124] Among them, the target images include the container area and the container truck body area.

[0125] Step 305, the camera sends the collected multiple frames of target images to the electronic device.

[0126] Step 306, the electronic device performs detection of preventing the container truck from being lifted according to the request for preventing the container truck from being lifted and the multiple frames of target images.

[0127] In this embodiment, after the locking signal is sent by the locking device, it indicates that the driver is about to perform manual lifting operation. In order to prevent the container truck from being lifted, a shooting start instruction is sent to the camera to control the camera to periodically shoot at least one side of the container and the container truck body, and multiple frames of target images are obtained. And a request for preventing the container truck from being lifted is sent to the electronic device, so that the electronic device uses the method for preventing the container truck from being lifted provided in any one of the above embodiments to determine whether the container truck is lifted.

[0128] Step 307, if the electronic device determines that the container truck is lifted, it sends an alarm message to the control system of the crane.

[0129] Step 308, the control system of the crane controls the crane to stop working.

[0130] In this embodiment, the alarm message can be in the form of voice, the form of an indicator light, or a combination of voice and indicator light, or other forms, and this embodiment does not limit this.

[0131] Step 309, if the electronic device determines that the container truck is not lifted, it obtains the travel height of the crane lifting the container. When the travel height reaches the safe operation height, the electronic device stops detection.

[0132] In this embodiment, the electronic device has been continuously determining whether the container truck is lifted, and after each determination that it is not lifted, it obtains the travel height of the crane lifting the container from the control system of the crane, compares the travel height with the safe operation height. If the travel height information reaches the safe operation height, the electronic device no longer detects whether the container truck is lifted.

[0133] Step 310, the control system of the crane switches the working mode from the manual operation mode to the automatic operation mode, and continues to lift the container until the target position is reached.

[0134] In this embodiment, the control system of the crane obtains the travel height of the crane when lifting the container. When the travel height reaches the safe operation height, it notifies the driver to stop manual operation and switches the manual operation mode to the automatic operation mode to continue lifting the container until the target position is reached.

[0135] The container anti-lifting method provided in this embodiment adopts the container anti-lifting method only before the crane lifts the container to the safe operation height. While ensuring container anti-lifting, it can effectively reduce the detection time of container anti-lifting, and thus can effectively save the computing resources of the electronic device.

[0136] Embodiment 4

[0137] Figure 11 is a schematic structural diagram of a container anti-lifting device provided in an embodiment of the present invention. As Figure 11 shown, the container anti-lifting device provided in this embodiment is located in the electronic device. Then, the container anti-lifting device 40 provided in this embodiment includes: an image acquisition module 41, a region detection module 42, a feature point group extraction module 43, a motion state determination module 44, and a lifting determination module 45.

[0138] Among them, the image acquisition module 41 is used to acquire multiple frames of target images collected by the camera. The target images include a container region and a container truck body region. The region detection module 42 is used to perform target region detection on each frame of the target image by using a preset deep learning model to obtain the container region and the container truck body region in each frame of the target image. The feature point group extraction module 43 is used to extract a first oriented FAST and rotated BRIEF (orb) feature point group in the container region of each frame of the target image and a second oriented FAST and rotated BRIEF (orb) feature point group in the container truck body region. The motion state determination module 44 is used to determine the motion states of the container and the container truck body according to the first oriented FAST and rotated BRIEF (orb) feature point group and the second oriented FAST and rotated BRIEF (orb) feature point group corresponding to each frame of the target image. The lifting determination module 45 is used to determine whether the container truck is lifted according to the motion states of the container and the container truck body.

[0139] The container anti-lifting device provided in this embodiment can execute the container anti-lifting method provided in the above Embodiment 1 or Embodiment 3. The specific implementation manners and principles are similar and will not be elaborated one by one.

[0140] Optionally, the preset deep learning model is an improved CenterNet model.

[0141] The region detection module 42 is specifically configured to input each frame of target image into the improved CenterNet model; detect the container region, the first significant feature region in the container region, the truck body region, and the second significant feature region in the truck body region of each frame of target image through the improved CenterNet model; and output the container region, the first significant feature region, the truck body region, and the second significant feature region through the improved CenterNet model.

[0142] Optionally, the improved CenterNet model includes two convolutional layer branches. Correspondingly, when the region detection module 42 detects the container region, the first significant feature region in the container region, the truck body region, and the second significant feature region in the truck body region of each frame of target image through the improved CenterNet model, it is specifically configured to extract the features of the overall region of each frame of target image through the first convolutional layer branch to detect the container region and the truck body region; and extract the features of the significant feature region of each frame of target image through the second convolutional layer branch to detect the first significant feature region and the second significant feature region.

[0143] Optionally, the container anti-lifting device provided in this embodiment further includes: a feature point screening module.

[0144] The feature point screening module is used to screen the feature points in the first oriented fast rotating orb feature point group according to the first significant feature region to obtain the first target feature points; and screen the feature points in the second oriented fast rotating orb feature point group according to the second significant feature region to obtain the second target feature points.

[0145] Optionally, when the feature point screening module screens the feature points in the first oriented fast rotating orb feature point group according to the first significant feature region to obtain the first target feature points, it is specifically configured to screen out the orb feature points in the first oriented fast rotating orb feature point group whose distance from the feature points in the first significant feature region is less than a preset distance threshold; and determine the orb feature points whose distance from the feature points in the first significant feature region is less than the preset distance threshold as the first target feature points.

[0146] Optionally, when the feature point screening module screens the feature points in the second oriented fast rotating orb feature point group according to the second significant feature region to obtain the second target feature points, it is specifically configured to screen out the orb feature points in the second oriented fast rotating orb feature point group whose distance from the feature points in the second significant feature region is less than a preset distance threshold; and determine the orb feature points whose distance from the feature points in the second significant feature region is less than the preset distance threshold as the second target feature points.

[0147] Optionally, the motion state determination module 44 is specifically configured to determine the motion state of the container according to the first target feature points corresponding to each frame of target image; and determine the motion state of the truck body according to the second target feature points corresponding to each frame of target image.

[0148] Optionally, when determining the motion state of the container according to the first target feature points corresponding to each frame of target image, the motion state determination module 44 is specifically configured to track the first target feature points corresponding to each frame of target image by using an optical flow algorithm to determine the motion state of the first target feature points; and determine the motion state of the container according to the motion state of the first target feature points.

[0149] Optionally, when determining the motion state of the truck body according to the second target feature points corresponding to each frame of target image, the motion state determination module 44 is specifically configured to track the second target feature points corresponding to each frame of target image by using an optical flow algorithm to determine the motion state of the second target feature points; and determine the motion state of the truck body according to the motion state of the second target feature points.

[0150] Optionally, the motion states of the container and the truck body include any one of the following states: stationary state, upward movement state, downward movement state, left and right movement state.

[0151] Optionally, the lifting determination module 45 is specifically configured to determine that the truck is lifted if it is determined that the motion states of both the container and the truck body are in the upward movement state. Or determine that the truck is lifted if it is determined that the motion state of the container is in the upward movement state and the motion state of the truck body is in the downward movement state.

[0152] The truck anti-lifting device provided in this embodiment can execute the truck anti-lifting method provided in the above Embodiment 2 or Embodiment 3. The specific implementation manners and principles are similar and will not be elaborated one by one.

[0153] Embodiment 5

[0154] Figure 12 It is a block diagram of an electronic device provided in an embodiment of the present invention. As Figure 12 shown, the electronic device 50 provided in this embodiment includes: at least one processor 52; and a memory 51 communicatively connected to the at least one processor.

[0155] Wherein, the memory 51 stores instructions executable by the at least one processor 52, and the instructions are executed by the at least one processor 52 so that the at least one processor 52 can execute the truck anti-lifting method provided in any embodiment of the present invention.

[0156] Wherein, the memory 51 and the processor 52 are connected through a bus 53.

[0157] For relevant descriptions, reference may be made to the relevant descriptions and effects corresponding to the steps of the container truck anti-lifting method provided in any one of the embodiments for understanding, and no further elaboration will be provided here.

[0158] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the container truck anti-lifting method provided in any one of the embodiments.

[0159] An embodiment of the present invention further provides a computer program product, including a computer program, and the computer program is used to execute the container truck anti-lifting method provided in any one of the above embodiments by a processor.

[0160] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0161] Furthermore, it should be noted that although the steps in the flowchart are sequentially shown according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0162] It should be understood that the above device embodiments are only illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0163] In addition, unless otherwise specified, in each embodiment of the present application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0164] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0165] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disc.

[0166] In the above embodiments, the descriptions of the various embodiments each have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

Claims

1. A method for preventing a container truck from being lifted, characterized in that, Including: Obtaining multiple frames of target images collected by a camera, where the target images include a container area and a truck body area; Using a preset deep learning model to perform target area detection on each frame of the target image to obtain the container area in each frame of the target image, the first significant feature area in the container area, the truck body area, and the second significant feature area in the truck body area; Extracting the first oriented FAST and rotated BRIEF (ORB) feature point group in the container area of each frame of the target image and the second oriented FAST and rotated BRIEF (ORB) feature point group in the truck body area; Screening out ORB feature points from the first oriented FAST and rotated BRIEF (ORB) feature point group whose distance from the feature points in the first significant feature area is less than a preset distance threshold; Determining the ORB feature points whose distance from the feature points in the first significant feature area is less than the preset distance threshold as the first target feature points; Screening out ORB feature points from the second oriented FAST and rotated BRIEF (ORB) feature point group whose distance from the feature points in the second significant feature area is less than a preset distance threshold; Determining the ORB feature points whose distance from the feature points in the second significant feature area is less than the preset distance threshold as the second target feature points; Using an optical flow algorithm to track the first target feature points corresponding to each frame of the target image to determine the motion state of the first target feature points; Determining the motion state of the container according to the motion state of the first target feature points; Using an optical flow algorithm to track the second target feature points corresponding to each frame of the target image to determine the motion state of the second target feature points; Determining the motion state of the truck body according to the motion state of the second target feature points; Determining whether the truck is lifted according to the motion states of the container and the truck body; 2. The method according to claim 1, wherein The preset deep learning model is an improved CenterNet model; The step of using a preset deep learning model to perform target area detection on each frame of the target image to obtain the container area and the truck body area in each frame of the target image includes: Inputting each frame of the target image into the improved CenterNet model; Detecting the container area in each frame of the target image, the first significant feature area in the container area, the truck body area, and the second significant feature area in the truck body area through the improved CenterNet model; Outputting the container area, the first significant feature area, the truck body area, and the second significant feature area through the improved CenterNet model; 3. The method according to claim 2, wherein, The improved CenterNet model includes two convolutional layer branches; The step of detecting the container area in each frame of the target image, the first significant feature area in the container area, the truck body area, and the second significant feature area in the truck body area through the improved CenterNet model includes: Performing feature extraction on the overall area of each frame of the target image through the first convolutional layer branch to detect the container area and the truck body area; Feature extraction of the significant feature regions of each frame of target image is performed through the second convolutional layer branch to detect the first significant feature region and the second significant feature region.

4. The method according to claim 2, wherein After extracting the first group of oriented FAST and rotated BRIEF (ORB) feature points in the container region of each frame of target image and the second group of oriented FAST and rotated BRIEF (ORB) feature points in the container truck body region, the following steps are further included: Filter the feature points in the first group of oriented FAST and rotated BRIEF (ORB) feature points according to the first significant feature region to obtain first target feature points; Filter the feature points in the second group of oriented FAST and rotated BRIEF (ORB) feature points according to the second significant feature region to obtain second target feature points.

5. The method according to claim 4, characterized in that, Determining the motion states of the container and the container truck body according to the first group of oriented FAST and rotated BRIEF (ORB) feature points and the second group of oriented FAST and rotated BRIEF (ORB) feature points corresponding to each frame of target image includes: Determining the motion state of the container according to the first target feature points corresponding to each frame of target image; Determining the motion state of the container truck body according to the second target feature points corresponding to each frame of target image.

6. The method according to any one of claims 1-5, characterized in that, The motion states of the container and the container truck body include any one of the following states: stationary state, upward movement state, downward movement state, left and right movement state; Determining whether the container truck is lifted according to the motion states of the container and the container truck body includes: If it is determined that the motion states of both the container and the container truck body are in the upward movement state, it is determined that the container truck is lifted; Or if it is determined that the motion state of the container is in the upward movement state and the motion state of the container truck body is in the downward movement state, it is determined that the container truck is lifted.

7. A truck anti-lifting device, characterized in that, It includes: An image acquisition module, configured to acquire multiple frames of target images collected by a camera, where the target images include a container region and a container truck body region; A region detection module, configured to perform target region detection on each frame of target image by using a preset deep learning model to obtain the container region, the first significant feature region in the container region, the container truck body region, and the second significant feature region in the container truck body region in each frame of target image; A feature point group extraction module, configured to extract the first group of oriented FAST and rotated BRIEF (ORB) feature points in the container region of each frame of target image and the second group of oriented FAST and rotated BRIEF (ORB) feature points in the container truck body region; A feature point screening module, configured to screen out ORB feature points with a distance less than a preset distance threshold from the feature points in the first significant feature region from the first group of oriented FAST and rotated BRIEF (ORB) feature points; determining the ORB feature points with a distance less than the preset distance threshold from the feature points in the first significant feature region as first target feature points; Screening out ORB feature points with a distance less than a preset distance threshold from the feature points in the second significant feature region from the second group of oriented FAST and rotated BRIEF (ORB) feature points; determining the ORB feature points with a distance less than the preset distance threshold from the feature points in the second significant feature region as second target feature points; A motion state determination module, configured to track first target feature points corresponding to each frame of target images by using an optical flow algorithm, determine the motion states of the first target feature points; determine the motion state of the container according to the motion states of the first target feature points; track second target feature points corresponding to each frame of target images by using the optical flow algorithm, determine the motion states of the second target feature points; determine the motion state of the truck body according to the motion states of the second target feature points; A hoisting judgment module, configured to determine whether the truck is hoisted according to the motion states of the container and the truck body.

8. An electronic device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1-6.

Images