Computer vision-based automatic container corner alignment method and system

Abstract

The present invention relates to the field of port hoisting equipment, and provides a computer vision-based automatic container corner alignment method and system. The method comprises the following steps: calibrating a spreader camera arranged on a container spreader to obtain the conversion relationship between an object point in a three-dimensional space and a camera image pixel; controlling the spreader camera to capture first image data downwards, and using image processing technology to detect and recognize container corners and ground markings; and converting the container corners and the ground markings into the three-dimensional space on the basis of the conversion relationship, calculating offsets between the container corners and the ground markings in the three-dimensional space, and inputting the offsets into a spreader control system to realize automatic container corner engagement. The present invention can calculate offsets between container corners and ground markings in real time, thereby realizing automatic container corner engagement.

Description

A computer vision-based automatic container corner alignment method and system

[0001] The present application relates to the field of port hoisting equipment, in particular to a computer vision-based automatic container corner alignment method and system. BACKGROUND

[0002] With the economic development of coastal cities, the port loading and unloading capacity is continuously expanding, and the lifting weight of the hoisting machinery is getting larger and larger, which puts forward higher requirements on energy consumption, efficiency and reliability. For the shore crane loading and unloading equipment, during the loading and unloading of containers, how to achieve accurate container alignment has always been a research topic that large-scale port equipment pays great attention to in the application process. At present, when the port is carrying out container loading and unloading operation, the operator controls the spreader to align the container on the ground mark to realize accurate container alignment. If the operator finds that the alignment position deviation is large, the spreader needs to be controlled manually for secondary container alignment. Therefore, manual container alignment has the problems of low safety and low efficiency. SUMMARY

[0003] In order to solve the technical problems in the background art, the present application provides a computer vision-based automatic container corner alignment method, system, electronic device, computer storage medium and computer program product.

[0004] The present application provides a computer vision-based automatic container corner alignment method, which comprises the following steps:

[0005] Calibrating the spreader camera arranged on the container spreader to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels;

[0006] Controlling the spreader camera to shoot the first image data downward, and using image processing technology to detect and identify the container corner and the ground mark;

[0007] According to the conversion relationship, converting each container corner and the ground mark into the three-dimensional space, calculating the offset amount of the container corner and the ground mark in the three-dimensional space, and inputting the offset amount into the spreader control system to realize automatic container corner alignment.

[0008] Optionally, the spreader camera arranged on the container spreader is calibrated to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels, which comprises:

[0009] Taking the set known calibration object as a reference, determining the state of the internal structure of the spreader camera when the spreader camera is collecting images, i.e. the first internal parameter and the first relationship between the first position and the first direction of the spreader camera relative to the photographed scene, and establishing the conversion relationship between the object points in the three-dimensional space and the camera image pixels according to the first relationship.

[0010] Optionally, the detecting and recognizing the corner of the container and the ground mark by using the image processing technology comprises:

[0011] detecting and recognizing the corner of the container and the ground mark by using the image processing technology comprises:

[0012] Optionally, the converting the corner of the container and the ground mark to the three-dimensional space according to the conversion relationship comprises:

[0013] determining a second relationship between the spreader camera and the second position and the second direction of the photographed scene according to the projection relationship, and calculating the three-dimensional coordinates of the image coordinates of the corner of the container and the ground mark in the three-dimensional space according to the conversion relationship and the second relationship.

[0014] Optionally, before the detecting and recognizing the corner of the container and the ground mark by using the image processing technology, the method further comprises:

[0015] acquiring second image data photographed by the spreader camera, and extracting a position fluctuation frequency of the target object according to the second image data;

[0016] inputting the position fluctuation frequency into the lifting prediction model, and the lifting prediction model outputs a lifting state probability;

[0017] when the lifting probability state is higher than a probability threshold, generating a detection signal, and the detection signal is used to trigger the detecting and recognizing the corner of the container and the ground mark by using the image processing technology.

[0018] Optionally, the lifting prediction model is trained by the following method:

[0019] publishing the data format of the lifting prediction model and the training data elements to a consortium chain network;

[0020] other nodes in the consortium chain network generate a training data set according to the data format, and perform distributed training on the lifting prediction model;

[0021] the local node receives the key parameters of the trained lifting prediction model fed back by each other node, replaces the key parameters one by one into the local lifting prediction model, tests the replaced lifting prediction model using a test data set, and determines the optimal key parameters and the corresponding lifting prediction model according to the test result;

[0022] The number of the other nodes is determined according to the average weight of the container hoisted by the spreader, and specifically, the number of the other nodes is negatively related to the average weight.

[0023] The application further provides a computer vision-based automatic alignment system for a container corner, characterized in that the system comprises a spreader camera and a processing module.

[0024] The processing module is configured to implement the following method steps.

[0025] The spreader camera arranged on the container spreader is calibrated to obtain a conversion relationship between a point in a three-dimensional space and a camera image pixel.

[0026] The spreader camera is controlled to downwardly capture first image data, and an image processing technology is used to detect and identify the container corner and the ground marking.

[0027] The container corner and the ground marking are converted to the three-dimensional space according to the conversion relationship, the offset of the container corner and the ground marking in the three-dimensional space is calculated, and the offset is input into a spreader control system to realize automatic alignment of the container corner.

[0028] The application further provides an electronic device, comprising a memory storing executable program codes, and a processor coupled with the memory; the processor invokes the executable program codes stored in the memory to execute the method according to any one of the preceding method embodiments.

[0029] The application further provides a computer storage medium, which stores a computer program; when the computer program is executed by a processor, the method according to any one of the preceding method embodiments is executed.

[0030] The application further provides a computer program product, which comprises a computer program stored on a non-transitory computer readable medium; when the computer program is executed by a processor, the method according to any one of the preceding method embodiments is executed.

[0031] In the application, the camera calibration technology, computer vision technology and nonlinear optimization technology are used to automatically identify and detect the positions of the container corner and the ground marking in the image, the structure of the container corner and the ground marking in the three-dimensional space is recovered from the image through the stereo vision technology, and the offset between the container corner and the ground marking is calculated in real time.

[0032] Since the application only needs to install a common camera, the application can be applied to various container yards of various ports and can adapt to various site environments on the basis of high precision. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0034] Fig. 1 is a flowchart of a computer vision-based automatic positioning method for container corners disclosed by an embodiment of the present application.

[0035] Fig. 2 is a schematic diagram of the training process of a hoisting prediction model disclosed by an embodiment of the present application.

[0036] Fig. 3 is a structural schematic diagram of a high-definition camera for construction site safety management disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0038] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0039] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0040] In the description of the present application, it should be noted that if the terms such as "upper", "lower", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0041] It should be noted that the features in the embodiments of the present application can be combined with each other without conflict.

[0042] As shown in Figure 1, the embodiment of the present application discloses a computer vision-based automatic container corner alignment method, which comprises the following steps:

[0043] Calibrate the spreader camera on the container spreader to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels.

[0044] Control the spreader camera to shoot downward the first image data, and use image processing technology to detect and identify the container corner and the ground marking.

[0045] According to the conversion relationship, convert the container corners and the ground marking to the three-dimensional space, calculate the offset of the container corner and the ground marking in the three-dimensional space, and input the offset into the spreader control system to realize the automatic container corner alignment.

[0046] Compared with the manual container alignment control method mentioned in the background art, the present application spreads the spreader camera on the container spreader, identifies the container corner and the ground marking in the image through image processing technology, calculates the offset of the container corner and the ground marking through the conversion relationship between the image pixels and the real world points, and automatically determines the correction control parameters based on the calculated offset by the spreader control system, thereby realizing the automatic container corner alignment.

[0047] Among them, the spreader camera needs to be calibrated first to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels, and the container corners and the ground marking are also converted to the three-dimensional space, and the above-mentioned offset is calculated in the three-dimensional space based on the above-mentioned conversion relationship.

[0048] Optionally, calibrate the spreader camera spread on the container spreader to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels, which comprises:

[0049] Determine the state of the internal structure of the spreader camera when it collects images, i.e. the first internal parameters and the first relationship between the first position and the first direction of the spreader camera relative to the photographed scene, according to the first relationship, and establish the conversion relationship between the object points in the three-dimensional space and the camera image pixels.

[0050] Since the position of the spreader camera on the spreader is fixed after installation, it only needs to be calibrated once after the spreader camera is installed, without repeated calibration. The calibration result is stored after computer processing as the input of the subsequent part.

[0051] Optionally, the use of image processing technology to detect and identify the container corner and the ground marking, comprising:

[0052] The first image data is subjected to straight line detection, feature point detection, feature point matching processing, the corners of the container and the ground marking are recognized, and the image coordinates of the corners of the container and the ground marking are determined.

[0053] The application solves the problems of recognizing and detecting the corners of the container and the ground marking in the port environment by using image processing technology.

[0054] Optionally, the conversion of the corners of the container and the ground marking into the three-dimensional space according to the conversion relationship comprises:

[0055] The second relationship between the spreader camera and the second position and the second direction of the photographed scene is determined according to the height of the spreader and the projection relationship, and the three-dimensional coordinates of the image coordinates of the corners of the container and the ground marking in the three-dimensional space are calculated according to the conversion relationship and the second relationship.

[0056] Optionally, before the detection and recognition of the corners of the container and the ground marking using the image processing technology, the method further comprises:

[0057] The second image data photographed by the spreader camera is obtained, and the position fluctuation frequency of the target object is obtained according to the second image data;

[0058] The position fluctuation frequency is input into the lifting prediction model, and the lifting prediction model outputs the lifting state probability;

[0059] When the lifting probability state is higher than the probability threshold, a detection signal is generated, and the detection signal is used to trigger the detection and recognition of the corners of the container and the ground marking using the image processing technology.

[0060] Before the detection and recognition of the corners of the container and the ground marking using the image processing technology, it is necessary to judge whether the spreader is lifting the container, that is, only when the spreader is lifting the container, the detection and recognition of the corners of the container and the ground marking using the image processing technology is triggered. By setting in this way, invalid detection and recognition of the corners of the container and the ground marking when the spreader is not performing lifting operation can be avoided, and the data processing load is reduced.

[0061] Specifically, since the sling is in a rhythmic slow swing when it is in a hoisting-free state, and the tension will be sharply increased when the hoisting starts, and then high-frequency vibration occurs, correspondingly, the sling camera can detect the position fluctuation frequency of the target object in the image. The present application also sets up a hoisting prediction model, which processes the position fluctuation frequency of the target object in the second image data in depth by the model to obtain the hoisting state probability, that is, the probability of the sling in the execution of the container hoisting. Wherein, the second image data is obtained by the sling camera according to the preset detection period, which can be multiple images or a video. Therefore, the present application predicts whether the sling is currently performing hoisting operation by analyzing the position fluctuation frequency of the target object (such as ground marking, other containers, etc.) in the second image data (that is, the position fluctuation frequency of the target object in the image, which can be the average, median, maximum value, etc. of the fluctuation frequency of multiple sub-periods within the detection period).

[0062] The hoisting prediction model is preferably based on ResNet (Residual Network). In a traditional deep neural network, if the number of network layers increases, the learning ability of the model will be stronger in theory, but in fact, too deep network will cause gradient disappearance or gradient explosion, making it difficult to train the network. ResNet solves this problem by adding skip connections or shortcut connections, which allow signals in the network to bypass one or more layers directly.

[0063] The training data set of the hoisting prediction model has a data element format of [position fluctuation frequency, hoisting state label]. The hoisting state label includes multiple different hoisting state probabilities, which are used to describe the probability of the sling being in the hoisting operation state under the associated position fluctuation frequency. The hoisting prediction model is fully trained and tested using the above training data set until the prediction accuracy index is met.

[0064] Optionally, referring to FIG. 2, the hoisting prediction model is trained in the following way:

[0065] The hoisting prediction model and the data format of the training data element are published to the alliance chain network;

[0066] Other nodes in the alliance chain network generate a training data set according to the data format and perform distributed training on the hoisting prediction model;

[0067] The local node receives the key parameters of the trained hoisting prediction model fed back by each other node, replaces the key parameters one by one into the local hoisting prediction model, tests the replaced hoisting prediction model using a test data set, and determines the optimal key parameters and the corresponding hoisting prediction model according to the test result.

[0068] The number of the other nodes is determined according to the average weight of the container hoisted by the spreader.

[0069] The present application adopts a distributed training method to train the hoisting prediction model. Specifically, the data format of the hoisting prediction model and the corresponding training data elements is published to the alliance chain network, and the other nodes in the alliance chain network generate training data sets by themselves for distributed training, that is, each other node independently generates training data sets and trains the model, and each other node feeds back the key parameters in the hoisting prediction model to the local node after completing the training, including the weight, bias and batch normalization parameters (scaling factor, offset) of each convolutional layer and fully connected layer. The local node inputs the received key parameters one by one into the hoisting prediction model and tests it using the local test data set, selects the optimal key parameters according to the test result, and replaces the key parameters into the local hoisting prediction model to obtain the final model.

[0070] The number of the other nodes participating in the distributed training of the model is determined according to the average weight of the container hoisted by the corresponding spreader. The average weight of the container hoisted by the corresponding spreader can be obtained by historical data statistics. When the average weight of the container hoisted by the corresponding spreader is larger, the vibration frequency of the spreader at the moment when hoisting starts is larger, and at this time, only a small number of other nodes are needed to participate in the distributed training. When the average weight of the container hoisted by the corresponding spreader is smaller, the vibration frequency of the spreader at the moment when hoisting starts is smaller, and at this time, more other nodes are needed to participate in the distributed training, so as to improve the accurate identification of the small amplitude of the vibration frequency of the spreader (the position fluctuation frequency of the target object is also smaller). Therefore, the present application sets the number of other nodes to be negatively related to the average weight.

[0071] Referring to FIG. 3, the present application also discloses a computer vision-based automatic positioning system for container corners, characterized in that the system comprises a spreader camera and a processing module; the spreader camera is electrically connected to the processing module.

[0072] The processing module is used to implement the following method steps:

[0073] The spreader camera arranged on the container spreader is calibrated to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels.

[0074] The spreader camera shoots downwardly to obtain first image data, and the image processing technology is used to detect and identify the corner of the container and the ground marking line;

[0075] According to the conversion relationship, the corner of the container and the ground marking line are converted into a three-dimensional space, the offset of the corner of the container and the ground marking line is calculated in the three-dimensional space, and the offset is input into the spreader control system to realize automatic container cornering of the container.

[0076] The embodiment of the present application also discloses an electronic device, comprising: a memory storing executable program codes; a processor coupled with the memory; the processor calls the executable program codes stored in the memory to execute the method as described in the foregoing embodiments.

[0077] The embodiment of the present application also discloses a computer storage medium, which stores a computer program, and the computer program is executed by a processor to execute the method as described in the foregoing embodiments.

[0078] The embodiment of the present application also discloses a computer program product, comprising a computer program stored on a non-transitory computer readable medium, and the computer program is executed by a processor to execute the method as described in any one of the foregoing embodiments.

[0079] It should be noted that the storage module (102) in the second embodiment, the memory in the third embodiment and the computer storage medium in the fourth embodiment can be, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or other memory technology, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD) or other optical storage, a magnetic cassette tape, and the like.

[0080] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0081] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks and / or flowchart flow or flows and / or block or blocks of the block diagram.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks and / or flowchart flow or flows and / or block or blocks of the block diagram.

[0083] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A computer vision based automatic container corner alignment method, characterized in that: The method comprises the following steps: Calibrate the spreader camera arranged on the container spreader to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels; Control the spreader camera to shoot downward the first image data, and use image processing technology to detect and identify the container corners and ground markings; According to the conversion relationship, convert the container corners and ground markings to the three-dimensional space, and calculate the offset of the container corners and ground markings in the three-dimensional space, and input the offset into the spreader control system to realize the automatic container corner landing.

2. The computer vision based automatic container corner alignment method of claim 1, wherein: Calibrate the spreader camera arranged on the container spreader to obtain the conversion relationship between the object points in the three-dimensional space and the camera image pixels, comprising: With the set known calibration object as the reference, determine the state of the internal structure of the spreader camera when it is collecting images, that is, the first internal parameters and the first relationship between the first position and the first direction of the spreader camera relative to the photographed scene, and establish the conversion relationship between the object points in the three-dimensional space and the camera image pixels according to the first relationship.

3. The computer vision based automatic container corner alignment method of claim 2, wherein: The use of image processing technology to detect and identify the container corners and ground markings, comprising: Perform straight line detection, feature point detection, and feature point matching processing on the first image data to identify the container corners and ground markings, and determine the image coordinates of the container corners and ground markings.

4. The computer vision-based automatic container corner alignment method of claim 3, wherein: According to the conversion relationship, convert the container corners and ground markings to the three-dimensional space, comprising: Determine the second relationship between the second position and the second direction of the spreader camera relative to the photographed scene in combination with the spreader height and the projection relationship, and calculate the three-dimensional coordinates of the image coordinates of the container corners and ground markings in the three-dimensional space according to the conversion relationship and the second relationship.

5. The computer vision based automatic container corner alignment method of claim 1, wherein: Before using image processing technology to detect and identify the container corners and ground markings, the method further comprises: Obtain the second image data shot by the spreader camera, and obtain the position fluctuation frequency of the target object according to the second image data; Input the position fluctuation frequency into the hoisting prediction model, and the hoisting prediction model outputs the hoisting state probability; When the hoisting probability state is higher than the probability threshold, generate a detection signal, which is used to trigger the use of image processing technology to detect and identify the container corners and ground markings.

6. The computer vision based automatic container corner alignment method of claim 5, wherein: The hoisting prediction model is trained in the following way: Publish the data format of the hoisting prediction model and the training data elements to the alliance chain network; Other nodes in the alliance chain network generate a training data set according to the data format and perform distributed training on the hoisting prediction model; The local node receives the key parameters of the trained hoisting prediction model fed back by each other node, replaces the key parameters one by one into the local hoisting prediction model, tests the replaced hoisting prediction model using a test data set, and determines the optimal key parameters and corresponding hoisting prediction model according to the test results; Wherein, the number of other nodes is determined according to the average weight of the container hoisted by the spreader, specifically, the number of other nodes is negatively related to the average weight.

7. A computer vision based automatic container corner alignment system, characterized by: The system comprises a spreader camera, a processing module; the spreader camera is electrically connected with the processing module; The processing module is used to realize the following method steps: Calibrating the spreader camera arranged on the container spreader to obtain a conversion relationship between a point in a three-dimensional space and a camera image pixel; Controlling the spreader camera to shoot a first image data downward, and using an image processing technology to detect and identify the container corner and the ground marking; Converting each container corner and the ground marking to the three-dimensional space according to the conversion relationship, calculating the offset of the container corner and the ground marking in the three-dimensional space, and inputting the offset into a spreader control system to realize automatic container corner landing.

8. An electronic device, comprising: A memory storing executable program codes; A processor coupled with the memory; characterized in that the processor calls the executable program codes stored in the memory to execute the method according to any one of claims 1-6.

9. A computer storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to execute the method according to any one of claims 1-6.

10. A computer program product comprising a computer program stored on a non-transitory computer readable medium, characterized in that: The computer program is executed by the processor to execute the method according to any one of claims 1-6.

Images