Container corner automatic alignment method and system based on computer vision

Abstract

The invention provides a container corner automatic alignment method and system based on computer vision, and belongs to the field of port hoisting equipment. The method comprises the following steps: calibrating a lifting appliance camera arranged on a container lifting appliance to obtain a conversion relation between an object point and a camera image pixel in a three-dimensional space; controlling the lifting appliance camera to shoot first image data downwards, and detecting and identifying corners and ground marking lines of the container by using an image processing technology; and according to the conversion relation, converting each container corner and the ground marking line of the container into a three-dimensional space, calculating the offset of the container corner and the ground marking line in the three-dimensional space, and connecting the offset into a lifting appliance control system to realize automatic container landing of the container corner of the container. The deviation value between the container corner and the ground marking line can be calculated in real time, and then automatic container landing of the container corner is achieved.

Description

Technical Field

[0001] The present invention relates to the field of port hoisting equipment, and in particular to a method and system for automatically aligning container corners based on computer vision. Background Art

[0002] With the economic development of coastal cities, the port loading and unloading capacity continues to expand, and the lifting weight of lifting machinery is getting larger and larger, which puts higher requirements on energy consumption, efficiency and reliability. For quay crane loading and unloading equipment, how to achieve accurate container placement and stacking in the process of loading and unloading containers has always been a research topic that large port equipment pays great attention to in the application process. At present, when ports are carrying out container loading and unloading operations, operators manually control the spreader through the operating handle to align the container with the ground marking line to achieve accurate container placement. If the operator finds that the position deviation is large, it is necessary to manually control the spreader to place the container again. Therefore, manual container placement has the problems of low safety and low efficiency. . Summary of the invention

[0003] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method, system, electronic device, computer storage medium and computer program product for automatic alignment of container corners based on computer vision.

[0004] The present invention provides a method for automatically aligning container corners based on computer vision, the method comprising the following steps: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels; Control the camera of the spreader to shoot first image data downward, and use image processing technology to detect and identify the corners of the container and the ground markings; According to the conversion relationship, each corner of the container and the ground marking are converted into three-dimensional space, the offset between the container corner and the ground marking is calculated in the three-dimensional space, and the offset is connected to the spreader control system to realize automatic landing of the container corner.

[0005] Optionally, a spreader camera arranged on the container spreader is calibrated to obtain a conversion relationship between object points in three-dimensional space and camera image pixels, including: With the set known calibration object as a reference, the state of the internal structure of the sling camera during image acquisition, i.e., the first internal parameter and the first relationship between the first position and the first direction of the sling camera relative to the captured scene, are determined, and a conversion relationship between object points and camera image pixels in three-dimensional space is established based on the first relationship.

[0006] Optionally, the detecting and identifying the corners of the container and the ground markings using image processing technology includes: The first image data is processed by straight line detection, feature point detection, and feature point matching to identify the corners of the container and ground markings, and determine the image coordinates of the corners of the container and the ground markings.

[0007] Optionally, converting each corner of the container and the ground markings into three-dimensional space according to the conversion relationship includes: The second relationship of the second position and the second direction of the spreader camera relative to the captured scene is determined in combination with the spreader height and the projection relationship, and the three-dimensional coordinates of the image coordinates of each container corner and the ground marking line in the three-dimensional space are calculated according to the conversion relationship and the second relationship.

[0008] Optionally, before using image processing technology to detect and identify the corners of the container and the ground markings, the method further includes: Acquire second image data captured by the sling camera, and extract the position fluctuation frequency of the target object according to the second image data; The position fluctuation frequency is input into a lifting prediction model, and the lifting prediction model outputs a lifting state probability; When the lifting probability state is higher than the probability threshold, a detection signal is generated, and the detection signal is used to trigger the use of image processing technology to detect and identify the container corners and ground markings.

[0009] Optionally, the lifting prediction model is trained in the following manner: Publishing the data format of the lifting prediction model and training data elements to the consortium chain network; Other nodes in the alliance chain network generate training data sets according to the data format and perform distributed training on the lifting prediction model; The local node receives the key parameters of the hoisting prediction model after training fed back by other nodes, replaces the key parameters one by one into the local hoisting prediction model, uses the test data set to test the replaced hoisting prediction model, and determines the optimal key parameters and the corresponding hoisting prediction model according to the test results; The number of the other nodes is determined according to the average weight of the containers hoisted by the hoist. Specifically, the number of the other nodes is negatively correlated with the average weight.

[0010] The present invention also provides a computer vision-based container corner automatic alignment system, characterized in that: the system comprises a spreader camera and a processing module; the spreader camera is electrically connected to the processing module; The processing module is used to implement the following method steps: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels; Control the camera of the spreader to shoot first image data downward, and use image processing technology to detect and identify the corners of the container and the ground markings; According to the conversion relationship, each corner of the container and the ground marking are converted into three-dimensional space, the offset between the container corner and the ground marking is calculated in the three-dimensional space, and the offset is connected to the spreader control system to realize automatic landing of the container corner.

[0011] The present invention also provides an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute any of the methods described above.

[0012] The present invention also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is executed.

[0013] The present invention also provides a computer program product, comprising a computer program stored on a non-transitory computer-readable medium, wherein the computer program is executed by a processor to execute the method as described in any of the preceding items.

[0014] The present invention utilizes camera calibration technology, computer vision technology, nonlinear optimization and other technologies to automatically identify and detect the positions of container corners and ground markings in images, and restores their structures in three-dimensional space from images through stereo vision technology, thereby calculating the offset values ​​between container corners and ground markings in real time.

[0015] Since the present invention only needs to install an ordinary camera, the present invention can be applied to various container yards in various ports on the basis of high precision and can adapt to various on-site environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 The present invention is a flowchart of a method for automatically aligning container corners based on computer vision disclosed in an embodiment of the present invention.

[0018] Figure 2It is a schematic diagram of the training process of the lifting prediction model disclosed in the embodiment of the present invention.

[0019] Figure 3 The invention discloses a high-definition camera for construction site safety management. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0023] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0024] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0025] like Figure 1 As shown, an embodiment of the present invention discloses a method for automatically aligning container corners based on computer vision, and the method comprises the following steps: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels; Control the camera of the spreader to shoot first image data downward, and use image processing technology to detect and identify the corners of the container and the ground markings; According to the conversion relationship, each corner of the container and the ground marking are converted into three-dimensional space, the offset between the container corner and the ground marking is calculated in the three-dimensional space, and the offset is connected to the spreader control system to realize automatic landing of the container corner.

[0026] Compared with the manual container landing control method mentioned in the background technology, the present invention installs a spreader camera on the container spreader, identifies the container corners and ground markings in the image through image processing technology, and calculates the offset between the container corners and the ground markings through the conversion relationship between the image pixel points and the real-world points. The spreader control system automatically determines the correction control parameters based on the calculated offset, thereby realizing automatic landing of the container corners.

[0027] Among them, it is necessary to first calibrate the spreader camera to obtain the conversion relationship between object points and camera image pixels in three-dimensional space, and also convert the container corners and ground markings into three-dimensional space, and calculate the above offset based on the above conversion relationship in three-dimensional space.

[0028] Optionally, a spreader camera arranged on the container spreader is calibrated to obtain a conversion relationship between object points in three-dimensional space and camera image pixels, including: With the set known calibration object as a reference, the state of the internal structure of the sling camera during image acquisition, i.e., the first internal parameter and the first relationship between the first position and the first direction of the sling camera relative to the captured scene, are determined, and a conversion relationship between object points and camera image pixels in three-dimensional space is established based on the first relationship.

[0029] Since the position of the spreader camera on the spreader is fixed after installation, it only needs to be calibrated once after the spreader camera is installed, and there is no need to repeat the calibration multiple times. The calibration results are processed by the computer and stored as input for the subsequent part.

[0030] Optionally, the detecting and identifying the corners of the container and the ground markings using image processing technology includes: The first image data is processed by straight line detection, feature point detection, and feature point matching to identify the corners of the container and ground markings, and determine the image coordinates of the corners of the container and the ground markings.

[0031] The present invention solves the problem of container corner and ground marking recognition and detection under port environment conditions through image processing technology. The automatic recognition and detection of container corner points and ground marking corner points are realized through straight line detection, feature point detection, feature point matching and other technologies, and the image coordinates of container corner points and ground marking corner points in the image are obtained.

[0032] Optionally, converting each corner of the container and the ground markings into three-dimensional space according to the conversion relationship includes: The second relationship of the second position and the second direction of the spreader camera relative to the captured scene is determined in combination with the spreader height and the projection relationship, and the three-dimensional coordinates of the image coordinates of each container corner and the ground marking line in the three-dimensional space are calculated according to the conversion relationship and the second relationship.

[0033] Optionally, before using image processing technology to detect and identify the corners of the container and the ground markings, the method further includes: Acquire second image data captured by the sling camera, and extract the position fluctuation frequency of the target object according to the second image data; The position fluctuation frequency is input into a lifting prediction model, and the lifting prediction model outputs a lifting state probability; When the lifting probability state is higher than the probability threshold, a detection signal is generated, and the detection signal is used to trigger the use of image processing technology to detect and identify the container corners and ground markings.

[0034] Before using image processing technology to detect and identify the corners of containers and ground markings, it is necessary to determine whether the spreader is currently lifting the container. That is, only when the spreader is lifting the container will the image processing technology be triggered to detect and identify the corners of containers and ground markings. This setting can avoid invalid detection and identification of the corners of containers and ground markings when the spreader is not performing lifting operations, thereby reducing the data processing load.

[0035] Specifically, since the sling is in a rhythmic and slow shaking state when it is not hoisted, and the tension will increase sharply when the hoisting is started, resulting in high-frequency vibration, the sling camera can detect the position fluctuation frequency of the target object in the image accordingly. The present invention also provides a hoisting prediction model, which performs in-depth processing on the position fluctuation frequency of the target object in the second image data to obtain the probability of the hoisting state, that is, the probability that the sling is performing container hoisting. Among them, the second image data is obtained by the sling camera according to a preset detection cycle, which can be a multi-frame image or a video. Therefore, the present invention predicts whether the sling is currently performing a hoisting operation by analyzing the position fluctuation frequency of the target object (such as ground markings, other containers, etc.) in the second image data (that is, the position change frequency of the target object in the image, which can be the average value, median value, maximum value, etc. of the change frequency of multiple sub-periods within the detection period).

[0036] The lifting prediction model is preferably built based on ResNet (residual network). In traditional deep neural networks, if the number of network layers increases, the learning ability of the model will be stronger in theory, but in practice, a network that is too deep will cause the gradient to vanish or explode, making the network difficult to train. ResNet solves this problem by adding skip connections or shortcut connections, allowing signals in the network to bypass one or more layers and pass directly. The lifting prediction model includes multiple residual blocks, each of which contains two or more convolutional layers and a skip connection that adds the input of the block directly to the output of the block.

[0037] The format of the data elements in the training data set of the lifting prediction model is [position fluctuation frequency, lifting state label]. The lifting state label includes multiple different lifting state probabilities, which are used to describe the probability that the sling is in the lifting operation state under the associated position fluctuation frequency. The lifting prediction model is fully trained and tested using the above training data set until the prediction accuracy index is met.

[0038] Optionally, refer to Figure 2 As shown, the lifting prediction model is trained in the following way: Publishing the data format of the lifting prediction model and training data elements to the consortium chain network; Other nodes in the alliance chain network generate training data sets according to the data format and perform distributed training on the lifting prediction model; The local node receives the key parameters of the hoisting prediction model after training fed back by other nodes, replaces the key parameters one by one into the local hoisting prediction model, uses the test data set to test the replaced hoisting prediction model, and determines the optimal key parameters and the corresponding hoisting prediction model according to the test results; The number of the other nodes is determined according to the average weight of the containers hoisted by the hoist. Specifically, the number of the other nodes is negatively correlated with the average weight.

[0039] The present invention adopts a distributed training method to train the lifting prediction model. Specifically, the data format of the lifting prediction model and the corresponding training data elements are published to the alliance chain network, and other nodes in the alliance chain network generate training data sets for distributed training, that is, each other node independently generates training data sets and trains the model. After each other node completes the training, it feeds back the key parameters in the lifting prediction model to the local node. The key parameters include the weights, biases, batch normalization parameters (scaling factors, offsets), etc. of each convolutional layer and fully connected layer. The local node inputs the received key parameters one by one into the lifting prediction model and tests it using the local test data set. The optimal key parameters are selected according to the test results, and the key parameters are replaced in the local lifting prediction model to obtain the final model.

[0040] The number of other nodes participating in the distributed training of the model is determined according to the average weight of the container hoisted by the corresponding sling. The average weight of the container hoisted by the corresponding sling can be obtained through historical data statistics. The larger the average weight of the container hoisted by the corresponding sling, the greater the vibration frequency of the sling at the moment of the start of hoisting. At this time, only a small number of other nodes are needed to participate in the distributed training; and the smaller the average weight of the container hoisted by the corresponding sling, the smaller the vibration frequency of the sling at the moment of the start of hoisting. At this time, more other nodes are needed to participate in the distributed training, so as to improve the accurate recognition of the vibration frequency of the sling with a small amplitude (the corresponding position fluctuation frequency of the target object is also smaller). Therefore, the number of other nodes set in the present invention is negatively correlated with the average weight.

[0041] Reference Figure 3 As shown, the embodiment of the present invention also discloses a container corner automatic alignment system based on computer vision, characterized in that: the system comprises a spreader camera and a processing module; the spreader camera is electrically connected to the processing module; The processing module is used to implement the following method steps: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels; Control the camera of the spreader to shoot first image data downward, and use image processing technology to detect and identify the corners of the container and the ground markings; According to the conversion relationship, each corner of the container and the ground marking are converted into three-dimensional space, the offset between the container corner and the ground marking is calculated in the three-dimensional space, and the offset is connected to the spreader control system to realize automatic landing of the container corner.

[0042] An embodiment of the present invention further discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the method described in the above embodiment.

[0043] The embodiment of the present invention further discloses a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the above embodiment is executed.

[0044] An embodiment of the present invention further discloses a computer program product, including a computer program stored on a non-transitory computer-readable medium, wherein the computer program is executed by a processor to perform any of the methods described in the preceding items.

[0045] It should be noted that the storage module (102) in the second embodiment of the present invention, the memory in the third embodiment, and the computer storage medium in the fourth embodiment can all be, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, etc.

[0046] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0047] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0048] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0049] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for automatically aligning container corners based on computer vision, characterized in that: The method comprises the following steps: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels; Control the camera of the spreader to shoot first image data downward, and use image processing technology to detect and identify the corners of the container and the ground markings; According to the conversion relationship, each corner of the container and the ground marking are converted into three-dimensional space, the offset between the container corner and the ground marking is calculated in the three-dimensional space, and the offset is connected to the spreader control system to realize automatic landing of the container corner.

2. The method for automatically aligning container corners based on computer vision according to claim 1, characterized in that: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels, including: With the set known calibration object as a reference, the state of the internal structure of the sling camera during image acquisition, i.e., the first internal parameter and the first relationship between the first position and the first direction of the sling camera relative to the captured scene, are determined, and a conversion relationship between object points and camera image pixels in three-dimensional space is established based on the first relationship.

3. The method for automatically aligning container corners based on computer vision according to claim 2, characterized in that: The detection and recognition of the container corners and ground markings using image processing technology includes: The first image data is processed by straight line detection, feature point detection, and feature point matching to identify the corners of the container and ground markings, and determine the image coordinates of the corners of the container and the ground markings.

4. The method for automatically aligning container corners based on computer vision according to claim 3, characterized in that: The converting of each container corner and ground marking line into three-dimensional space according to the conversion relationship includes: The second relationship of the second position and the second direction of the spreader camera relative to the captured scene is determined in combination with the spreader height and the projection relationship, and the three-dimensional coordinates of the image coordinates of each container corner and the ground marking line in the three-dimensional space are calculated according to the conversion relationship and the second relationship.

5. The method for automatically aligning container corners based on computer vision according to claim 1, characterized in that: Before using the image processing technology to detect and identify the container corners and ground markings, the method further includes: Acquire second image data captured by the sling camera, and extract the position fluctuation frequency of the target object according to the second image data; The position fluctuation frequency is input into a lifting prediction model, and the lifting prediction model outputs a lifting state probability; When the lifting probability state is higher than the probability threshold, a detection signal is generated, and the detection signal is used to trigger the use of image processing technology to detect and identify the container corners and ground markings.

6. The method for automatically aligning container corners based on computer vision according to claim 5, characterized in that: The lifting prediction model is trained in the following way: Publishing the data format of the lifting prediction model and training data elements to the consortium chain network; Other nodes in the alliance chain network generate training data sets according to the data format and perform distributed training on the lifting prediction model; The local node receives the key parameters of the hoisting prediction model after training fed back by other nodes, replaces the key parameters one by one into the local hoisting prediction model, uses the test data set to test the replaced hoisting prediction model, and determines the optimal key parameters and the corresponding hoisting prediction model according to the test results; The number of the other nodes is determined according to the average weight of the containers hoisted by the hoist. Specifically, the number of the other nodes is negatively correlated with the average weight.

7. A computer vision-based container corner automatic alignment system, characterized by: The system includes a spreader camera and a processing module; the spreader camera is electrically connected to the processing module; The processing module is used to implement the following method steps: Calibrate the spreader camera installed on the container spreader to obtain the conversion relationship between object points in three-dimensional space and camera image pixels; Control the camera of the spreader to shoot first image data downward, and use image processing technology to detect and identify the corners of the container and the ground markings; According to the conversion relationship, each corner of the container and the ground marking are converted into three-dimensional space, the offset between the container corner and the ground marking is calculated in the three-dimensional space, and the offset is connected to the spreader control system to realize automatic landing of the container corner.

8. An electronic device comprising: A memory storing executable program code; A processor coupled to the memory; characterized in that: the processor calls the executable program code stored in the memory to execute the method according to any one of claims 1-6.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program stored on a non-transitory computer readable medium, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images