A box detection method and device based on machine vision

Abstract

The application discloses a kind of based on machine vision's box detection method and device, the method includes according to the box feature of container that monitoring device is gathered to judge the type of container, after determining the type of container, whether the matching of container and spreader is judged, if match, continue automatic hoisting;If not match, the operation mode of gantry crane is adjusted from automatic control to manual control, and the position of corner piece is detected in real time, according to the spatial coordinates of corner piece, the height variation of corner piece and horizontal offset value are calculated, if the height variation of corner piece and horizontal offset value are greater than preset threshold, send early warning information to control system, and stop the current hoisting operation.Using the above technical scheme, by monitoring the position of corner piece in real time, it can be judged whether hoisting is safe, and continue operation when it is determined that hoisting is safe, stop operation when it is determined that hoisting is unsafe, ensure safety, and improve the work efficiency of hoisting operation.

Description

Technical Field

[0001] This invention relates to the field of port automation equipment technology, and in particular to a machine vision-based container inspection method and device. Background Technology

[0002] With the gradual increase in port cargo throughput, higher requirements are now placed on the automation performance of port equipment. During the automated operation of gantry cranes in the yard, some container bodies are similar in size, such as double 20-foot containers and 40-foot containers with similar lengths, making them difficult to distinguish. This can easily lead to situations where the spreader for a 40-foot container grabs a double 20-foot container, potentially resulting in damage to the spreader corner fittings or the container falling off.

[0003] When automating port loading and unloading equipment, the safety requirements of the operating equipment cannot be ignored. Therefore, a container detection method that can identify container types in real time is needed to ensure the safety of the gantry crane lifting process. Summary of the Invention

[0004] In view of the fact that some container bodies are similar in size during the gantry crane lifting process, which can easily lead to mismatch between the lifting equipment and the container, thus affecting the lifting safety, the present invention proposes an embodiment to provide a container inspection method based on machine vision to overcome the above problems or at least partially solve the above problems, as well as a corresponding container inspection device based on machine vision.

[0005] To address the aforementioned problems, this invention discloses a machine vision-based box inspection method. This method is applied to gantry crane hoisting operations, where a monitoring device is installed on the side of the gantry crane's legs. The method includes:

[0006] Acquire image data of the container to be hoisted, the image data including the container body features collected by the monitoring device, the container body features including the position of the side corner pieces of the container, the number of the side corner pieces, and the characters on the side of the container;

[0007] The container body features are input into a preset container type detection model, and the identification result corresponding to the container to be lifted is obtained based on the number of corner pieces and the characters on the side of the container. The identification result includes whether the container to be lifted matches the spreader of the gantry crane or whether the container to be lifted does not match the spreader of the gantry crane.

[0008] If the container to be lifted does not match the spreader of the gantry crane, the operation mode of the gantry crane is changed from automatic control to manual control, and the position of the side corner pieces of the container is detected in real time, and the spatial coordinates of the corner pieces are calculated.

[0009] Calculate the height change and horizontal offset of the corner piece based on its spatial coordinates, and determine whether the height change and horizontal offset of the corner piece are greater than a preset threshold.

[0010] If the height change and horizontal offset of the corner piece exceed a preset threshold, an early warning message is sent to the control system, and the current hoisting operation is stopped.

[0011] Optionally, the method further includes:

[0012] If the height change and horizontal offset of the corner piece are less than or equal to a preset threshold, the gantry crane is manually controlled to lift the container and the position of the corner piece is monitored in real time.

[0013] Optionally, the monitoring device includes a camera, and the internal parameter K of the camera is calculated as follows:

[0014]

[0015] Where f represents the focal length of the camera, dx and dy represent the width of the image pixels in the x and y directions, respectively, and u0 and v0 represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system.

[0016] Set the center of the loading lane side door leg as the world coordinate origin O. c With the vertical front leg as the positive X-axis and the horizontal left side of the door leg as the positive Y-axis, and the coordinate system passing through the origin O... c Establish a world coordinate system W-XYZ with the point perpendicular to the ground pointing upwards as the positive Z-axis. o After obtaining the camera's internal parameters, the calibration plate is adjusted to be within the camera's field of view and placed vertically on the loading lane.

[0017] The real-time detection of the position of the container side corner fittings and the calculation of the spatial coordinates of the corner fittings include:

[0018] Based on the positions of several corner pieces on the calibration board, determine the pixel coordinates corresponding to each corner piece, where the pixel coordinate Z is... c The specific calculation formula is as follows:

[0019]

[0020] Among them, u n v n Let X be the pixel coordinates on the calibration board, K be the camera's intrinsic parameters, R and T be the camera's rotation and translation matrices, respectively, and X be the coordinates of the pixel points on the calibration board. n Y n Z nThese are the spatial coordinates of the corner point, calculated based on the world coordinate origin.

[0021] Based on the known dimensions of the corner piece and the pixel coordinates Z of the corner piece. c Establish a world coordinate origin O with the bottom right corner of the corner piece. l The vehicle's world coordinate system is W-XYZ, with the positive X-axis pointing forward perpendicular to the door leg, the positive Y-axis pointing to the left parallel to the lane line, and the positive Z-axis pointing upward perpendicular to the ground. l The lock hole on the corner piece has spatial coordinates (0, 0, 0) in the world coordinate system. The corner piece is located in the W-XYZ world coordinate system. l The spatial coordinates are: (0, 0, 0), (lock_w, 0, 0), (0, lock_h, 0), (lock_w, lock_h, 0), where lock_w and lock_h are the width and height of the corner piece, respectively.

[0022] Optionally, the container detection method is used to determine whether the container to be hoisted is a 40-foot container or a double 20-foot container. The character on the side of the 40-foot container is the first character, and the character on the double 20-foot container is the second character. The 40-foot container is provided with four corner pieces, and the double 20-foot container is provided with eight corner pieces.

[0023] The process of inputting the container's body features into a preset container type detection model and obtaining the identification result corresponding to the container to be hoisted based on the number of corner pieces and the characters on the side of the container includes:

[0024] According to preset rules, the number of detected corner pieces and the characters on the side of the container are assigned corresponding scores. The score corresponding to the number of corner pieces is the first score, and the score corresponding to the characters on the side of the container is the second score.

[0025] The type of container to be hoisted is determined based on the first and second scores, and the identification result is obtained.

[0026] Optionally, the step of assigning corresponding scores to the number of detected corner pieces and the characters on the side of the container according to preset rules includes:

[0027] If the number of detected corner pieces in the image data is greater than four, and the pixel distance between any two corner pieces is not greater than a preset corner piece distance threshold, the container is determined to be a double 20-foot container, and the first score is 0.7 points.

[0028] If the number of detected corner pieces in the image data is less than or equal to four, and the pixel distance between any two corner pieces is greater than a preset corner piece distance threshold, the container is determined to be a 40-foot container, and the first score is 0.

[0029] If the first character detected on the side of the container in the image data is the first character, it is determined that the container is a 40-foot container, and the second score is 0 points.

[0030] If the detected character on the side of the container in the image data is the second character, it is determined that the container is a double 20-foot container, and the second score is 0.3 points.

[0031] Optionally, the step of determining the type of the container to be hoisted based on the first score and the second score, and obtaining the identification result, includes:

[0032] After determining the first and second scores corresponding to the container to be hoisted based on the image data, the first and second scores are summed to obtain the third score corresponding to the container to be hoisted. The third score includes 0, 0.3, 0.7 and 1 points.

[0033] If the third score is 0, the container to be hoisted is determined to be a 40-foot container;

[0034] If the third score is 0.7 or 1, the container to be hoisted is determined to be a double 20-foot container.

[0035] If the third score is equal to 0.3 points, the operation mode of the gantry crane will be changed from automatic control to manual control, and the type of container to be lifted will be determined manually.

[0036] Optionally, the method further includes:

[0037] If the container to be lifted matches the spreader of the gantry crane, the operation of the gantry crane continues to be controlled in automatic mode, and the information on the matching of the container and the spreader is sent to the control system.

[0038] On the other hand, embodiments of the present invention disclose a machine vision-based box inspection device, which is applied to gantry crane hoisting operations. A monitoring device is installed on the side of the gantry crane leg. The device includes:

[0039] The image data acquisition module is used to acquire image data of the container to be hoisted. The image data includes the container body features acquired by the monitoring device. The container body features include the position of the side corner pieces of the container, the number of side corner pieces, and the characters on the side of the container.

[0040] The container type recognition module is used to input the container body features into a preset container body type detection model, and obtain the recognition result corresponding to the container to be lifted based on the number of corner pieces and the characters on the side of the container. The recognition result includes whether the container to be lifted matches the spreader of the gantry crane or whether the container to be lifted does not match the spreader of the gantry crane.

[0041] The corner component data processing module is used to switch the gantry crane's operation mode from automatic control to manual control if the container to be lifted does not match the spreader of the gantry crane, and to detect the position of the corner component on the side of the container in real time and calculate the spatial coordinates of the corner component.

[0042] The corner component change judgment module is used to calculate the height change and horizontal offset value of the corner component based on the spatial coordinates of the corner component, and to determine whether the height change and horizontal offset value of the corner component are greater than a preset threshold.

[0043] The hoisting control module is used to send an early warning message to the control system and stop the current hoisting operation if the height change and horizontal offset of the corner piece are greater than a preset threshold.

[0044] On the other hand, embodiments of the present invention also provide an electronic device, which includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the machine vision-based box detection method.

[0045] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the machine vision-based box detection method.

[0046] This invention discloses a container detection method and apparatus based on machine vision. The method includes determining the type of container to be hoisted based on the container characteristics collected by a monitoring device. These characteristics include the position of the container's side corner fittings, the number of corner fittings, and characters on the container's side. After determining the container type, the method checks whether the container to be hoisted matches the lifting equipment. If they match, automated hoisting continues. If they do not match, the gantry crane's operation mode is switched from automatic control to manual control, and the position of the corner fittings is detected in real time. The height change and horizontal offset of the corner fittings are calculated based on their spatial coordinates. If the height change and horizontal offset of the corner fittings exceed a preset threshold, an early warning message is sent to the control system, and the current hoisting operation is stopped.

[0047] By adopting the above technical solution, the technical solution of the present invention has the following technical effects:

[0048] (1) Determine the type of container to be lifted by the characteristics of the container body. If the container to be lifted matches the spreader, continue the automated lifting. If they do not match, change the gantry crane’s operation mode from automatic control to manual control and monitor the position of the corner parts in real time to ensure the safety of the lifting operation.

[0049] (2) If the container to be lifted does not match the lifting equipment, the position of the corner pieces can be monitored in real time to determine whether the lifting is safe. If the lifting is safe, the operation can continue. If the lifting is unsafe, the operation can be stopped. This ensures safety and improves the efficiency of the lifting operation. Attached Figure Description

[0050] Figure 1 A flowchart illustrating the steps of a machine vision-based box detection method provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the installation location of a monitoring device provided in an embodiment of the present invention;

[0052] Figure 3 A side view of a 40-foot box provided for an embodiment of the present invention;

[0053] Figure 4 A side view of a double 20-foot box provided in an embodiment of the present invention;

[0054] Figure 5 This is a structural block diagram of a box inspection device based on machine vision, provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] Figure 1 This invention provides a flowchart of a machine vision-based box inspection method. The method is applied to gantry crane lifting operations, where a monitoring device is installed on the side of the gantry crane legs. This monitoring device can be a camera, video camera, or other imaging equipment. The method includes the following steps:

[0057] Step 101: Obtain image data of the container to be hoisted. The image data includes the container body features collected by the monitoring device. The container body features include the position of the side corner pieces of the container, the number of side corner pieces, and the characters on the side of the container.

[0058] In the existing technology, the camera used to capture images of the container is generally set on the side of the spreader, so that the camera is in a downward position relative to the container. When working at night, the light will cast shadows on the surface of the container due to the obstruction of the spreader, which will affect the camera's shooting and recognition effect. Figure 2 This is a schematic diagram of the installation position of a monitoring device provided in an embodiment of the present invention. A camera 300 is installed on the side of the gantry crane leg 100. The container 200 and the loading and unloading lane 400 are both located between the gantry crane legs 100 on both sides. With this installation method, the camera 300 can easily take pictures of the container 200 and obtain relatively clear container features. This is beneficial for identifying the type of container based on the container features. It can effectively avoid the influence of changing ambient light during the day and reduce the influence of shadows cast on the container by the gantry crane's supplementary lighting at night due to the obstruction of the spreader. It avoids the problem of low container feature detection rate. In other words, this installation method avoids the impact of light and extreme weather (rain, snow) on the safety protection function.

[0059] During automated operations of gantry cranes in container yards, some containers have similar dimensions, such as double 20-foot containers and 40-foot containers with similar lengths, making them difficult to distinguish. This can easily lead to situations where the spreader for a 40-foot container grabs a double 20-foot container or vice versa, potentially resulting in damage to the spreader's corner fittings or the container falling off. Figure 3 This is a side view diagram of a 40-foot box provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the side structure of a double 20-foot container provided in an embodiment of the present invention. The 40-foot container has four corner pieces 201 and 40-foot side characters 202 and 203, while the double 20-foot container has eight corner pieces 211 and double 20-foot side characters 202 and 203. It should be noted that, for ease of distinction between the 40-foot and double 20-foot side characters, the characters on the side of the 40-foot container will be referred to as the first character, and the characters on the double 20-foot container as the second character. This application does not limit the specific content and form of the side characters. For example, 20-foot and 40-foot containers have different container type numbers, such as 22G1 representing a 20-foot standard height container, and 42G1 and 45G1 both representing 40-foot standard height containers. In actual inspection, double 20-foot and 40-foot containers can be distinguished based on their container type.

[0060] Step 102: Input the container body features into the preset container type detection model, and obtain the identification result corresponding to the container to be lifted based on the number of corner pieces and the characters on the side of the container. The identification result includes whether the container to be lifted matches the spreader of the gantry crane or whether the container to be lifted does not match the spreader of the gantry crane.

[0061] Before using a container type detection model to detect the container's features, the model needs to be trained in advance until it meets the design requirements. The training process may include: collecting video data of the container's side profile on-site, annotating the container's side features frame-by-frame using a labelImg tool, and training the model using the images and corresponding annotation data until the model meets the design requirements. After training, the model and its code are deployed to the gantry crane site, establishing TCP communication with the PLC. The container grabbing process is monitored in real-time by a monitoring device. If two 20-foot containers are detected while the 40-foot spreader is in operation, an alarm is triggered and feedback is sent to the PLC.

[0062] During the process of identifying the container to be hoisted based on the container type detection model, corresponding scores can be assigned to the number of detected corner pieces and the characters on the side of the container according to preset rules. The first score corresponds to the number of corner pieces, and the second score corresponds to the characters on the side of the container. The type of the container to be hoisted is determined based on the first and second scores, and the identification result is obtained.

[0063] For example, let's illustrate with two 20-foot containers and a 40-foot container. The determination is based on a combination of the number of detected corner pieces and the characters on the side of the container. The specific operating procedure is as follows:

[0064] 1. Container corner fittings. A 40-foot container has four corner fittings located at the four corners, while a double 20-foot container has eight corner fittings. The distribution of the corner fittings is as follows: Figure 3 and Figure 4 As shown, the number of detected corner pieces is used as the basis: if more than 4 corner pieces are detected, and the pixel distance between any two detected corner pieces is within the threshold range, it is considered a double 20-foot container, scoring 0.7 points; if 4 or fewer corner pieces are detected, and the pixel distance between any two corner pieces exceeds the threshold range, it is considered a single 40-foot container, scoring 0 points. Those skilled in the art can reasonably determine an appropriate threshold range based on the relative distances of corner pieces in a 40-foot container and in a double 20-foot container, thus facilitating the determination of container type based on the number of detected corner pieces and their relative distances.

[0065] 2. Container Type Number. 20-foot and 40-foot containers have different container type numbers. For example, 22G1 represents a 20-foot standard height container, and 42G1 represents a 40-foot standard height container. The container type number can be used to distinguish between double 20-foot and 40-foot containers.

[0066] It should be noted that because container type numbers are prone to wear and tear, their recognition accuracy is lower than that of corner parts, and therefore, they are assigned a lower score. After determining the first and second scores corresponding to the container to be lifted based on the image data, the first and second scores are added together to obtain a third score corresponding to the container to be lifted. The third score includes 0, 0.3, 0.7, and 1 points. If the third score is equal to 0 points, the container to be lifted is determined to be a 40-foot container; if the third score is 0.7 or 1 point, the container to be lifted is determined to be a double 20-foot container; if the third score is equal to 0.3 points, the gantry crane's operation mode is changed from automatic control to manual control, and the type of container to be lifted is manually determined.

[0067] Step 103: If the container to be lifted does not match the spreader of the gantry crane, change the operation mode of the gantry crane from automatic control to manual control, and detect the position of the side corner pieces of the container in real time, and calculate the spatial coordinates of the corner pieces;

[0068] The monitoring device includes cameras, all of which have undergone internal parameter calibration. Internal parameter calibration can be achieved using Zhang Dingyou's calibration method. The calculation method for the camera's internal parameter K is as follows:

[0069]

[0070] Where f represents the camera's focal length, dx and dy represent the width of the image pixels in the x and y directions, respectively, and u0 and v0 represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system. After calibrating the camera's intrinsic parameters K, the camera intrinsic parameter matrix determined by the camera hardware settings and the camera distortion coefficients D = [k1, k2, k3, k4, k5] can be obtained.

[0071] Set the center of the loading lane side door leg as the world coordinate origin O. c With the vertical front leg as the positive X-axis and the horizontal left side of the door leg as the positive Y-axis, and the coordinate system passing through the origin O... c Establish a world coordinate system W-XYZ with the point perpendicular to the ground pointing upwards as the positive Z-axis. o After acquiring the camera's internal parameters, the calibration plate is adjusted to be within the camera's field of view and placed perpendicular to the loading lane; where the positive XYZ directions are as follows: Figure 2 As shown.

[0072] Based on the positions of several corner pieces on the calibration board, determine the pixel coordinates corresponding to each corner piece, where the pixel coordinate Z is... c The specific calculation formula is as follows:

[0073]

[0074] Among them, u n v n Let X be the pixel coordinates on the calibration board, K be the camera's intrinsic parameters, R and T be the camera's rotation and translation matrices, respectively, and X be the coordinates of the pixel points on the calibration board. n Y n Z n These are the spatial coordinates of the corner point, calculated based on the world coordinate origin.

[0075] Determine the Z coordinates of multiple pixels corresponding to the corner piece. c Then, it can be determined by multiple pixel coordinates Z. c Obtain multiple relevant equations and calculate the camera extrinsic parameters Ex based on the PNP algorithm. c Camera extrinsic parameters Ex c Includes rotation matrix R o Translation matrix T o .

[0076] Step 104: Calculate the height change and horizontal offset of the corner piece based on its spatial coordinates, and determine whether the height change and horizontal offset of the corner piece are greater than a preset threshold.

[0077] Based on the known dimensions of the corner piece and the pixel coordinates Z of the corner piece. c Establish a world coordinate origin O with the bottom right corner of the corner piece. l The vehicle's world coordinate system is W-XYZ, with the positive X-axis pointing forward perpendicular to the door leg, the positive Y-axis pointing to the left parallel to the lane line, and the positive Z-axis pointing upward perpendicular to the ground. l The lock hole on the corner piece has spatial coordinates (0, 0, 0) in the world coordinate system. The corner piece is located in the W-XYZ world coordinate system. l The spatial coordinates are: (0, 0, 0), (lock_w, 0, 0), (0, lock_h, 0), (lock_w, lock_h, 0), where lock_w and lock_h are the width and height of the corner piece, respectively.

[0078] Based on the camera intrinsic parameter matrix K, distortion coefficients D, and the PNP algorithm for solving extrinsic parameters, the world coordinate system W-XYZ of the corner component is calculated. l With respect to the camera's world coordinate system W-XYZ c The transformation relationship, i.e., the external parameter R l T l Then, based on the external parameter R... l T l Set the corner piece's world coordinates (X) lc ,Y lc Z lc Transform to camera coordinate system (X) lp,Y lp Z lp The calculation relationship is as follows:

[0079]

[0080] According to external parameter R o T o The corner coordinates (X) in the camera coordinate system lp ,Y lp Z lp Transform to the large vehicle world coordinate system W-XYZ c (X) lo ,Y lo Z lo Solve for the spatial coordinate values ​​in this coordinate system. The transformation relationship is as follows:

[0081]

[0082] Based on the above transformation relationship, the center coordinates of the corner piece, as well as its height and horizontal offset in the world coordinate system of the vehicle, can be calculated.

[0083] Step 105: If the height change and horizontal offset of the corner piece are greater than the preset threshold, send an early warning message to the control system and stop the current hoisting operation.

[0084] It should be noted that those skilled in the art can set preset thresholds for the height change and horizontal offset of the corner pieces according to actual needs. This embodiment does not limit the specific preset thresholds. If the height change and horizontal offset of the corner pieces are less than or equal to the preset thresholds, the gantry crane is manually controlled to lift the equipment, and the position of the corner pieces on the side of the container is monitored in real time to ensure safety during the lifting process.

[0085] During container hoisting, the monitoring system maintains a constant state of activity, and the spatial coordinates of features such as corner pieces are calculated based on the camera's intrinsic and extrinsic parameter calibration results. The real-time monitoring screen displays changes in the height of container features; if the change exceeds a threshold, an alarm is continuously sent to the PLC to stop the current operation and prevent dangerous situations caused by operational errors. A monitoring camera is installed on the side of the gantry crane's leg, with its optical axis parallel to the ground and perpendicular to the container's side. The camera acquires container data, and deep learning methods are used to extract feature data, including corner pieces and side characters. Throughout the entire container lifting process, the system analyzes in real-time whether there is a risk of lifting two containers simultaneously. By adopting the above technical solution, the present invention has the following technical effects:

[0086] (1) Determine the type of container to be lifted by the characteristics of the container body. If the container to be lifted matches the spreader, continue the automated lifting. If they do not match, change the gantry crane’s operation mode from automatic control to manual control and monitor the position of the corner parts in real time to ensure the safety of the lifting operation.

[0087] (2) If the container to be lifted does not match the lifting equipment, the position of the corner pieces can be monitored in real time to determine whether the lifting is safe. If the lifting is safe, the operation can continue. If the lifting is unsafe, the operation can be stopped. This ensures safety and improves the efficiency of the lifting operation.

[0088] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0089] To implement the aforementioned machine vision-based box detection method, Figure 5 This invention provides a structural block diagram of a machine vision-based box inspection device. The device is applied to gantry crane lifting operations, and a monitoring device is installed on the side of the gantry crane leg. The device includes:

[0090] The image data acquisition module 501 is used to acquire image data of the container to be hoisted. The image data includes the container body features acquired by the monitoring device. The container body features include the position of the side corner pieces of the container, the number of side corner pieces, and the characters on the side of the container.

[0091] The container type identification module 502 is used to input the container body features into a preset container body type detection model, and obtain the identification result corresponding to the container to be lifted based on the number of corner pieces and the characters on the side of the container. The identification result includes whether the container to be lifted matches the spreader of the gantry crane or whether the container to be lifted does not match the spreader of the gantry crane.

[0092] The corner component data processing module 503 is used to switch the gantry crane's operation mode from automatic control to manual control if the container to be lifted does not match the lifting device of the gantry crane, and to detect the position of the corner component on the side of the container in real time and calculate the spatial coordinates of the corner component.

[0093] The corner component change judgment module 504 is used to calculate the height change and horizontal offset value of the corner component based on the spatial coordinates of the corner component, and to determine whether the height change and horizontal offset value of the corner component are greater than a preset threshold.

[0094] The first hoisting control module is used to send an early warning message to the control system and stop the current hoisting operation if the height change and horizontal offset of the corner piece are greater than a preset threshold.

[0095] In an optional embodiment, the device further includes:

[0096] The second hoisting control module is used to manually control the hoisting operation of the gantry crane if the height change and horizontal offset of the corner piece are less than or equal to a preset threshold, and to monitor the position of the container side corner piece in real time.

[0097] In one optional embodiment, the monitoring device includes a camera, and the internal parameter K of the camera is calculated as follows:

[0098]

[0099] Where f represents the focal length of the camera, dx and dy represent the width of the image pixels in the x and y directions, respectively, and u0 and v0 represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system.

[0100] Set the center of the loading lane side door leg as the world coordinate origin O. c With the vertical front leg as the positive X-axis and the horizontal left side of the door leg as the positive Y-axis, and the coordinate system passing through the origin O... c Establish a world coordinate system W-XYZ with the point perpendicular to the ground pointing upwards as the positive Z-axis. o After obtaining the camera's internal parameters, the calibration plate is adjusted to be within the camera's field of view and placed vertically on the loading lane.

[0101] The corner component data processing module 503 may include:

[0102] The first pixel coordinate calculation submodule is used to determine the pixel coordinates corresponding to each corner piece based on the positions of several corner pieces on the calibration board. The pixel coordinate Z is... c The specific calculation formula is as follows:

[0103]

[0104] Among them, u n v n Let X be the pixel coordinates on the calibration board, K be the camera's intrinsic parameters, R and T be the camera's rotation and translation matrices, respectively, and X be the coordinates of the pixel points on the calibration board. n Y n Zn These are the spatial coordinates of the corner point, calculated based on the world coordinate origin.

[0105] The second pixel coordinate calculation submodule is used to calculate the pixel coordinates Z of the corner piece based on the known size of the corner piece. c Establish a world coordinate origin O with the bottom right corner of the corner piece. l The vehicle's world coordinate system is W-XYZ, with the positive X-axis pointing forward perpendicular to the door leg, the positive Y-axis pointing to the left parallel to the lane line, and the positive Z-axis pointing upward perpendicular to the ground. l The lock hole on the corner piece has spatial coordinates (0, 0, 0) in the world coordinate system. The corner piece is located in the W-XYZ world coordinate system. l The spatial coordinates are: (0, 0, 0), (lock_w, 0, 0), (0, lock_h, 0), (lock_w, lock_h, 0), where lock_w and lock_h are the width and height of the corner piece, respectively.

[0106] The second pixel coordinate calculation submodule is used to calculate the corner component's world coordinate system W-XYZ based on the camera intrinsic parameter matrix K, distortion coefficients D, and the PNP algorithm used to solve for the extrinsic parameters. l With respect to the camera's world coordinate system W-XYZ c The transformation relationship, i.e., the external parameter R l T l Then, based on the external parameter R... l T l Set the corner piece's world coordinates (X) lc ,Y lc Z lc Transform to camera coordinate system (X) lp ,Y lp Z lp The calculation relationship is as follows:

[0107]

[0108] According to external parameter R o T o The corner coordinates (X) in the camera coordinate system lp ,Y lp Z lp Transform to the large vehicle world coordinate system W-XYZ c (X) lo ,Y lo Z lo Solve for the spatial coordinate values ​​in this coordinate system. The transformation relationship is as follows:

[0109]

[0110] Based on the above transformation relationship, the center coordinates of the corner piece, as well as its height and horizontal offset in the world coordinate system of the vehicle, can be calculated.

[0111] In one optional embodiment, the container detection device is used to determine whether the container to be hoisted is a 40-foot container or a double 20-foot container. The character on the side of the 40-foot container is the first character, and the character on the double 20-foot container is the second character. The 40-foot container is provided with four corner pieces, and the double 20-foot container is provided with eight corner pieces.

[0112] The container type identification module 502 may include:

[0113] The first type of identification submodule is used to assign corresponding scores to the number of detected corner pieces and the characters on the side of the container according to preset rules. The score corresponding to the number of corner pieces is the first score, and the score corresponding to the characters on the side of the container is the second score.

[0114] The second type identification submodule is used to determine the type of the container to be hoisted based on the first score and the second score, and to obtain the identification result.

[0115] In one alternative embodiment, the first type identification submodule may include:

[0116] The first scoring unit is used to determine that the container is a double 20-foot container when the number of detected corner pieces in the image data is greater than four and the pixel distance between any two corner pieces is not greater than a preset corner piece distance threshold. The first score is 0.7 points.

[0117] The second scoring unit is used to determine that the container is a 40-foot container when the number of detected corner pieces in the image data is less than or equal to four, and the pixel distance between any two corner pieces is greater than a preset corner piece distance threshold, and the first score is 0.

[0118] The third scoring unit is used to determine that the container is a 40-foot container when the detected character on the side of the container in the image data is the first character, and the second score is 0 points.

[0119] The fourth scoring unit is used to determine that the container is a double 20-foot container when the character detected on the side of the container in the image data is the second character, and the second score is 0.3 points.

[0120] In one optional embodiment, the second type identification submodule includes:

[0121] The fifth scoring unit is used to sum the first and second scores corresponding to the container to be hoisted after determining the first score and the second score based on the image data, and to obtain the third score corresponding to the container to be hoisted. The third score includes 0 points, 0.3 points, 0.7 points and 1 point.

[0122] The first container type determination unit is used to determine that the container to be hoisted is a 40-foot container if the third score is equal to 0.

[0123] The second container type determination unit is used to determine that the container to be hoisted is a double 20-foot container if the third score is 0.7 or 1.

[0124] The third container type determination unit is used to adjust the operation mode of the gantry crane from automatic control to manual control if the third score is equal to 0.3 points, and to manually determine the type of container to be lifted.

[0125] In an optional embodiment, the device further includes:

[0126] The third lifting control module is used to continue controlling the operation of the gantry crane in automatic control mode if the container to be lifted matches the spreader of the gantry crane, and to send the information of container matching with spreader to the control system.

[0127] This invention discloses a container inspection device based on machine vision. The device includes determining the type of container to be lifted based on the container characteristics collected by a monitoring device. The container characteristics include the position of the container's side corner pieces, the number of the corner pieces, and the characters on the container's side. After determining the container type, the device checks whether the container to be lifted matches the lifting equipment. If they match, the automatic lifting continues; if they do not match, the gantry crane's operation mode is switched from automatic control to manual control, and the position of the corner pieces is detected in real time. The device calculates the height change and horizontal offset of the corner pieces based on their spatial coordinates. If the height change and horizontal offset of the corner pieces exceed a preset threshold, an early warning message is sent to the control system, and the current lifting operation is stopped.

[0128] By adopting the above technical solution, the technical solution of the present invention has the following technical effects:

[0129] (1) Determine the type of container to be lifted by the characteristics of the container body. If the container to be lifted matches the spreader, continue the automated lifting. If they do not match, change the gantry crane’s operation mode from automatic control to manual control and monitor the position of the corner parts in real time to ensure the safety of the lifting operation.

[0130] (2) If the container to be lifted does not match the lifting equipment, the position of the corner pieces can be monitored in real time to determine whether the lifting is safe. If the lifting is safe, the operation can continue. If the lifting is unsafe, the operation can be stopped. This ensures safety and improves the efficiency of the lifting operation.

[0131] On the other hand, embodiments of the present invention also provide an electronic device, which includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the machine vision-based box detection method.

[0132] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the machine vision-based box detection method.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0134] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0140] This document uses specific examples to illustrate the principles and implementation methods of the present invention. For those skilled in the art, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A machine vision-based box inspection method, characterized in that, The method is applied to gantry crane hoisting operations, wherein a monitoring device is installed on the side of the gantry crane leg, and the method includes: Acquire image data of the container to be hoisted, the image data including the container body features collected by the monitoring device, the container body features including the position of the side corner pieces of the container, the number of the side corner pieces, and the characters on the side of the container; The container body features are input into a preset container type detection model, and the identification result corresponding to the container to be lifted is obtained based on the number of corner pieces and the characters on the side of the container. The identification result includes whether the container to be lifted matches the spreader of the gantry crane or whether the container to be lifted does not match the spreader of the gantry crane. If the container to be lifted does not match the spreader of the gantry crane, the operation mode of the gantry crane is changed from automatic control to manual control, and the position of the side corner pieces of the container is detected in real time, and the spatial coordinates of the corner pieces are calculated. Calculate the height change and horizontal offset of the corner piece based on its spatial coordinates, and determine whether the height change and horizontal offset of the corner piece are greater than a preset threshold. If the height change and horizontal offset of the corner piece exceed a preset threshold, an early warning message is sent to the control system, and the current hoisting operation is stopped. The container detection method is used to determine whether the container to be hoisted is a 40-foot container or a double 20-foot container. The character on the side of the 40-foot container is the first character, and the character on the double 20-foot container is the second character. The 40-foot container is equipped with four corner pieces, and the double 20-foot container is equipped with eight corner pieces. The process of inputting the container's body features into a preset container type detection model and obtaining the identification result corresponding to the container to be hoisted based on the number of corner pieces and the characters on the side of the container includes: According to preset rules, the number of detected corner pieces and the characters on the side of the container are assigned corresponding scores. The score corresponding to the number of corner pieces is the first score, and the score corresponding to the characters on the side of the container is the second score. The type of container to be hoisted is determined based on the first and second scores, and the identification result is obtained.

2. The enclosure detection method according to claim 1, characterized in that, The method further includes: If the height change and horizontal offset of the corner piece are less than or equal to a preset threshold, the gantry crane is manually controlled to lift the container and the position of the corner piece is monitored in real time.

3. The enclosure detection method according to claim 1, characterized in that, The monitoring device includes a camera, and the internal parameter K of the camera is calculated as follows: ， Where f represents the camera's focal length, and dx and dy represent the width of the image pixels in the x and y directions, respectively. , These represent the coordinates of the center of the camera's image sensor in the pixel coordinate system; Set the center of the loading lane side door leg as the world coordinate origin. With the vertical front leg as the positive X-axis and the horizontal left side of the door leg as the positive Y-axis, and the coordinate system passing through the origin... Establish a world coordinate system with the point perpendicular to the ground pointing upwards as the positive Z-axis. After obtaining the camera's internal parameters, the calibration plate is adjusted to be within the camera's field of view and placed vertically on the loading lane. The real-time detection of the position of the container side corner fittings and the calculation of the spatial coordinates of the corner fittings include: Based on the positions of several corner pieces on the calibration board, the pixel coordinates corresponding to each corner piece are determined. The specific formula for calculating the pixel coordinate Zc is as follows: ， in, , For the pixel coordinates on the calibration board, These are the camera's internal parameters. , These are the rotation and translation matrices corresponding to the camera. , , These are the spatial coordinates of the corner points, calculated based on the world coordinate origin. Based on the known dimensions of the corner piece and its pixel coordinates Zc, a world coordinate system is established with the bottom right corner of the corner piece as the origin. The vehicle's world coordinate system is defined as follows: the positive X-axis is perpendicular to the door leg pointing forward, the positive Y-axis is parallel to the lane line pointing to the left, and the positive Z-axis is perpendicular to the ground pointing upward. The spatial coordinates of the lock hole on the corner piece in the world coordinate system are (0, 0, 0). The spatial coordinates are: (0, 0, 0), (lock_w, 0, 0), (0, lock_h, 0), (lock_w, lock_h, 0), where lock_w and lock_h are the width and height of the corner piece, respectively.

4. The enclosure detection method according to claim 1, characterized in that, The process of assigning corresponding scores to the number of detected corner pieces and the characters on the side of the container according to preset rules includes: If the number of detected corner pieces in the image data is greater than four, and the pixel distance between any two corner pieces is not greater than a preset corner piece distance threshold, the container is determined to be a double 20-foot container, and the first score is 0.7 points. If the number of detected corner pieces in the image data is less than or equal to four, and the pixel distance between any two corner pieces is greater than a preset corner piece distance threshold, the container is determined to be a 40-foot container, and the first score is 0. If the first character detected on the side of the container in the image data is the first character, it is determined that the container is a 40-foot container, and the second score is 0. If the detected character on the side of the container in the image data is the second character, it is determined that the container is a double 20-foot container, and the second score is 0.3 points.

5. The enclosure detection method according to claim 4, characterized in that, The process of determining the type of container to be hoisted based on the first and second scores, and obtaining the identification result, includes: After determining the first and second scores corresponding to the container to be hoisted based on the image data, the first and second scores are summed to obtain the third score corresponding to the container to be hoisted. The third score includes 0, 0.3, 0.7 and 1 points. If the third score is 0, the container to be hoisted is determined to be a 40-foot container; If the third score is 0.7 or 1, the container to be hoisted is determined to be a double 20-foot container. If the third score is equal to 0.3 points, the operation mode of the gantry crane will be changed from automatic control to manual control, and the type of container to be lifted will be determined manually.

6. The enclosure detection method according to claim 1, characterized in that, The method further includes: If the container to be lifted matches the spreader of the gantry crane, the operation of the gantry crane continues to be controlled in automatic mode, and the information on the matching of the container and the spreader is sent to the control system.

7. A machine vision-based box inspection device, used to perform a machine vision-based box inspection method as described in any one of claims 1-6, characterized in that, The device is used in gantry crane hoisting operations, and a monitoring device is installed on the side of the gantry crane leg. The device includes: The image data acquisition module is used to acquire image data of the container to be hoisted. The image data includes the container body features acquired by the monitoring device. The container body features include the position of the side corner pieces of the container, the number of side corner pieces, and the characters on the side of the container. The container type recognition module is used to input the container body features into a preset container body type detection model, and obtain the recognition result corresponding to the container to be lifted based on the number of corner pieces and the characters on the side of the container. The recognition result includes whether the container to be lifted matches the spreader of the gantry crane or whether the container to be lifted does not match the spreader of the gantry crane. The corner component data processing module is used to switch the gantry crane's operation mode from automatic control to manual control if the container to be lifted does not match the spreader of the gantry crane, and to detect the position of the corner component on the side of the container in real time and calculate the spatial coordinates of the corner component. The corner component change judgment module is used to calculate the height change and horizontal offset value of the corner component based on the spatial coordinates of the corner component, and to determine whether the height change and horizontal offset value of the corner component are greater than a preset threshold. The first hoisting control module is used to send an early warning message to the control system and stop the current hoisting operation if the height change and horizontal offset of the corner piece are greater than a preset threshold.

8. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the method as described in any one of claims 1-6.

Images