Container door detection method and system, field bridge and electronic equipment
By obtaining a top view of the container before the spreader grabs it and using preset key point detection to construct the container skeleton, the problem of misjudgment in the orientation and status of the container door is solved, accurate door orientation and status detection is achieved, and computing resources are saved.
Patent Information
- Application Number
- CN202510858503.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-24
- Publication Date
- 2025-10-17
AI Technical Summary
Existing technologies have misjudgment problems in identifying the orientation and status of container doors. In particular, when there is a bulge in the container or the connection between the door and the container body is loose, radar scanning and image recognition methods cannot accurately identify the orientation and status of the door.
By obtaining a top view of the container before the spreader grabs it, the box skeleton is constructed using preset key point detection. The orientation and status of the door are determined based on the box skeleton, including detecting the four top corners of the container, the vertex of the door groove, the top corner of the unrotated end and the center point of the upper surface. The box skeleton is formed through distance constraint association processing, and the orientation and status of the door are accurately identified.
The system achieves accurate simultaneous detection of the door's orientation and status, avoids misjudgment caused by confusion of image features in existing technologies, saves computing resources, and improves detection efficiency.
Smart Images
Figure CN120808000A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and more particularly, to a container door detection method and system, a yard crane, and an electronic device. BACKGROUND
[0002] Currently, in the process of port automation operation, the orientation and state of the container door need to be identified to facilitate better completion of loading and unloading tasks.
[0003] Currently, the orientation of the container door is mainly identified by scanning the container body based on radar or by identifying the image of the container. However, in the process of scanning the container by radar, the defects of the container itself will affect the judgment of the orientation of the container door, for example, in the case of a bulge on the container, the position of the bulge will also be considered as the container door. By identifying the image of the container, there is also a problem that the appearance features of the container door and the container tail cannot be distinguished, which leads to the inability to identify the orientation of the container door. In addition, when the connection between the container door and the container body is loose, and there is a certain gap between the container door and the container body when the container door is closed, the gap will also cause the state of the container door to be misjudged as open.
[0004] Therefore, how to accurately identify the orientation and state of the container door is a technical problem that needs to be solved by those skilled in the art. SUMMARY
[0005] Therefore, the embodiments of the present application aim to provide a container door detection method and system, a yard crane, and an electronic device to solve the problem that the orientation and state of the container door cannot be accurately identified based on radar or existing image recognition methods.
[0006] In a first aspect, the present application provides a container door detection method, comprising:
[0007] obtaining a target image, the target image being a top view of a container taken by a camera arranged on a spreader before the spreader grabs the container;
[0008] performing a preset key point detection process on the target image to obtain a preset key point for representing the container body structure of the container;
[0009] associating the preset key points through distance constraints to obtain a container body skeleton of the container;
[0010] determining the orientation and state of the container door of the container based on the container body skeleton.
[0011] In a possible implementation, the preset key points include a first key point, a second key point, a third key point, and a fourth key point.
[0012] wherein the first key point is a vertex representing a top corner of the container;
[0013] the second key point is a vertex representing two top corners of a groove on the container for mounting the door along a width direction of the container;
[0014] the third key point is a vertex representing a top corner not connected to the container along a width direction of the door;
[0015] the fourth key point is a midpoint representing a center of an upper surface of the container.
[0016] In a possible implementation, when the preset key points obtained by performing a preset key point detection process on the target image include the first key point, the second key point, and the fourth key point, the preset key points are associated by distance constraint to obtain a container body skeleton, including:
[0017] based on pixel coordinates of the preset key points in the target image, distances between each two of the preset key points are calculated as first distances;
[0018] two first line segments are determined in ascending order of the first distances, the first line segment being a line segment with the second key point and the first key point closest to the second key point as end points;
[0019] distances between the first line segment and each of the preset key points are determined, the preset key point closest to the first line segment is determined as the fourth key point, and the fourth key point is connected to two first key points not included in the first line segment to obtain a second line segment;
[0020] distances between the fourth key point and end points of the first line segment are determined, the two preset key points closest to the fourth key point in the first line segment are determined as the second key point, and the second key point is connected to the fourth key point to obtain a third line segment;
[0021] the first line segment, the second line segment, and the third line segment are used to form the container body skeleton.
[0022] In a possible implementation, the orientation and state of the door are determined based on the container body skeleton, including:
[0023] the first line segment side is determined as the orientation of the door based on the container body skeleton, and the state of the door is determined as closed.
[0024] In one possible implementation, when preset key point detection processing is performed on the target image, and the obtained preset key points include: the first key point, the second key point, the third key point, and the fourth key point, the associating processing of the preset key points using a distance constraint to obtain the container skeleton includes:
[0025] Based on the pixel coordinates of the preset key points in the target image, calculating the distances between all two of the preset key points as second distances;
[0026] Determine two fourth line segments according to the second distances in ascending order, where the fourth line segments are line segments having the second key point and the first key point closest to the second key point as endpoints;
[0027] determining the distances between the fourth line segment and each of the preset key points, determining the preset key point closest to the fourth line segment as the third key point, and determining the preset key point second closest to the fourth line segment as the fourth key point;
[0028] Connect the second key points to the fourth key points respectively to obtain a fifth line segment;
[0029] determining the distance between the second key point and the remaining preset key points, determining the preset key point closest to the second key point among the remaining preset key points as the third key point, and connecting the third key point to the second key point closest to the third key point to obtain a sixth line segment;
[0030] Connecting the fourth key point to two of the first key points not included in the fourth line segment, respectively, to obtain a seventh line segment;
[0031] The box skeleton is constructed based on the fourth line segment, the fifth line segment, the sixth line segment, and the seventh line segment.
[0032] In a possible implementation, determining the orientation and state of the door based on the box frame includes:
[0033] Based on the box frame, determining the fourth line segment side as the direction of the box door;
[0034] Connect the two second key points to obtain an eighth line segment;
[0035] determining an angle between the sixth line segment and the eighth line segment;
[0036] Based on the relationship between the included angle and the preset angle, it is determined whether the state of the door is open.
[0037] In a possible implementation, the determining whether the state of the container door is open based on the relationship between the included angle and the preset angle comprises:
[0038] when the included angle is less than or equal to the preset angle, determining that the state of the container door is closed;
[0039] when the included angle is greater than the preset angle, determining that the state of the container door is open, and taking the included angle as an opening angle of the container door.
[0040] In a second aspect, the present application provides a container door detection system, comprising:
[0041] an image acquisition unit configured to acquire a target image, the target image being a top view of a container captured by a camera arranged on a spreader before the spreader grabs the container;
[0042] a key point detection unit configured to perform a preset key point detection process on the target image to obtain preset key points representing a box structure of the container;
[0043] a skeleton construction unit configured to associate the preset key points through distance constraint to obtain a box skeleton of the container;
[0044] a container door detection unit configured to determine an orientation and a state of a container door of the container based on the box skeleton.
[0045] In a third aspect, the present application provides a yard crane, comprising a spreader, a camera and a controller.
[0046] The spreader is configured to lift a container during container transfer;
[0047] The camera is arranged on the spreader and configured to capture an image of the container before the spreader grabs the container to obtain a top view of the container as a target image;
[0048] The controller is configured to acquire the target image, perform a preset key point detection process on the target image to obtain preset key points representing a box structure of the container, and associate the preset key points through distance constraint to obtain a box skeleton of the container, so as to determine an orientation and a state of a container door of the container based on the box skeleton, and execute a preset strategy when the state of the container door is determined to be open.
[0049] In a fourth aspect, the present application provides an electronic device, comprising:
[0050] a processor;
[0051] a memory for storing the processor-executable instructions;
[0052] the processor is configured to execute the method provided in the first aspect of the application.
[0053] According to the container door detection method provided by the application, by acquiring a target image, i.e. a top view of the container captured by a camera arranged on the spreader before the spreader grabs the container, performing preset key point detection processing on the target image, obtaining preset key points for characterizing the container body structure, then performing correlation processing on the obtained preset key points through distance constraint, obtaining the container body skeleton, and determining the orientation and state of the container door based on the container body skeleton, the orientation and state of the container door can be determined. On the one hand, by processing the top view of the container captured before the spreader grabs the container, the problem that the container door orientation cannot be recognized because the appearance features of the container door and the container tail are the same on the image based on the image captured after the spreader grabs the container is solved. On the other hand, by performing key point detection on the image and constructing the container body skeleton for the detection of the container door orientation and state, the problem of misjudgment of the container door state caused by the detection based on the image recognition on the image is avoided, thereby improving the accuracy of the detection of the container door orientation and state, and realizing the simultaneous detection of the container door orientation and state, and saving the operation resources. BRIEF DESCRIPTION OF DRAWINGS
[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.
[0055] Figure 1 Fig. 1 shows a flowchart of a container door detection method provided by an embodiment of the present application;
[0056] Figure 2 Fig. 2 shows a top view of a container whose door is in a closed state as a target image provided by an embodiment of the present application;
[0057] Figure 3 Fig. 3 shows a preset key point distribution diagram of a container example whose door is in a closed state provided by an embodiment of the present application;
[0058] Figure 4 Fig. 4 shows a top view of a container example whose one door is in an open state provided by an embodiment of the present application;
[0059] Figure 5 Fig. 5 shows a top view of a container example whose two doors are in an open state provided by an embodiment of the present application;
[0060] Figure 6 Fig. 4 shows a preset key point distribution diagram corresponding to a top view of Figure 4 Fig. 5 shows a preset key point distribution diagram corresponding to a top view of
[0061] Figure 7 Fig. 6 shows a preset key point distribution diagram corresponding to a top view of Figure 5 Fig. 7 shows a preset key point distribution diagram corresponding to a top view of
[0062] Figure 8 Fig. 8 shows a schematic diagram of a box skeleton corresponding to Figure 3 Fig. 9 shows a schematic diagram of a box skeleton corresponding to
[0063] Figure 9 Fig. 10 shows a schematic diagram of a box skeleton corresponding to Figure 6 Fig. 11 shows a schematic diagram of a box skeleton corresponding to
[0064] Figure 10 Fig. 12 shows a schematic diagram of a box skeleton corresponding to Figure 7 Fig. 13 shows a schematic diagram of a box skeleton corresponding to
[0065] Figure 11 Fig. 14 shows a structural diagram of a container door detection system according to an embodiment of the present application;
[0066] Figure 12 Fig. 15 shows a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0067] Unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application shall have the same meaning as understood by a person of ordinary skill in the art to which the embodiments of the present application belong. The terms “first”, “second”, and the like used in the embodiments of the present application do not indicate any order, number, or importance, but are used to avoid confusion of the constituent elements.
[0068] Unless otherwise required by the context, throughout the specification, “plurality” means “at least two”, and “comprising” is interpreted to be open, inclusive meaning, i.e., “including, but not limited to”. In the description of the specification, the terms “one embodiment”, “some embodiments”, “exemplary embodiments”, “example”, “specific example”, or “some examples” are intended to mean that the specific feature, structure, material, or characteristic related to the embodiment or example is included in at least one embodiment or example of the specification. The illustrative representation of the above terms does not necessarily mean the same embodiment or example.
[0069] The technical solutions in the embodiments of the present specification will be described clearly and completely in the following with reference to the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present specification.
[0070] At present, there are many methods for detecting the orientation and opening and closing state of the container door during the container transfer process, but there are some common problems. For example, when the container door is detected based on image recognition, the image of the container is usually shot after the spreader grabs the container, and then the container door is detected based on the image. However, the image shot after the spreader grabs the container is easily affected by the installation position of the camera, so that the appearance features of the container door and the container tail shown in the shot image are the same, thereby causing the container door orientation to be unable to be recognized. For another example, the orientation of the container door and the opening and closing state of the container door need to be detected by two methods respectively, without simultaneous recognition, and without camera reuse, which not only is complex but also occupies operation resources. For another example, when the connection between the container door and the container body is loose, because the container door is closed and there is a gap between the container door and the container body, when based on image recognition, the container door is easily misjudged as being opened.
[0071] The present application aims to solve the above problems, by acquiring an overhead view of the container shot before the spreader grabs the container as an image, using the structural features of the container, constructing the container body skeleton through preset key point detection, thereby realizing the simultaneous and accurate detection of the orientation and state of the container door through the container body skeleton, and further solving the problem that the orientation and state of the container door cannot be simultaneously detected by the prior art, and the result is not accurate enough.
[0072] The container door detection method provided by the embodiments of the present application is executed on an electronic device, which can be a controller of a spreader, or a smart terminal device such as a notebook computer, a mobile phone, a tablet computer, etc., and of course can also be a server remotely connected with the controller of the spreader.
[0073] Based on the above content, referring to Figure 1 , Figure 1 is a flowchart of a container door detection method provided by the embodiments of the present application, as shown in Figure 1 , which mainly includes the following steps:
[0074] S100, acquiring a target image.
[0075] Among them, the target image is an overhead view of the container shot based on the camera arranged on the spreader before the spreader grabs the container.
[0076] It can be understood that the containers currently used are generally rectangular, and a recess is arranged on one side of the container head, and a container door is arranged on the recess. Therefore, in view of the structural features of the container currently used, the difference between the side of the container door and the side of the container tail can be seen from the top view of the container, that is, the target image. Figure 2
[0077] Further, when performing port automation operation, a spreader needs to be used to grab and transfer the container. Before the spreader grabs the container, the spreader is located above the container. Therefore, before the spreader grabs the container, the top view of the container can be captured by the camera arranged on the spreader.
[0078] In some optional embodiments, the target image is preferably captured by the four-corner camera of the spreader.
[0079] S110, performing preset key point detection processing on the target image to obtain preset key points for characterizing the container body structure.
[0080] It should be noted that the detection processing of the preset key points can adopt any method capable of detecting key points, for example, by collecting the top view of the container as sample data to train a model constructed based on a deep learning algorithm, and then performing preset key point detection processing on the target image by using the trained model.
[0081] It should be noted that the method provided in the embodiment is used to construct the container body skeleton by using the preset key points. Therefore, the preset key points should be points capable of exhibiting the structural features of the container body, and therefore, the points characterizing the corners of the container are preferably used as the preset key points.
[0082] Based on this, in a preferred embodiment, the preset key points include a first key point, a second key point, a third key point, and a fourth key point.
[0083] The first key point is a vertex characterizing the four top corners of the container.
[0084] The second key point is two vertices along the width direction of the container, which characterize the recess for mounting the container door.
[0085] The third key point is a vertex characterizing the top corner at one end of the container door width direction which is not rotatably connected to the container.
[0086] The fourth key point is a midpoint characterizing the center of the upper surface of the container.
[0087] Specifically, taking the target image of the container in which the container door is in a closed state as shown in Figure 2 , the preset key points are seven points as shown in Figure 3 .
[0088] More specifically, for the container with the doors in the two open states as shown in Figure 4 and Figure 5 , the preset key points are 8 points as shown in Figure 4 for the container as shown in Figure 6 , and the preset key points are 9 points as shown in Figure 5 for the container as shown in Figure 7 .
[0089] S120, the preset key points are associated by distance constraint to obtain the container body skeleton.
[0090] In this embodiment, after the preset key points are detected, the corresponding positions of the preset key points on the container can be determined according to the distance relationship between the preset key points.
[0091] Further, after the positions of the preset key points corresponding to the container are determined, the container body skeleton can be obtained by connecting the preset key points.
[0092] As described above, when the door states are different, the number of the preset key points detected by the target image detection is different. Therefore, the specific method of associating the preset key points by distance constraint to obtain the container body skeleton is also different for different preset key points detected.
[0093] Based on this, in a preferred embodiment, when the preset key points detected by the target image detection include the first key point, the second key point and the fourth key point, the preset key points are associated by distance constraint to obtain the container body skeleton, including:
[0094] Based on the pixel coordinates of the preset key points in the target image, the distance between each two preset key points is calculated as the first distance;
[0095] According to the order of the first distance from small to large, two first line segments are determined.
[0096] The first line segment is a line segment with the second key point and the first key point closest to the second key point as end points;
[0097] The distance between the first line segment and each preset key point is determined, the preset key point closest to the first line segment is determined as the fourth key point, and the fourth key point is connected to the two first key points not included in the first line segment to obtain a second line segment;
[0098] determining the distance between the fourth key point and the end points of the first line segment, determining two preset key points closest to the fourth key point in the first line segment as the second key points, and connecting the second key points with the fourth key point to obtain a third line segment;
[0099] Based on the first line segment, the second line segment and the third line segment, the box skeleton is formed.
[0100] Specifically, when the target image is subjected to the preset key point detection processing, the obtained preset key points include the first key point, the second key point and the fourth key point, the top point of the top corner not connected with one end of the container along the width direction of the container door is represented, that is, the case of the closed container door as shown in FIG. 1. Figure 2
[0101] Specifically, as shown in FIG. 1, Figure 3 It can be seen that in this case, a total of 7 preset key points P1-P7 can be detected. Then, according to the pixel coordinates of the key points in the image, the distance constraint is used, that is, P1, P2 and P3, P4 are paired to obtain the skeleton P1P2, P3P4 line segment, that is, the first line segment.
[0102] In some possible embodiments, the distance constraint can be a Euclidean constraint. The topK algorithm can be used for selection: that is, the distances between all 2 points of the above-mentioned 7 preset key points are first calculated, and then the first 2 line segments with the shortest distance are selected, that is, the P1P2, P3P4 line segments.
[0103] Further, according to the pixel coordinates, the preset key point closest to the P1P2, P3P4 line segment is found, that is, the point P5, that is, the fourth key point. Then, in the P1P2, P3P4 line segment, the points (using the horizontal coordinates, the vertical coordinates and the Euclidean distance) close to the point P5 are found, that is, P2, P3, that is, the second key points, and the remaining two end points of the first line segment are the two first key points, and then the line segments P2P5, P3P5, that is, the third line segment, are obtained. The last remaining two preset key points P6, P7 are the remaining two first key points, which form the line segments P5P6, P5P7, that is, the second line segment, with the point P5. At this time, the topological relationship of all points is confirmed, and the box skeleton composed of the first line segment, the second line segment and the third line segment as shown in FIG. 1 is obtained. Figure 8
[0104] In one preferred embodiment, when the target image is subjected to the preset key point detection processing, the obtained preset key points include the first key point, the second key point, the third key point and the fourth key point, the preset key points are subjected to the correlation processing through the distance constraint to obtain the box skeleton of the container, including:
[0105] Based on the pixel coordinates of the preset key points in the target image, distances between all the preset key points are calculated as second distances;
[0106] According to the second distances from small to large, two fourth line segments are determined, the fourth line segment being a line segment with the second key point and the first key point closest to the second key point as end points;
[0107] Distances between the fourth line segment and the preset key points are determined, the preset key point closest to the fourth line segment being determined as a third key point, and the preset key point second closest to the fourth line segment being determined as a fourth key point;
[0108] Distances between the fourth key point and the end points of the fourth line segment are determined, the two preset key points closest to the fourth key point in the fourth line segment being determined as second key points, and the second key points being connected to the fourth key point to obtain a fifth line segment;
[0109] The third key point is connected to the second key point closest to the third key point to obtain a sixth line segment;
[0110] The fourth key point is connected to the two first key points not included in the fourth line segment to obtain a seventh line segment;
[0111] Based on the fourth line segment, the fifth line segment, the sixth line segment and the seventh line segment, a box skeleton is formed.
[0112] Specifically, when the target image is subjected to preset key point detection processing, the obtained preset key points include: first key points, second key points, third key points and fourth key points, it is detected that one or two top points of a top corner along the width direction of the container door are not connected to one end of the container, that is, as shown in Figure 4 or Figure 5 the container door is opened.
[0113] Specifically, in this case, as shown in Figure 6 8 preset key points P1-P8 are detected, or as shown in Figure 7 9 preset key points P1-P9 are detected. Then, according to the pixel coordinates of the key points in the image, the distance constraint is used, and P1, P2 and P3, P4 can be paired first to obtain the skeleton P1P2, P3P4 line segment, that is, the fourth line segment.
[0114] In some possible embodiments, the distance constraint can be a Euclidean constraint. The topK algorithm can be used for selection: that is, the distances between all the 8 or 9 preset key points are calculated first, and then the first two line segments with the shortest distance are selected, that is, P1P2, P3P4 line segments.
[0115] Further, according to the pixel coordinates, the preset key point closest to the line segment P1P2, P3P4 is found, that is, the third key point P8 and / or P9 is determined, and then the second closest preset key point P5 to the line segment P1P2, P3P4 is found, that is, the fourth key point. Then, the points close to the point P5 (using the horizontal coordinate, the vertical coordinate, and the Euclidean distance) in the line segment P1P2, P3P4 are found, that is, the second key points P2 and P3, and the remaining two end points of the fourth line segment are the two first key points, and then the line segments P2P5 and P3P5, that is, the fifth line segment, is obtained. Then, the line segment P2P8 or P3P9, that is, the sixth line segment, is obtained by connecting the P8 and / or P9 with the second key point P2 or P3 closest to the P8 and / or P9, and the remaining two preset key points P6 and P7 are the remaining two first key points, and the line segments P5P6 and P5P7, that is, the seventh line segment, is obtained by connecting the P6 and P7 with the point P5. Thus, the topological relationship of all points is determined, and the box skeleton composed of the fourth line segment, the fifth line segment, the sixth line segment, and the seventh line segment is obtained as shown in Figure 9 or Figure 10
[0116] S130, determining the orientation and state of the container door based on the box skeleton.
[0117] Specifically, as shown in Figure 8 , Figure 9 or Figure 10 the box skeleton can be seen that the container door is oriented to the side of the line segment P1P2 and P3P4, and when the P8 and / or P9 exist, it indicates that there is a gap between the container door and the container body or the container door is opened, so that the state of the container door can be further determined based on the box skeleton.
[0118] Further, for the case that the third key point does not exist in the box skeleton, as shown in Figure 8 , it is indicated that there is no gap between the container door and the container body, and the state of the container door can be determined to be closed based on the box skeleton.
[0119] Based on this, in a preferred embodiment, the orientation and state of the container door are determined based on the box skeleton, including:
[0120] Based on the box skeleton, the side of the first line segment is determined as the orientation of the container door, and the state of the container door is determined to be closed.
[0121] Further, for the case that the third key point exists in the box skeleton, as shown in Figure 9 or Figure 10 , it is indicated that there is a gap between the container door and the container body or the container door is in an open state, and it is further needed to determine whether the state of the container door is open.
[0122] Based on this, in a preferred embodiment, determining the direction and state of the door based on the box frame includes:
[0123] Based on the box frame, the fourth line segment side is determined as the direction of the box door;
[0124] Connect the two second key points to obtain the eighth line segment;
[0125] determining the angle between the sixth line segment and the eighth line segment;
[0126] Based on the relationship between the included angle and the preset angle, it is determined whether the state of the door is open.
[0127] Specifically, after determining the fourth line segment side as the direction of the door based on the box skeleton, the two second key points, that is, Figure 9 or Figure 10 Connecting points P2 and P3, the resulting line segment P2P3 represents the ideal position of the door on the container body when fully closed. However, it is understandable that due to the age of the container and manufacturing errors, the connection between the door and the container body may become loose to varying degrees. When this occurs, a gap may exist between the door and the container body when the door is closed. In this case, the third key point can also be detected. Therefore, if the door is determined to be open based on the detection of the third key point, a situation where there is a gap between the closed door and the container body may be mistakenly interpreted as an open door.
[0128] Based on this, in this embodiment, by constructing the eighth line segment and then calculating the angle between the sixth and eighth line segments, the angle at which the door is opened can be quantified. Then, by determining the relationship between the angle and the preset angle, in a preferred embodiment, determining whether the door is open based on the relationship between the angle and the preset angle includes:
[0129] When the included angle is less than or equal to the preset angle, the door is determined to be closed;
[0130] When the included angle is greater than a preset angle, the state of the door is determined to be open, and the included angle is used as the opening angle of the door.
[0131] This allows for accurate differentiation between a closed door with a gap and an open door, which is impossible with direct graphical recognition of container images.
[0132] Based on the content of the above embodiment, the container door detection provided by the embodiment of the application realizes accurate detection of the orientation and opening and closing state of the container door at the same time, achieves camera multiplexing, saves operation resources, and further reduces the risk of misjudgment of the state of the container door by quantifying the opening angle of the container door. By setting the target image as the overhead view of the container taken before the spreader grabs the container, the container door condition can be known earlier than through container door detection based on the container image taken after the spreader grabs the container, the detection efficiency is improved, and the problem of being unable to identify the orientation and state of the container door due to the appearance features of the container door and the container tail being the same in the taken image is avoided.
[0133] Next, a container door detection system provided by an embodiment of the application is introduced. The container door detection system described below can be considered as a module architecture for implementing the container door detection method provided by the embodiment of the application. The content described below can be referred to with the content described above.
[0134] Optionally, referring to Figure 11 , Figure 11 is a structural block diagram of a container door detection system provided by an embodiment of the application. The system can include:
[0135] An image acquisition unit 10 is configured to acquire a target image. The target image is an overhead view of a container taken by a camera arranged on a spreader before the spreader grabs the container.
[0136] A key point detection unit 20 is configured to perform preset key point detection processing on the target image to obtain preset key points for representing the container body structure of the container.
[0137] A skeleton construction unit 30 is configured to associate the preset key points through distance constraint to obtain a container body skeleton of the container.
[0138] A container door detection unit 40 is configured to determine the orientation and state of the container door of the container based on the container body skeleton.
[0139] Optionally, the preset key points include a first key point, a second key point, a third key point, and a fourth key point.
[0140] The first key point is a vertex representing four corners of the container.
[0141] The second key point is two vertices representing grooves for mounting the container door on the container along the width direction of the container.
[0142] The third key point is a vertex representing a corner not connected to one end of the container along the width direction of the container door.
[0143] The fourth key point is a midpoint representing the center of the upper surface of the container.
[0144] Optionally, when the preset key points obtained by the key point detection unit 20 include the first key point, the second key point, and the fourth key point, the skeleton construction unit 30 is specifically configured to:
[0145] Based on the pixel coordinates of the preset key points in the target image, the distances between all the preset key points are calculated as first distances.
[0146] According to the order of the first distances from small to large, two first line segments are determined, and the first line segment is a line segment with the second key point and the first key point closest to the second key point as end points.
[0147] The distances between the first line segment and each preset key point are determined, the preset key point closest to the first line segment is determined as the fourth key point, and the fourth key point is connected to the two first key points not included in the first line segment to obtain a second line segment.
[0148] The distances between the fourth key point and the end points of the first line segment are determined, the two preset key points closest to the fourth key point in the first line segment are determined as the second key point, and the second key point is connected to the fourth key point to obtain a third line segment.
[0149] Based on the first line segment, the second line segment, and the third line segment, a box skeleton is constructed.
[0150] Optionally, the box door detection unit 40 is specifically configured to:
[0151] Based on the box skeleton, the first line segment side is determined as the direction of the box door, and the state of the box door is determined as closed.
[0152] Optionally, when the preset key points obtained by the key point detection unit 20 include the first key point, the second key point, the third key point, and the fourth key point, the skeleton construction unit 30 is specifically configured to:
[0153] Based on the pixel coordinates of the preset key points in the target image, the distances between all the preset key points are calculated as second distances.
[0154] According to the order of the second distances from small to large, two fourth line segments are determined, and the fourth line segment is a line segment with the second key point and the first key point closest to the second key point as end points.
[0155] The distances between the fourth line segment and each preset key point are determined, the preset key point closest to the fourth line segment is determined as the third key point, and the preset key point second closest to the fourth line segment is determined as the fourth key point.
[0156] determine the distance between the fourth key point and the end point of the fourth line segment, determine two preset key points closest to the fourth key point in the fourth line segment as second key points, and connect the second key points and the fourth key point respectively to obtain a fifth line segment;
[0157] determine a sixth line segment by connecting the third key point and the second key point closest to the third key point;
[0158] connect the fourth key point and the two first key points not included in the fourth line segment respectively to obtain a seventh line segment;
[0159] based on the fourth line segment, the fifth line segment, the sixth line segment and the seventh line segment, form a box skeleton.
[0160] Optionally, the box door detection unit 40 is specifically used for:
[0161] based on the box skeleton, determine the fourth line segment side as the orientation of the box door;
[0162] connect the two second key points to obtain an eighth line segment;
[0163] determine the included angle between the sixth line segment and the eighth line segment;
[0164] based on the relationship between the included angle and a preset angle, determine whether the state of the box door is open.
[0165] Optionally, the box door detection unit 40 is more specifically used for:
[0166] when the included angle is less than or equal to the preset angle, determine that the state of the box door is closed;
[0167] when the included angle is greater than the preset angle, determine that the state of the box door is open, and take the included angle as the opening angle of the box door.
[0168] Optionally, the embodiment of the present application also provides a yard crane, comprising: a lifting tool, a camera and a controller;
[0169] the lifting tool is used for lifting the container during the container transfer process;
[0170] the camera is arranged on the lifting tool and is used for shooting the image of the container before the lifting tool grabs the container, so as to obtain the top view of the container and serve as a target image;
[0171] the controller is used for acquiring the target image, performing preset key point detection processing on the target image to obtain preset key points representing the box structure of the container, and performing distance constraint association processing on the preset key points to obtain the box skeleton of the container, so as to determine the orientation and state of the box door of the container based on the box skeleton, and execute a preset strategy when it is determined that the state of the box door is open.
[0172] Specifically, the preset strategy can include triggering prompt information, sending a pause box-grasping action instruction to the spreader, etc., which are not limited here.
[0173] The electronic device provided by the embodiment of the present application will be described below with reference to Figure 12 The electronic device provided by the embodiment of the present application can include at least one processor 100, at least one communication interface 200, at least one memory 300 and at least one communication bus 400.
[0174] In the embodiment of the present application, the number of the processor 100, the communication interface 200, the memory 300 and the communication bus 400 is at least one, and the processor 100, the communication interface 200 and the memory 300 complete the communication with each other through the communication bus 400. Obviously, Figure 12 The communication connection shown by the processor 100, the communication interface 200, the memory 300 and the communication bus 400 shown is only optional.
[0175] Optionally, the communication interface 200 can be the interface of the communication module, such as the interface of the GSM module; the processor 100 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiment of the present application.
[0176] The memory 300 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.
[0177] The processor 100 is specifically configured to execute the application program in the memory to implement the steps of the container door detection method described above.
[0178] The basic principles of the present application are described above in combination with specific embodiments, but it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present application are only examples and are not limited, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present application. In addition, the above-mentioned specific details are only for the purpose of example and for the purpose of understanding, and are not limited to the above-mentioned specific details, and the above-mentioned details do not limit the present application to the above-mentioned specific details.
[0179] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply that the connection, arrangement, configuration must be as shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have", and the like are open-ended words, mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably, unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.
[0180] It should also be noted that in the devices, apparatuses and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application.
[0181] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the present application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0182] It should be understood that the adjectives "first", "second", "third", "fourth", "fifth" and "sixth" used in the embodiments of the present application are only used for more clearly describing the technical solutions, and cannot be used to limit the protection scope of the present application.
[0183] The above description has been given for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A container door detection method, characterized in that: include: Acquire a target image, where the target image is a top view of the container captured by a camera disposed on the spreader before the spreader grabs the container; Performing preset key point detection processing on the target image to obtain preset key points for characterizing the box structure of the container; Associating the preset key points through distance constraints to obtain the container skeleton; Based on the container frame, the orientation and status of the container door are determined.
2. The method according to claim 1, characterized in that The preset key points include a first key point, a second key point, a third key point and a fourth key point; Wherein, the first key points are vertices representing the four corners of the container; The second key points are two vertices of a groove on the container for mounting the door along the width direction of the container; The third key point is the vertex representing the vertex angle of the end of the door that is not screwed to the container along the width direction of the door; The fourth key point is a midpoint representing the center of the upper surface of the container.
3. The method according to claim 2, characterized in that When the target image is subjected to preset key point detection processing, and the obtained preset key points include: the first key point, the second key point, and the fourth key point, the preset key points are subjected to association processing using distance constraints to obtain the container skeleton, including: Based on the pixel coordinates of the preset key points in the target image, calculating the distances between all two of the preset key points as first distances; Determine two first line segments according to the order of the first distances from smallest to largest, where the first line segments are line segments having the second key point and the first key point closest to the second key point as endpoints; determining the distance between the first line segment and each of the preset key points, determining the preset key point closest to the first line segment as the fourth key point, and connecting the fourth key point to two of the first key points not included in the first line segment, to obtain a second line segment; Determining the distance between the fourth key point and the endpoint of the first line segment, determining the two preset key points in the first line segment that are closest to the fourth key point as the second key points, and connecting the second key points to the fourth key points respectively to obtain a third line segment; The box skeleton is constructed based on the first line segment, the second line segment and the third line segment.
4. The method according to claim 3, characterized in that The determining of the direction and state of the door based on the box frame includes: Based on the box frame, the first line segment side is determined as the direction of the box door, and the state of the box door is determined to be closed.
5. The method according to claim 2, characterized in that When the target image is subjected to preset key point detection processing, and the obtained preset key points include: the first key point, the second key point, the third key point, and the fourth key point, the preset key points are associated with each other through distance constraints to obtain the container skeleton, including: Based on the pixel coordinates of the preset key points in the target image, calculating the distances between all two of the preset key points as second distances; Determine two fourth line segments according to the second distances in ascending order, where the fourth line segments are line segments having the second key point and the first key point closest to the second key point as endpoints; determining the distances between the fourth line segment and each of the preset key points, determining the preset key point closest to the fourth line segment as the third key point, and determining the preset key point second closest to the fourth line segment as the fourth key point; determining a distance between the fourth key point and an endpoint of the fourth line segment, determining two of the preset key points in the fourth line segment that are closest to the fourth key point as the second key points, and connecting the second key points to the fourth key points respectively to obtain a fifth line segment; Connecting the third key point to the second key point closest to the third key point to obtain a sixth line segment; Connecting the fourth key point to two of the first key points not included in the fourth line segment, respectively, to obtain a seventh line segment; The box skeleton is constructed based on the fourth line segment, the fifth line segment, the sixth line segment, and the seventh line segment.
6. The method according to claim 5, characterized in that The determining of the direction and state of the door based on the box frame includes: Based on the box frame, determining the fourth line segment side as the direction of the box door; Connect the two second key points to obtain an eighth line segment; determining an angle between the sixth line segment and the eighth line segment; Based on the relationship between the included angle and the preset angle, it is determined whether the state of the door is open.
7. The method according to claim 6, characterized in that The determining whether the door is open based on the relationship between the included angle and the preset angle includes: When the included angle is less than or equal to the preset angle, determining that the state of the door is closed; When the included angle is greater than the preset angle, the state of the door is determined to be open, and the included angle is used as the opening angle of the door.
8. A container door detection system, characterized in that: include: An image acquisition unit, configured to acquire a target image, wherein the target image is a top view of the container captured by a camera arranged on the spreader before the spreader grabs the container; a key point detection unit, configured to perform preset key point detection processing on the target image to obtain preset key points for characterizing the box structure of the container; A skeleton construction unit, configured to perform association processing on the preset key points through distance constraints to obtain a body skeleton of the container; The door detection unit is used to determine the orientation and status of the container door based on the container frame.
9. A field bridge, characterized in that: include: spreader, camera, and controller; The sling is used to lift the container during container transshipment; The camera is provided on the spreader and is used to capture an image of the container before the spreader grabs the container, so as to obtain a top view of the container as a target image; The controller is used to acquire the target image, perform preset key point detection processing on the target image, obtain preset key points for characterizing the box structure of the container, and perform association processing on the preset key points through distance constraints to obtain the box skeleton of the container, so as to determine the orientation and state of the container door based on the box skeleton, and execute a preset strategy when it is determined that the state of the box door is open.
10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the method according to any one of claims 1 to 7.