Container corner fitting positioning method, device and equipment and storage medium

Abstract

The invention discloses a container corner fitting positioning method, device and equipment and a storage medium, a target area on a container is shot through a monocular camera and a three-dimensional structured light camera, an RGB image and a depth image corresponding to the target area are obtained, and the target area comprises corner fittings; performing target detection processing on the RGB image to obtain a side hole detection frame corresponding to a side hole in the corner fitting; determining a target image area corresponding to the side hole detection frame in the depth image; converting the target image area into a first three-dimensional point cloud; extracting edge point clouds of the side holes from the first three-dimensional point clouds; determining first attitude information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud; and determining second attitude information of the lockhole in the corner piece relative to the three-dimensional structured light camera based on the first attitude information. According to the embodiment of the invention, the problems of low robustness, weak real-time performance, large scene dependence and high construction cost existing in current corner fitting positioning can be solved.

Description

Technical Field

[0001] The present application relates to the technology of automated loading and unloading of container lock pins, and in particular to a method, device, equipment and storage medium for positioning container corner fittings. Background Art

[0002] In order to ensure the safety of the container during transportation and prevent the container from moving / dumping, the container needs to be fixed during transportation. At present, the connection and fixation between adjacent containers is mainly achieved by setting corner fittings on the container and installing locking pins on the corner fittings of adjacent containers. When the existing container terminal loads and unloads the berthed container transport ship, it is necessary to load and unload the locking pins. The automatic loading and unloading system of container locking pins is of great significance to further improve the degree of automation of terminal operations and reduce the labor intensity of on-site staff.

[0003] In order to accurately install the lock pin on the corner fitting or remove the lock pin from the corner fitting, the container lock pin automatic loading and unloading system needs to locate the corner fitting when automatically loading and unloading the lock pin. At present, the positioning method based on binocular camera or the positioning method based on laser rangefinder is mainly used to locate the installation position of the lock pin. Among them, the positioning method based on binocular camera mainly detects corner points based on image optical feature points, performs three-dimensional reconstruction based on corner points, and then measures and calculates the distance from the fixture to the corner fitting. However, the accuracy and robustness of the results are easily affected by the on-site shooting lighting environment, and a light source with specific brightness, contrast and good robustness is required in the working environment to provide a stable lighting environment; at the same time, when a large number of corner points are used for three-dimensional reconstruction, there is a problem that the real-time performance cannot meet the on-site operation, and only extracting specific corner points has strict requirements on the shooting scene and lighting conditions, and a large amount of cost needs to be invested in the early stage to complete the construction of a stable shooting scene. In the positioning method based on laser rangefinder, the specific points of the container corner fitting are measured by a laser rangefinder installed in a fixed position to achieve corner fitting positioning. This method requires that the container to be positioned be placed in a specific location or in a positioning device before using laser ranging, which makes the solution dependent on a fixed location or a large mechanical positioning device, and has the problem of high implementation cost.

[0004] In summary, the current two mainstream positioning algorithms have problems such as low robustness, weak real-time performance, high scene dependence and high construction cost. Summary of the invention

[0005] The embodiments of the present application provide a container corner fitting positioning method, device, equipment and storage medium, which can solve the problems of low robustness, weak real-time performance, high scene dependence and high construction cost in the current corner fitting positioning.

[0006] In a first aspect, an embodiment of the present application provides a container corner fitting positioning method, characterized by comprising:

[0007] The target area on the container is photographed by a monocular camera and a 3D structured light camera respectively to obtain the RGB image and depth image corresponding to the target area, which includes the corner pieces.

[0008] Perform object detection processing on the RGB image to obtain the side hole detection frame corresponding to the side hole on the corner piece;

[0009] Determine a target image region corresponding to the side hole detection frame in the depth image;

[0010] Converting the target image region into a first three-dimensional point cloud;

[0011] Extracting edge point clouds of the side holes from the first three-dimensional point cloud;

[0012] Determine first posture information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud;

[0013] Based on the first posture information, second posture information of the keyhole on the corner piece relative to the three-dimensional structured light camera is determined.

[0014] In a second aspect, an embodiment of the present application provides a container corner fitting positioning device, characterized in that it includes:

[0015] A shooting module is used to shoot a target area on the container using a monocular camera and a 3D structured light camera to obtain an RGB image and a depth image corresponding to the target area, where the target area includes corner fittings;

[0016] The target detection module is used to perform target detection processing on the RGB image to obtain a side hole detection frame corresponding to the side hole on the corner piece;

[0017] A region determination module, used to determine a target image region corresponding to a side hole detection frame in a depth image;

[0018] A three-dimensional reconstruction module, used for converting the target image area into a first three-dimensional point cloud;

[0019] An edge extraction module, used for extracting an edge point cloud of the side hole from the first three-dimensional point cloud;

[0020] A positioning module, used to determine first posture information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud;

[0021] The positioning module is also used to determine second posture information of the lock hole on the corner piece relative to the three-dimensional structured light camera based on the first posture information.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0023] When the processor executes the computer program instructions, the container corner fitting positioning method according to the first aspect is implemented.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the container corner fitting positioning method of the first aspect is implemented.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the container corner fitting positioning method as described in the first aspect.

[0026] The container corner fitting positioning method, device, equipment and storage medium of the embodiment of the present application integrates the target detection of RGB images and the three-dimensional reconstruction of depth images, and uses the detection results of the corner fitting side holes in the RGB image to segment the target image area corresponding to the side holes in the depth image, and only performs three-dimensional reconstruction on the target image area, which can achieve high-precision container corner fitting side hole positioning, thereby achieving high-precision corner fitting lock hole pose estimation. Compared with the positioning method based on binocular cameras, the complexity of three-dimensional reconstruction and the amount of reconstruction calculations are reduced, the real-time positioning is improved, and there is no need to consider lighting conditions, which improves robustness. Compared with the positioning method based on laser rangefinders, there is no need to place the container in a specific position, and the dependence on the scene is small and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] Figure 1 It is a schematic diagram of the flow of the container corner fitting positioning method provided in an embodiment of the present application;

[0029] Figure 2 is a structural schematic diagram of a container corner fitting positioning device provided in an embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0032] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0033] Before describing the technical solution provided by the embodiment of the present application, in order to facilitate the understanding of the embodiment of the present application, the present application first describes the automatic loading and unloading system of container lock pins.

[0034] Generally speaking, one or more robotic arms are used in an automated loading and unloading system for container lock pins. With the help of a visual system, the lock pins installed on the corner fittings and / or the corner fittings are positioned to complete the loading and unloading of container lock pins. Among them, the visual system for loading and unloading container lock pins is an indispensable part for realizing the automated removal of container lock pins by the robotic arm. In general, a visual system for loading and unloading container lock pins can be composed of two main parts: a visual recognition module and a distance measurement module. Among them, with the accuracy and robustness of the deep learning model in the image detection results, the visual recognition module can detect the type of container lock pins to be removed and the installation status of the lock pins on the current corner fittings, provide the state of the corner fittings or the type of lock pins installed thereon for the automated loading and unloading system for container lock pins, and guide the robotic arm in the automated loading and unloading system for container lock pins to replace the loading and unloading tooling used for the corresponding lock pins. The main purpose of the distance measurement module is to measure the posture transformation information between the lock pin loading and unloading tooling and the corner fittings, that is, to locate the lock pins installed on the corner fittings or the corner fittings, so that the robotic arm can accurately install the lock pins on the corner fittings or remove the lock pins from the corner fittings. For the entire process of automated loading and unloading of container lock pins, the visual recognition module needs to be able to accurately identify the type of container lock pins to ensure that the state recognition of the entire system is correct, so that the robot arm can correctly load and unload the container lock pins; at the same time, the distance measurement module needs to be able to accurately locate the corner pieces or the lock pins installed on the corner pieces, so that the robot arm can successfully remove the container lock pins on the one hand, and on the other hand, the fixture, container lock pins and even the robot arm itself will not be damaged during the loading and unloading process. Therefore, it is very necessary to study the visual recognition technology and positioning technology for automated loading and unloading of container lock pins.

[0035] The embodiment of the present application mainly provides a novel container corner fitting positioning method, device, equipment, medium and program to solve the problems of low robustness, weak real-time performance, high scene dependence and high construction cost in the existing positioning methods. The container corner fitting positioning method provided by the embodiment of the present application is first described below.

[0036] The container corner fitting positioning method provided in the embodiment of the present application can be applied to the container lock pin automatic loading and unloading system. Figure 1 The container corner fitting positioning method provided in the embodiment of the present application is described.

[0037] See also Figure 1 , is a flow chart of a container corner fitting positioning method provided in an embodiment of the present application, such as Figure 1 As shown, the method may include the following steps:

[0038] S11. The target area on the container is photographed by a monocular camera and a 3D structured light camera respectively to obtain an RGB image and a depth image corresponding to the target area, wherein the target area includes corner pieces.

[0039] The container corner fitting positioning method provided in the embodiment of the present application can be applied to a container corner fitting positioning device in an automated loading and unloading system for container lock pins, and the hardware part of the container corner fitting positioning device includes a monocular camera and a three-dimensional structured light camera. In order to enable the automated loading and unloading system for container lock pins to accurately load and unload lock pins, the container corner fitting positioning device positions the corner fittings used to install the lock pins on the container during the loading and unloading process.

[0040] When locating the corner fittings, the target area in the container is photographed using a monocular camera and a 3D structured light camera, respectively, to obtain an RGB image of the target area photographed by the monocular camera and a depth image of the target area photographed by the 3D structured light camera. The target area includes the corner fittings on the container, so both the RGB image and the depth image contain images of the corner fittings. Among them, the RGB image is an image format that uses three basic colors: red, green, and blue.

[0041] In some embodiments of the present application, an area within a preset range of the corner fitting installation position on the container may be used as a target area, and the target area may include a complete corner fitting.

[0042] S12. Perform target detection processing on the RGB image to obtain a side hole detection frame corresponding to the side hole on the corner piece.

[0043] In some embodiments of the present application, the corner piece may include a side surface, a bottom surface and a top surface that are perpendicular to each other, a side hole is provided on the side surface, and a lock hole for installing a lock pin is provided on the top surface and / or the bottom surface. Positioning the corner piece is mainly to determine the position of the lock hole, but since the lock hole is located on the bottom surface or top surface of the corner piece, it is not convenient to shoot, so the side hole on the corner piece is positioned.

[0044] In some embodiments of the present application, after the RGB image is captured, a deep learning target detection model can be used to perform target detection processing on the RGB image to obtain a side hole detection frame corresponding to the side hole on the corner piece. The target detection model can use a two-dimensional image target detection model pre-trained with on-site image data, with the RGB image captured by a monocular camera as input, and the result output is a side hole detection frame of the captured corner piece detected by the target detection model reasoning.

[0045] S13. Determine a target image area corresponding to the side hole detection frame in the depth image.

[0046] In some embodiments of the present application, the coordinate systems corresponding to the monocular camera and the 3D structured light camera are different. The camera coordinate system corresponding to the monocular camera is the monocular camera coordinate system, and the coordinate system corresponding to the 3D structured light camera is the 3D structured light camera coordinate system. Therefore, the position information corresponding to the side hole of the corner piece in the RGB image is the same as the position information corresponding to the depth image. Based on this, when determining the target image area corresponding to the side hole detection frame in the depth image, the following steps S131-S133 can be used:

[0047] S131. Determine the first position information of the side hole detection frame in the monocular camera coordinate system;

[0048] S132. Based on the internal parameters of the monocular camera, the internal parameters of the 3D structured light camera, and the external parameters of the 3D structured light camera relative to the monocular camera, coordinate transformation is performed on the first position information to obtain the second position information of the side hole detection frame in the 3D structured light camera coordinate system;

[0049] S133. Use the image area in the depth image corresponding to the second position information as the target image area.

[0050] In this embodiment, the first position information is information that can characterize the position of the side hole detection frame in the RGB image, based on which, the second position information obtained by coordinate conversion of the first position information can characterize the corresponding position of the side hole detection frame in the depth image. In this way, after obtaining the second position information, the image area determined from the depth image based on the second position information can be used as the target image area corresponding to the side hole detection frame in the depth image.

[0051] In some examples, the side hole detection frame may be a rectangular frame, and the first position information may include the pixel coordinates of two corner points of the side hole detection frame on any diagonal line in the monocular camera coordinate system. i ={C tl,i ,C lr,i}, where C tl,i Indicates the pixel coordinates of the upper left corner of the side hole detection box, C lr,i represents the pixel coordinate of the lower right corner of the side hole detection frame. Based on this, the second position information can be expressed as formula (1):

[0052]

[0053] In the formula, B i,dual Indicates the second position information, K dual represents the internal parameters of the 3D structured light camera, K mono represents the internal parameters of the monocular camera, Represents the external parameters of the 3D structured light camera relative to the monocular camera.

[0054] S14. Convert the target image area into a first three-dimensional point cloud.

[0055] In some embodiments of the present application, after obtaining the target image area, a 3D point cloud can be generated based on the target image area, and the generated 3D point cloud is used as the first 3D point cloud. In this way, only the high-precision 3D point cloud of the target image area within the side hole detection frame needs to be reconstructed.

[0056] At present, there are related mature technologies for generating three-dimensional point clouds based on depth images. This embodiment can use the existing generated technology to convert the target image area into a first three-dimensional point cloud, which will not be described in detail here.

[0057] S15. Extract edge point cloud of the side hole from the first three-dimensional point cloud.

[0058] In this embodiment, in addition to the point cloud corresponding to the edge of the side hole, the first three-dimensional point cloud may also include some other point clouds. In order to accurately locate the side hole, the point cloud corresponding to the edge of the side hole, that is, the edge point cloud, is extracted from the first three-dimensional point cloud, and the side hole is accurately located based on the edge point cloud.

[0059] In some embodiments of the present application, the edge point cloud of the side hole can be extracted from the first three-dimensional point cloud by using the following steps S151-S153:

[0060] S151. Downsample the first three-dimensional point cloud to obtain a second three-dimensional point cloud.

[0061] In some embodiments of the present application, a voxel downsampling method may be used to downsample the first three-dimensional point cloud to obtain a second three-dimensional point cloud.

[0062] By downsampling the first three-dimensional point cloud, the amount of computational data can be reduced.

[0063] S152. Remove noise points in the second three-dimensional point cloud to obtain a third three-dimensional point cloud.

[0064] In some embodiments of the present application, a preprocessing method such as a straight-through filter, a radius filter, and a random sample consensus (RANSAC) can be used to remove point cloud noise points in the second three-dimensional point cloud to obtain a third three-dimensional point cloud.

[0065] The accuracy of 3D point cloud can be improved by removing noise points.

[0066] S153. Extract the edge point cloud of the side hole from the third three-dimensional point cloud.

[0067] In some embodiments of the present application, there are currently related mature technologies for edge feature extraction of point clouds. In this embodiment, the existing mature technologies can be used to extract the edge point cloud of the side hole from the third three-dimensional point cloud. For example, the edge point cloud of the side hole can be extracted from the third three-dimensional point cloud using a point cloud edge extraction algorithm or a machine learning algorithm. The machine learning algorithm may include, but is not limited to, principal component analysis (PCA) and DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm, etc.

[0068] In some examples, when extracting the edge point cloud of the side hole from the third three-dimensional point cloud, the main components of the third three-dimensional point cloud can be first extracted through PCA, so as to compress the third three-dimensional point cloud to a low-dimensional space while retaining the main features, and then the reduced dimensionality third three-dimensional point cloud is clustered through the DBSCAN clustering algorithm to obtain the edge point cloud of the side hole.

[0069] S16. Determine first posture information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud.

[0070] In some embodiments of the present application, the position information of the edge point cloud can characterize the relative position relationship between the side hole and the three-dimensional structured light camera. Based on this, the first posture information of the side hole relative to the three-dimensional structured light camera can be determined based on the edge point cloud.

[0071] In some embodiments of the present application, the first posture information of the side hole relative to the three-dimensional structured light camera is determined based on the edge point cloud in the following manner:

[0072] The edge point cloud is registered with the target point cloud to obtain the attitude angle information and displacement information of the side hole relative to the 3D structured light camera. The target point cloud is used to represent the edge point cloud when the side hole is facing the 3D structured light camera.

[0073] The attitude angle information and the displacement information are determined as first attitude information of the side hole relative to the three-dimensional structured light camera.

[0074] Among them, the point cloud registration algorithm refers to the input of two point clouds P src and P tar , output a transformation matrix T (i.e., attitude angle information R and displacement information t) so that the overlap between T(Ps) and Pt is as high as possible. Point cloud registration of the edge point cloud with the target point cloud refers to using the edge point cloud as P src , the target point cloud used to represent the edge point cloud when the side hole is facing the 3D structured light camera is taken as Ptar , the two are point cloud registered to obtain the transformation matrix T. Because the target point cloud represents the edge point cloud when the side hole is facing the 3D structured light camera, the attitude angle information R in the transformation matrix T is the attitude angle information of the side hole relative to the 3D structured light camera, and the displacement information t in the transformation matrix T is the displacement information of the side hole relative to the 3D structured light camera.

[0075] In some embodiments of the present application, the target point cloud may be a preset side hole template.

[0076] In some embodiments of the present application, an ICP (Iterative Closest Point) algorithm may be used to perform point cloud registration on the edge point cloud and the target point cloud. The optimization equation of the ICP algorithm is shown in formula (2):

[0077]

[0078] Where argmin represents the minimum index, e = ∑||p j,tar -(Rp i,src +t)||,p i,src Represents any point i, p in the edge point cloud j,tar Represents the target point cloud P tar In and p i,src Relationship point, R represents the attitude angle information of the side hole relative to the 3D structured light camera, and t represents the displacement information of the side hole relative to the 3D structured light camera.

[0079] Before performing point cloud registration on the edge point cloud and the target point cloud according to formula (2), the displacement information t of the side hole relative to the three-dimensional structured light camera can be determined first, and then the determined t is substituted into formula (2) to obtain the attitude angle information R of the side hole relative to the three-dimensional structured light camera.

[0080] In some embodiments of the present application, the distance between the midpoint of the edge point cloud and the midpoint of the target point cloud can be used as the displacement information of the side hole relative to the three-dimensional structured light camera, and the midpoint of the target point cloud is the origin of the three-dimensional structured light camera coordinate system. Based on this, the displacement information of the side hole relative to the three-dimensional structured light camera can be determined in the following manner:

[0081] Determine the midpoint of the edge point cloud;

[0082] The position information of the midpoint is used as the displacement information of the side hole relative to the three-dimensional structured light camera.

[0083] In some embodiments of the present application, the midpoint of the edge point cloud may be determined using the following formula (3):

[0084]

[0085] In the formula, Represents the midpoint of the edge point cloud, and n represents the number of midpoints in the edge point cloud.

[0086] After obtaining the displacement information of the side hole relative to the 3D structured light camera Afterwards, you can Substituting into formula (2), we can obtain the attitude angle information R of the side hole relative to the 3D structured light camera.

[0087] After obtaining the attitude angle information R and the displacement information t of the side hole relative to the three-dimensional structured light camera, the transformation matrix T composed of the attitude angle information R and the displacement information t can be used as the first attitude information of the side hole relative to the three-dimensional structured light camera.

[0088] S17. Determine second posture information of the keyhole on the corner piece relative to the three-dimensional structured light camera based on the first posture information.

[0089] Because the container corner fitting is usually a standard fitting, the relative position relationship between the corner fitting side hole and the corner fitting lock hole can be regarded as a constant. Based on this, when the second posture information of the lock hole in the corner fitting relative to the three-dimensional structured light camera is determined based on the first posture information, the first relative position relationship between the lock hole and the side hole of the corner fitting can be obtained, and the first posture information of the lock hole relative to the three-dimensional structured light camera can be determined based on the posture angle information, the displacement information and the first relative position relationship.

[0090] As some examples, the first relative position relationship of the corner fitting lock hole with respect to the corner fitting side hole can be expressed as First attitude information Based on this, the second posture information of the keyhole relative to the three-dimensional structured light camera can be determined according to the following formula (4):

[0091]

[0092] In the formula, Indicates the second posture information of the keyhole relative to the 3D structured light camera.

[0093] The container corner fitting positioning method provided in this embodiment integrates the target detection of the RGB image and the three-dimensional reconstruction of the depth image, and uses the detection result of the corner fitting side hole in the RGB image to segment the target image area corresponding to the side hole in the depth image, and only performs three-dimensional reconstruction on the target image area, which can achieve high-precision container corner fitting side hole positioning, thereby achieving high-precision corner fitting lock hole pose estimation. Compared with the positioning method based on the binocular camera, the complexity of the three-dimensional reconstruction and the amount of reconstruction calculation are reduced, the real-time positioning is improved, and there is no need to consider the lighting conditions, which improves the robustness. Compared with the positioning method based on the laser rangefinder, there is no need to place the container in a specific position, the dependence on the scene is small, and the cost is low.

[0094] In some embodiments, in the case where a locking pin is installed on the corner piece, the locking pin needs to be removed. In order to accurately remove the locking pin, the locking pin installed on the corner piece can be positioned. Based on this, after the above step S11, the following steps can also be performed:

[0095] Determine whether a locking pin is installed on the corner piece based on the RGB image.

[0096] In some embodiments of the present application, after obtaining the RGB image, the RGB image can be detected by a target detection model to determine whether it contains a locking pin. If it is determined that the locking pin is contained, it is determined that a locking pin is installed on the corner piece; otherwise, it is determined that the locking pin is not installed on the corner piece.

[0097] Furthermore, in some embodiments of the present application, when it is determined that a locking pin is installed on the corner fitting, a deep learning model can also be used to detect the RGB image to determine the detection frame of the locking pin and the lock pin category corresponding to the detection frame. The identified lock pin category can be further used to guide the robotic arm in the container lock pin automated loading and unloading system to replace the corresponding lock pin fixture.

[0098] Accordingly, after determining the second posture information of the keyhole in the corner piece relative to the three-dimensional structured light camera based on the first posture information, the following steps may also be performed:

[0099] When a locking pin is installed on the corner piece, obtaining a second relative position relationship between the locking pin and the locking hole;

[0100] Based on the first posture information and the second relative position relationship, third posture information of the locking pin relative to the three-dimensional structured light camera is determined.

[0101] In some embodiments of the present application, when the lock pin is installed on the lock hole, the second relative position relationship between the lock pin and the lock hole is also relatively fixed, which can be recorded as Based on this, the third posture information of the locking pin relative to the three-dimensional structured light camera can be determined according to the following formula (5):

[0102]

[0103] In the formula, Indicates the third posture information of the locking pin relative to the 3D structured light camera.

[0104] In this way, the locking pin installed on the corner piece can be accurately positioned, making it easier to remove the locking pin from the corner piece accurately.

[0105] Based on the container corner fitting positioning method provided in the above embodiment, the present application also provides a specific implementation of a container corner fitting positioning device. Please refer to the following embodiment.

[0106] See also Figure 2 The container corner fitting positioning device 200 provided in the embodiment of the present application may include the following modules:

[0107] The shooting module 201 is used to shoot the target area on the container through a monocular camera and a 3D structured light camera respectively to obtain an RGB image and a depth image corresponding to the target area, wherein the target area includes corner fittings;

[0108] The target detection module 202 is used to perform target detection processing on the RGB image to obtain a side hole detection frame corresponding to the side hole on the corner piece;

[0109] The region determination module 203 is used to determine the target image region corresponding to the side hole detection frame in the depth image;

[0110] A three-dimensional reconstruction module 204, configured to convert the target image region into a first three-dimensional point cloud;

[0111] An edge extraction module 205 is used to extract an edge point cloud of the side hole from the first three-dimensional point cloud;

[0112] A positioning module 206, configured to determine first posture information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud;

[0113] The positioning module 206 is further configured to determine second posture information of the lock hole on the corner piece relative to the three-dimensional structured light camera based on the first posture information.

[0114] The container corner fitting positioning device of the embodiment of the present application integrates the target detection of the RGB image and the three-dimensional reconstruction of the depth image, and uses the detection result of the corner fitting side hole in the RGB image to segment the target image area corresponding to the side hole in the depth image, and only performs three-dimensional reconstruction on the target image area, which can achieve high-precision positioning of the container corner fitting side hole, thereby achieving high-precision position estimation of the corner fitting lock hole. Compared with the positioning method based on the binocular camera, the complexity of the three-dimensional reconstruction and the amount of reconstruction calculation are reduced, the real-time positioning is improved, and there is no need to consider the lighting conditions, which improves the robustness. Compared with the positioning method based on the laser rangefinder, there is no need to place the container in a specific position, the dependence on the scene is small, and the cost is low.

[0115] In some embodiments, the region determination module 203 is configured to:

[0116] Determine the first position information of the side hole detection frame in the monocular camera coordinate system;

[0117] The first position information is coordinate-converted based on the internal parameters of the monocular camera, the internal parameters of the three-dimensional structured light camera, and the external parameters of the three-dimensional structured light camera relative to the monocular camera to obtain the second position information of the side hole detection frame in the three-dimensional structured light camera coordinate system;

[0118] An image area in the depth image corresponding to the second position information is used as a target image area.

[0119] In some embodiments, the edge extraction module 205 is used to:

[0120] Downsampling the first three-dimensional point cloud to obtain a second three-dimensional point cloud;

[0121] Remove noise points in the second three-dimensional point cloud to obtain a third three-dimensional point cloud;

[0122] Extract the edge point cloud of the side hole from the third 3D point cloud.

[0123] In some embodiments, the positioning module 206 includes:

[0124] A point cloud registration unit is used to perform point cloud registration on the edge point cloud and the target point cloud to obtain the attitude angle information and displacement information of the side hole relative to the three-dimensional structured light camera. The target point cloud is used to represent the edge point cloud when the side hole is facing the three-dimensional structured light camera.

[0125] The determination unit is used to determine the attitude angle information and the displacement information as the first attitude information of the side hole relative to the three-dimensional structured light camera.

[0126] In some embodiments, the point cloud registration unit is used to:

[0127] Determine the midpoint of the edge point cloud;

[0128] The position information of the midpoint is used as the displacement information of the side hole relative to the three-dimensional structured light camera;

[0129] The edge point cloud and the target point cloud are registered based on the nearest neighbor iterative algorithm and displacement information to obtain the attitude angle information of the side hole relative to the three-dimensional structured light camera.

[0130] In some embodiments, the positioning module 206 is used to:

[0131] Acquire a first relative position relationship between the lock hole and the side hole of the corner piece;

[0132] Based on the first posture information and the first relative position relationship, second posture information of the keyhole relative to the three-dimensional structured light camera is determined.

[0133] In some embodiments, the target detection module 202 is further configured to:

[0134] determining whether a locking pin is installed on the corner piece based on the RGB image;

[0135] The positioning module 206 is further used for:

[0136] When a locking pin is installed on the corner piece, obtaining a second relative position relationship between the locking pin and the locking hole;

[0137] Based on the second posture information and the second relative position relationship, third posture information of the locking pin relative to the three-dimensional structured light camera is determined.

[0138] The container corner fitting positioning device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0139] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0140] The electronic device 300 may include a processor 301 and a memory 302 storing computer program instructions.

[0141] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0142] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 302 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 302 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 302 is a non-volatile solid-state memory. The memory 302 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Therefore, generally, the memory 302 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it may perform the operations described in any one of the container corner fitting positioning methods in the above-mentioned embodiments.

[0143] The processor 301 reads and executes computer program instructions stored in the memory 302 to implement any one of the container corner fitting positioning methods in the above embodiments.

[0144] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0145] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0146] Bus 310 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0147] In addition, in combination with the container corner fitting positioning method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any container corner fitting positioning method in the above embodiment is implemented.

[0148] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0149] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0150] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0151] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A container corner fitting positioning method, characterized in that: include: The target area on the container is photographed by a monocular camera and a three-dimensional structured light camera respectively to obtain an RGB image and a depth image corresponding to the target area, wherein the target area includes corner fittings; Performing target detection processing on the RGB image to obtain a side hole detection frame corresponding to the side hole on the corner piece; Determine a target image area corresponding to the side hole detection frame in the depth image; Converting the target image area into a first three-dimensional point cloud; Extracting an edge point cloud of the side hole from the first three-dimensional point cloud; Determine first posture information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud; Based on the first posture information, second posture information of the keyhole on the corner piece relative to the three-dimensional structured light camera is determined.

2. The container corner fitting positioning method according to claim 1, characterized in that: The determining of the target image area corresponding to the side hole detection frame in the depth image includes: Determine first position information of the side hole detection frame in the monocular camera coordinate system; Performing coordinate transformation on the first position information based on the internal parameters of the monocular camera, the internal parameters of the three-dimensional structured light camera, and the external parameters of the three-dimensional structured light camera relative to the monocular camera to obtain second position information of the side hole detection frame in the three-dimensional structured light camera coordinate system; An image area in the depth image corresponding to the second position information is used as a target image area.

3. The container corner fitting positioning method according to claim 1, characterized in that: The step of extracting the edge point cloud of the side hole from the first three-dimensional point cloud comprises: Downsampling the first three-dimensional point cloud to obtain a second three-dimensional point cloud; removing noise points in the second three-dimensional point cloud to obtain a third three-dimensional point cloud; The edge point cloud of the side hole is extracted from the third three-dimensional point cloud.

4. The container corner fitting positioning method according to claim 1, characterized in that: The determining, based on the edge point cloud, first posture information of the side hole relative to the three-dimensional structured light camera includes: Performing point cloud registration on the edge point cloud and the target point cloud to obtain attitude angle information and displacement information of the side hole relative to the three-dimensional structured light camera, wherein the target point cloud is used to characterize the edge point cloud when the side hole is facing the three-dimensional structured light camera; The attitude angle information and the displacement information are determined as first attitude information of the side hole relative to the three-dimensional structured light camera.

5. The container corner fitting positioning method according to claim 4, characterized in that: The step of performing point cloud registration on the edge point cloud and the target point cloud to obtain attitude angle information and displacement information of the side hole relative to the three-dimensional structured light camera includes: Determining the midpoint of the edge point cloud; Using the position information of the midpoint as the displacement information of the side hole relative to the three-dimensional structured light camera; Point cloud registration is performed on the edge point cloud and the target point cloud based on a nearest neighbor iterative algorithm and the displacement information to obtain attitude angle information of the side hole relative to the three-dimensional structured light camera.

6. The container corner fitting positioning method according to claim 1, characterized in that: The determining, based on the first posture information, first posture information of the keyhole on the corner piece relative to the three-dimensional structured light camera comprises: Acquire a first relative position relationship between the lock hole of the corner piece and the side hole; Based on the first posture information and the first relative position relationship, second posture information of the keyhole relative to the three-dimensional structured light camera is determined.

7. The container corner fitting positioning method according to claim 1, characterized in that: After obtaining the RGB image and the depth image corresponding to the corner piece, the method further includes: determining whether a locking pin is installed on the corner piece based on the RGB image; After determining the first posture information of the keyhole on the corner piece relative to the three-dimensional structured light camera based on the first posture information, the method further includes: When a locking pin is installed on the corner piece, obtaining a second relative position relationship between the locking pin and the locking hole; Based on the second posture information and the second relative position relationship, third posture information of the locking pin relative to the three-dimensional structured light camera is determined.

8. A container corner fitting positioning device, characterized in that: include: A shooting module is used to shoot a target area on the container using a monocular camera and a three-dimensional structured light camera respectively to obtain an RGB image and a depth image corresponding to the target area, wherein the target area includes corner fittings; An object detection module is used to perform object detection processing on the RGB image to obtain a side hole detection frame corresponding to the side hole on the corner piece; A region determination module, used to determine a target image region in the depth image corresponding to the side hole detection frame; A three-dimensional reconstruction module, used for converting the target image area into a first three-dimensional point cloud; An edge extraction module, used for extracting an edge point cloud of the side hole from the first three-dimensional point cloud; A positioning module, used to determine first posture information of the side hole relative to the three-dimensional structured light camera based on the edge point cloud; The positioning module is further used to determine second posture information of the lock hole on the corner piece relative to the three-dimensional structured light camera based on the first posture information.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the container corner fitting positioning method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the container corner fitting positioning method according to any one of claims 1 to 7 is implemented.

Images