Container positioning method and device, container accessing equipment and storage medium

Abstract

The application provides a container positioning method and device, a container access equipment and a storage medium. The container positioning method comprises the following steps: obtaining a container image obtained by image acquisition of a target container by a vision sensor; performing edge detection on the container image, determining a plurality of edge line intersection points of the target container, and performing contour detection on the container image, determining a plurality of contour vertexes of the target container; verifying the plurality of edge line intersection points and the plurality of contour vertexes, and determining a target vertex of the target container; and determining a positioning result of the target container according to the target vertex. The container image collected by the vision sensor is subjected to edge detection and contour detection, and a plurality of edge line intersection points and a plurality of contour vertexes of the target container are obtained. The plurality of edge line intersection points and the plurality of contour vertexes are verified, the target vertex of the target container is determined, the positioning result of the target container is determined according to the target vertex, the positioning result of the target container is more accurate, and the positioning accuracy of the target container is improved.

Description

Technical Field

[0001] This invention relates to the field of warehousing technology, and in particular to a container positioning method, apparatus, container storage and retrieval equipment, and storage medium. Background Technology

[0002] In recent years, with the rapid development of e-commerce, the number of user orders has increased exponentially. A warehouse needs to store a massive amount of goods, and how to automatically sort these goods has become the key to improving warehousing efficiency.

[0003] Currently, warehouses mostly use container retrieval equipment to automate the sorting of goods. This equipment retrieves the target container containing the goods from the shelf to be sorted and then moves it to the designated location. To enable the automated retrieval function, the equipment is generally equipped with a depth camera. Based on the images captured by the depth camera, the target container is located, and the container is retrieved accordingly based on the location result.

[0004] However, container localization technology based on depth cameras has high requirements for the material of the target container, the surface integrity, and the environment in which the target container is located. If the target container is made of reflective / light-absorbing material or has a large area of ​​surface damage, accurate localization results cannot be obtained. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a container positioning method, apparatus, container access device, and storage medium to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the present invention, a container positioning method is provided, comprising:

[0007] Acquire container images obtained by a vision sensor from image acquisition of the target container;

[0008] Edge detection is performed on the container image to determine the intersection points of multiple edge lines of the target container, and contour detection is performed on the container image to determine multiple contour vertices of the target container.

[0009] Verify multiple edge line intersections and multiple contour vertices to determine the target vertices of the target container;

[0010] Based on the target vertex, determine the positioning result of the target container.

[0011] Optionally, the step of performing edge detection on the container image to determine the intersection points of multiple edge lines of the target container includes:

[0012] Edge detection is performed on the container image to obtain the target edge detection image, which includes multiple edge lines of the target container in the container image;

[0013] The intersection of multiple edge lines in the target edge detection image is identified as the intersection of multiple edge lines of the target container.

[0014] Optionally, the step of performing edge detection on the container image to obtain the target edge detection image includes:

[0015] Edge detection is performed on the container image to obtain an initial edge detection image;

[0016] Each edge line in the initial edge detection image is fitted to obtain the target edge detection image.

[0017] Optionally, the step of performing edge detection on the container image to obtain an initial edge detection image includes:

[0018] Gradient calculation is performed on the container image to obtain the initial edge detection image.

[0019] Optionally, the step of performing contour detection on the container image to determine multiple contour vertices of the target container includes:

[0020] Contour detection is performed on the container image to obtain the target contour detection image, wherein the target contour detection image includes the contour graphic of the target container in the container image;

[0021] The vertices of the contour figures in the target contour detection image are identified as multiple contour vertices of the target container.

[0022] Optionally, the step of performing contour detection on the container image to obtain a target contour detection image includes:

[0023] Perform contour detection on the container image to obtain an initial contour detection image;

[0024] The target contour detection image is obtained by fitting the contour pattern in the initial contour detection image.

[0025] Optionally, the step of performing contour detection on the container image to obtain an initial contour detection image includes:

[0026] The container image is binarized to obtain the initial contour detection image.

[0027] Optionally, the step of verifying multiple edge line intersections and multiple contour vertices to determine the target vertex of the target container includes:

[0028] Determine the intersection of the first edge lines and the first contour vertex located at the same position;

[0029] Obtain the pixel information of each first pixel within a preset range of the intersection of the first edge lines and the pixel information of each second pixel within a preset range of the first contour vertex;

[0030] The target vertex of the target container is determined based on the pixel information of each first pixel and each second pixel.

[0031] Optionally, the step of determining the intersection of the first edge lines and the first contour vertex located at the same position includes:

[0032] Identify preset markers in the container image, wherein the preset markers are pre-set at each vertex of the target container;

[0033] Calculate the first distance between the intersection of multiple edge lines and the preset marker, and the second distance between multiple contour vertices and the preset marker;

[0034] Based on the first distance and the second distance, determine the intersection of the first edge lines and the first contour vertex located at the same position.

[0035] Optionally, the step of determining the target vertex of the target container based on the pixel information of each first pixel and the pixel information of each second pixel includes:

[0036] Based on the pixel information of each first pixel, the grayscale distribution of the first pixel within a preset range of the intersection of the first edge lines is determined, and based on the pixel information of each second pixel, the grayscale distribution of the second pixel within a preset range of the first contour vertex is determined.

[0037] Determine the target pixel grayscale distribution that conforms to the preset distribution rules from the grayscale distribution of the first pixel and the grayscale distribution of the second pixel;

[0038] Based on the grayscale distribution of the target pixels, the target vertices of the target container are determined from the intersection of the first edge lines and the first contour vertices.

[0039] Optionally, the outline shape of the target container is quadrilateral;

[0040] Accordingly, the step of determining the positioning result of the target container based on the target vertex includes:

[0041] Obtain vertex information for any three target vertices from the four target vertices of the target container;

[0042] Based on the vertex information of the three target vertices, a preset perspective projection algorithm is used to calculate the pose information of the target container relative to the visual sensor.

[0043] Optionally, after the step of calculating the pose information of the target container relative to the visual sensor using a preset perspective projection algorithm based on the vertex information of the three target vertices, the method further includes:

[0044] Based on the pose information, the remaining target vertices, excluding the three target vertices, are projected onto the container image to obtain the projected coordinates of the remaining target vertices.

[0045] Obtain the vertex coordinates of the remaining target vertices in the container image;

[0046] Calculate the error value based on the projected coordinates and vertex coordinates;

[0047] Based on the error value, update the pose information and return to the step of projecting the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image to obtain the projected coordinates of the remaining target vertices.

[0048] If the error value is less than or equal to a preset threshold, the updated pose information is determined to be the target pose information of the target container relative to the vision sensor.

[0049] Optionally, before performing edge detection on the container image to determine the intersection points of multiple edge lines of the target container, and performing contour detection on the container image to determine multiple contour vertices of the target container, the method further includes:

[0050] The container image is filtered to obtain a noise-removed container image.

[0051] Optionally, the target container is a cargo box.

[0052] According to a second aspect of the present invention, a container positioning device is provided, comprising:

[0053] The acquisition module is configured to acquire container images obtained by the vision sensor from image acquisition of the target container;

[0054] The detection module is configured to perform edge detection on the container image to determine the intersection points of multiple edge lines of the target container, and to perform contour detection on the container image to determine multiple contour vertices of the target container.

[0055] The verification module is configured to verify multiple edge line intersections and multiple contour vertices to determine the target vertices of the target container;

[0056] The localization result determination module is configured to determine the localization result of the target container based on the target vertex.

[0057] Optionally, the detection module is further configured to perform edge detection on the container image to obtain a target edge detection image, wherein the target edge detection image includes multiple edge lines of the target container in the container image; and to identify the intersection points of the multiple edge lines in the target edge detection image as the multiple edge line intersection points of the target container.

[0058] Optionally, the detection module is further configured to perform edge detection on the container image to obtain an initial edge detection image; and to fit each edge line in the initial edge detection image to obtain a target edge detection image.

[0059] Optionally, the detection module is further configured to perform gradient calculation on the container image to obtain an initial edge detection image.

[0060] Optionally, the detection module is further configured to perform contour detection on the container image to obtain a target contour detection image, wherein the target contour detection image includes the contour graphic of the target container in the container image; and to identify the vertices of the contour graphic in the target contour detection image as multiple contour vertices of the target container.

[0061] Optionally, the detection module is further configured to perform contour detection on the container image to obtain an initial contour detection image; and to fit the contour pattern in the initial contour detection image to obtain a target contour detection image.

[0062] Optionally, the detection module is further configured to binarize the container image to obtain an initial contour detection image.

[0063] Optionally, the verification module is further configured to determine the intersection of the first edge lines and the first contour vertex located at the same position; obtain the pixel information of each first pixel point within a preset range of the intersection of the first edge lines and the pixel information of each second pixel point within a preset range of the first contour vertex; and determine the target vertex of the target container based on the pixel information of each first pixel point and the pixel information of each second pixel point.

[0064] Optionally, the verification module is further configured to identify a preset identifier in the container image, wherein the preset identifier is set in advance at each vertex of the target container; calculate a first distance between multiple edge line intersections and the preset identifier, and a second distance between multiple contour vertices and the preset identifier; and determine a first edge line intersection and a first contour vertex located at the same position based on the first distance and the second distance.

[0065] Optionally, the verification module is further configured to determine the grayscale distribution of the first pixel within a preset range of the intersection of the first edge lines based on the pixel information of each first pixel, and to determine the grayscale distribution of the second pixel within a preset range of the first contour vertices based on the pixel information of each second pixel; to determine the grayscale distribution of the target pixel that conforms to a preset distribution rule from the grayscale distribution of the first pixel and the grayscale distribution of the second pixel; and to determine the target vertex of the target container from the intersection of the first edge lines and the first contour vertices based on the grayscale distribution of the target pixel.

[0066] Optionally, the outline shape of the target container is quadrilateral;

[0067] Accordingly, the localization result determination module is further configured to obtain the vertex information of any three target vertices from the four target vertices of the target container; and to calculate the pose information of the target container relative to the visual sensor based on the vertex information of the three target vertices and a preset perspective projection algorithm.

[0068] Optionally, the localization result determination module is further configured to: project the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image based on the pose information to obtain the projected coordinates of the remaining target vertices; obtain the vertex coordinates of the remaining target vertices in the container image; calculate the error value based on the projected coordinates and vertex coordinates; update the pose information based on the error value, and return to execute the step of projecting the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image based on the pose information to obtain the projected coordinates of the remaining target vertices; and determine the updated pose information as the target pose information of the target container relative to the visual sensor if the error value is less than or equal to a preset threshold.

[0069] Optionally, the device further includes:

[0070] The filtering module is configured to filter the container image to obtain a container image with noise removed.

[0071] Optionally, the target container is a cargo box.

[0072] According to a third aspect of the present invention, a container access device is provided, comprising: a vision sensor, a memory, and a processor;

[0073] The vision sensor is used to acquire images of the target container and transmit the container images to the processor;

[0074] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which implement the container positioning method described above when executed by the processor.

[0075] According to a fourth aspect of the present invention, a computer-readable storage medium is provided that stores computer-executable instructions which, when executed by a processor, implement the container positioning method described above.

[0076] This embodiment provides a container positioning method, apparatus, container access device, and storage medium. The container positioning method includes: acquiring a container image obtained by a vision sensor capturing an image of a target container; performing edge detection on the container image to determine multiple intersection points of edge lines of the target container, and performing contour detection on the container image to determine multiple contour vertices of the target container; verifying the multiple intersection points of edge lines and the multiple contour vertices to determine the target vertex of the target container; and determining the positioning result of the target container based on the target vertex.

[0077] By performing edge detection and contour detection on the container image acquired by the vision sensor, multiple edge line intersections and multiple contour vertices of the target container are obtained. By verifying the multiple edge line intersections and multiple contour vertices, the target vertices of the target container are determined more accurately. Therefore, the positioning result of the target container is determined more accurately based on the target vertices, thus improving the positioning accuracy of the target container. Attached Figure Description

[0078] Figure 1 This is a flowchart of a container positioning method provided in one embodiment of the present invention;

[0079] Figure 2 This is a schematic diagram of the front end outline of a target container provided in one embodiment of the present invention;

[0080] Figure 3 This is a projected view of a target container provided in one embodiment of the present invention;

[0081] Figure 4 This is a flowchart of another container positioning method provided in one embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram of the structure of a container positioning device provided in one embodiment of the present invention;

[0083] Figure 6 This is a structural block diagram of a container access device provided in one embodiment of the present invention. Detailed Implementation

[0084] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0085] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0086] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0087] First, the terminology used in one or more embodiments of the present invention will be explained.

[0088] Containers: also commonly known as bins or cargo boxes, are physical containers for holding goods / materials, including plastic boxes, cartons, cardboard boxes, plastic crates, etc.

[0089] Container retrieval equipment: also known as C-robot (Carry robot), is an automated device that can pick up and move containers from shelves.

[0090] QR code: also known as two-dimensional barcode, the most common QR code is Quick Response Code (QR Code), which is an encoding method that has become very popular on mobile devices in recent years. It can store more information and represent more data types than traditional barcodes.

[0091] ArUco code: a square code consisting of only black and white colors.

[0092] AprilTag code: Used in machine vision, it can be simply understood as a simplified version of a QR code, featuring long recognition distance, fast recognition speed, and high accuracy. However, it stores far less information than a QR code.

[0093] Vision sensors are the direct source of information for machine vision systems. They mainly consist of one or two image sensors, sometimes accompanied by light projectors and other auxiliary equipment. The primary function of a vision sensor is to acquire sufficient raw images for the machine vision system to process. Image sensors can be cameras, laser scanners, linear and area CCD cameras, TV cameras, or the latest digital cameras, etc.

[0094] Binarization is a technique that sets the grayscale value of pixels in an image to 0 or 255, thus presenting the entire image as a distinct black and white visual effect.

[0095] Currently, there are two main container localization schemes: depth camera-based and tag-based. Depth camera-based schemes directly obtain container location information by identifying container instances. Depth cameras include, but are not limited to, Time-of-Flight (TOF) cameras, active and passive binocular cameras, and RGB-D cameras. Tag-based schemes locate containers by identifying tags on them. Tag codes include, but are not limited to, QR codes, ARCIO codes, and AprilTag codes.

[0096] Container localization schemes based on depth cameras have certain requirements regarding the material and surface integrity of the container. First, when the container is made of reflective or light-absorbing material, or when there are reflective or light-absorbing adhesives on the container surface, TOF cameras and active binocular cameras cannot accurately acquire depth data, affecting the container's localization results. Second, when the container surface is not a complete plane, such as a grid-like surface or a container with large areas of damage, the depth camera will generate many fly-through points / noise at the grid edges. Third, most existing container edges are curved or not neat right-angled, and depth cameras will generate fly-through points / noise at these edges, reducing edge localization accuracy. Fourth, container localization based on RGB-D cameras mostly uses deep learning methods, which have high computational requirements, cannot achieve real-time localization, and are difficult to achieve stable high-precision localization at container edges. Fifth, existing depth cameras with high localization accuracy are generally expensive.

[0097] In container positioning solutions based on identification codes, firstly, it is necessary to affix identification codes to the containers in advance and maintain these codes afterward to ensure they are clean and undamaged; secondly, it is necessary to ensure that the position of the container relative to the identification code is relatively accurate. If the container is offset or rotated relative to the identification code, the position of the identified identification code will not reflect the true position of the container; thirdly, for high-density warehousing scenarios with numerous storage locations, a large number of identification codes are required, resulting in high costs.

[0098] To address the aforementioned problems, the present invention provides a container positioning method, a container positioning device, a container access device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0099] The execution subject of the container positioning method provided in this embodiment of the invention can be a computing device, server, or other device used for container positioning, or a container access device used for automatic container access. The container positioning method provided in this embodiment of the invention can be implemented in at least one of the following: software, hardware circuitry, or logic circuitry within the execution subject.

[0100] Figure 1 The flowchart illustrates a container positioning method according to an embodiment of the present invention, which may specifically include the following steps.

[0101] Step 102: Obtain the container image obtained by the vision sensor acquiring the image of the target container.

[0102] In this embodiment of the invention, the vision sensor can be a common 2D vision sensor. Compared to depth cameras, common 2D vision sensors are lower in cost and have better versatility in terms of container material. Of course, in this embodiment of the invention, the vision sensor is not limited to a common 2D vision sensor; any vision sensor capable of acquiring images of the target container and with low requirements for the container material can be used.

[0103] The visual sensor acquires images of the scene within its field of view. The field of view must include the target container, so the acquired container image must include the target container. Then, the acquired container image is transmitted to the execution subject of this embodiment of the invention so that the execution subject can locate the target container based on the container image.

[0104] In practical applications, the field of view of a vision sensor must encompass the entire front outline of the target container. Specifically, this can be ensured in three ways: First, after acquiring the image from the vision sensor, the number of container vertices in the image is identified. It is first determined whether the container has four vertices. If the number is less than four, the positional relationship between the container's edge and the image edge is identified. If the positional relationship indicates that the container exceeds the field of view, a control signal is sent to the container access device to move it in the direction of exceeding the field of view. Second, a vision sensor with a large field of view, such as a fisheye camera, can be directly used. Third, after acquiring the image, it can be displayed to the administrator, who can then determine whether the container exceeds the field of view. If it does, the container access device is manually moved in the direction of exceeding the field of view.

[0105] Step 104: Perform edge detection on the container image to determine the intersection points of multiple edge lines of the target container, and perform contour detection on the container image to determine multiple contour vertices of the target container.

[0106] After acquiring the container image collected by the vision sensor, it is necessary to identify and locate the target container in the container image. However, in order to ensure applicability, a common 2D vision sensor can be used. The vision sensor does not collect rich image information (such as depth data, color data, etc.). However, regardless of the type of vision sensor, it can collect the edge and contour of the target container. Therefore, in order to adapt to various types of vision sensors, this embodiment of the invention uses edge detection and contour detection to locate the target container.

[0107] While edge detection or contour detection alone can achieve target container localization, this method has poor accuracy. To improve the accuracy of target container localization, this embodiment of the invention uses a combination of edge detection and contour detection for container localization.

[0108] Edge detection aims to identify points in an image with significant grayscale changes. In container images used in warehousing, these points often represent container edges. Therefore, edge detection on a container image can reveal the edge lines of the target container. After detecting multiple edge lines, their intersection points can be calculated to obtain the intersection points of the target container's edge lines. In some embodiments, edge detection can directly identify the edge lines of the target container within the container image, thereby directly obtaining the intersection points of these edge lines.

[0109] In one implementation of this invention, step 104, which involves edge detection of the container image to determine the intersection points of multiple edge lines of the target container, can be implemented as follows: edge detection is performed on the container image to obtain a target edge detection image, wherein the target edge detection image includes multiple edge lines of the target container in the container image; the intersection points of the multiple edge lines in the target edge detection image are identified as the intersection points of multiple edge lines of the target container.

[0110] In this embodiment of the invention, edge detection can be performed directly on the container image to obtain a target edge detection image. This image emphasizes multiple edge lines of the target container within the container image (either by displaying only the edge lines, or by making them thicker or more emphasized). This allows for direct identification of the intersections of these edge lines, thus obtaining the multiple edge line intersections of the target container. This edge detection method enables the rapid determination of multiple edge line intersections of the target container within the container image, improving the efficiency of container localization.

[0111] In one implementation of this invention, the step of performing edge detection on a container image to obtain a target edge detection image can be specifically implemented as follows: performing edge detection on the container image to obtain an initial edge detection image; fitting each edge line in the initial edge detection image to obtain the target edge detection image.

[0112] Generally, due to factors such as the image acquisition angle and lighting, the acquired container image will have slight errors compared to the actual image. Furthermore, different edge detection methods can also lead to discrepancies between the detected edge lines and the actual ones. To ensure a more accurate target edge detection image and thus precise target container positioning, it is necessary to fit each edge line in the initial edge detection image separately.

[0113] Since containers are generally quadrilaterals, meaning the edges of the target container are typically straight lines, a linear fitting method can be used when fitting each edge line individually. There are many linear fitting methods, such as least squares, Hough transform, gradient descent, and univariate linear regression. In this embodiment, the specific method used for fitting is not limited, as long as it achieves the goal of edge line fitting. Of course, in some special scenarios, the container may not be quadrilateral. For non-quadrilateral containers, when fitting the edge lines, if they are curves, a corresponding curve fitting method can be used, or the curve can be divided into multiple straight lines before using a linear fitting method.

[0114] In one implementation of this invention, the step of performing edge detection on a container image to obtain an initial edge detection image can be specifically implemented as follows: performing gradient calculation on the container image to obtain the initial edge detection image.

[0115] As mentioned above, the purpose of edge detection is to detect points in an image where there are significant changes in grayscale. How do we define whether such changes are significant? We can use the derivative (gradient) to measure the rate of change of grayscale in an image. Therefore, edge detection can be achieved by calculating the gradient of the container image. Gradient calculation can quickly and accurately calculate the grayscale changes in an image, improving the accuracy and efficiency of edge detection.

[0116] The gradient expression of the image function f(x,y) is shown in Equation (1):

[0117]

[0118] The amplitude is shown in formula (2):

[0119]

[0120] The direction angle is shown in formula (3):

[0121]

[0122] For images, this is equivalent to finding the gradient of a two-dimensional discrete function, using the difference to approximate the derivative, as shown in formula (4):

[0123]

[0124] Therefore, the gradient value at pixel (x,y) is shown in Equation (5), and the gradient direction is shown in Equation (6):

[0125]

[0126]

[0127] From the above deduction, we find that the direction of the gradient is the direction of the fastest change of the function. Therefore, when there are edges in the function, there must be a large gradient value. Conversely, when there are relatively smooth parts in the image, the gray value changes less, and the corresponding gradient is also smaller. In image processing, the magnitude of the gradient is simply called the gradient, and the image composed of the image gradient is called the gradient image.

[0128] Some classic image gradient calculations consider the gray-level changes within a neighborhood of each pixel in the image. Utilizing the variation of the first or second derivative of pixels near edges, gradient operators are applied to a neighborhood of pixels in the original image. This is typically done using convolution with a small region template, and includes operators such as Sobel, Robinson, and Laplace. In this embodiment of the invention, gradient operators can be directly applied to the container image. The horizontal gradient operator can be set to [1,1,1; 0,0,0; 1,1,1], and the vertical gradient operator can be set to [1,0,1; 1,0,1; 1,0,1]. By applying the horizontal and vertical gradient operators to the container image, G in the above formula can be obtained. x (x,y) and G y (x,y), further, the gradient value and gradient direction can be calculated, thus obtaining the gradient calculation result of the entire container image. Based on the gradient calculation result of the container image, the edge line of the target container in the container image can be obtained.

[0129] Contour detection refers to the process of extracting the contour of a target from an image containing both a target and a background, ignoring the influence of textures and noise interference within the background and the target. In this embodiment of the invention, contour detection is performed on a container image to obtain the contour of the target container within the container image. After detecting the contour of the target container, multiple contour vertices of the target container can be obtained by identifying the vertices of the contour shape.

[0130] In one implementation of this invention, step 104, which involves performing contour detection on the container image to determine multiple contour vertices of the target container, can be implemented as follows: performing contour detection on the container image to obtain a target contour detection image, wherein the target contour detection image includes the contour graphic of the target container in the container image; and identifying the vertices of the contour graphic in the target contour detection image as multiple contour vertices of the target container.

[0131] In this embodiment of the invention, contour detection can be performed directly on the container image to obtain a target contour detection image. This image highlights the contour of the target container, allowing direct identification of the vertices of the contour, thus obtaining multiple contour vertices of the target container. This contour detection method enables the rapid determination of multiple contour vertices of the target container within the container image, improving the efficiency of container localization.

[0132] In one implementation of this invention, the step of performing contour detection on a container image to obtain a target contour detection image can be specifically implemented as follows: performing contour detection on the container image to obtain an initial contour detection image; fitting the contour pattern in the initial contour detection image to obtain the target contour detection image.

[0133] Generally, due to factors such as the image acquisition angle and lighting, the acquired container image will have slight errors compared to the actual object. Furthermore, different contour detection methods can also lead to discrepancies between the detected contour and the actual object. To ensure a more accurate target contour detection image and thus precise target container positioning, it is necessary to fit the contour pattern in the initial contour detection image.

[0134] Since containers are generally quadrilaterals, quadrilateral fitting can be used when fitting the contour shape. Quadrilateral fitting involves fitting each side of the collected contour shape to make it more closely resemble an actual quadrilateral. Of course, in some special scenarios, the container may not be quadrilateral. For polygonal containers, appropriate polygon fitting methods can be used to fit the contour shape.

[0135] In practical implementation, polygon fitting methods provided by open-source cross-platform computer vision and machine learning software libraries (such as OpenCV) can be used to fit the contour graphics in the initial contour detection image. OpenCV provides several methods for quadrilateral fitting, including minimum envelope rectangle, minimum envelope rectangle, quadrilateral obtained by fitting using boundary points, and minimum envelope quadrilateral. These methods are all designed based on the algorithmic idea of ​​the iterative endpoint fitting algorithm (also known as the Douglas-Puk algorithm).

[0136] In one implementation of this invention, the step of performing contour detection on a container image to obtain an initial contour detection image can be specifically implemented as follows: performing binarization processing on the container image to obtain the initial contour detection image.

[0137] In this embodiment of the invention, contour detection can be performed using binarization processing, specifically by binarizing the container image to obtain an initial contour detection image. Adaptive binarization can be used to determine whether a pixel in the container image is a darker or brighter region within its vicinity. This is achieved by comparing the pixel value of that pixel with the average or weighted average of the pixel values ​​of the surrounding region, thus obtaining a binarized image (i.e., the initial contour detection image). The binarized image exhibits a clear black-and-white visual effect, clearly distinguishing the coverage area of ​​the target container from its surrounding areas. Therefore, the coverage area of ​​the target container can be identified from the binarized image. When binarizing the container image, since the lighting conditions in different parts of the container image may be uneven, the image can be binarized in blocks using multiple thresholds, with different thresholds applied to each image block. This binarization method improves the efficiency and accuracy of locating the target container.

[0138] In one implementation of this invention, before performing step 104, the following step may also be performed: filtering the container image to obtain a noise-removed container image.

[0139] Due to environmental influences, container images acquired by visual sensors often contain noise. Image noise refers to unnecessary or redundant interference information present in image data. The presence of noise severely affects image quality; therefore, noise removal is necessary before image processing. In this embodiment of the invention, noise removal is achieved by filtering the container image. Filtering methods mainly include bilateral filtering, median filtering, and Gaussian filtering. This embodiment of the invention preferentially uses bilateral filtering to filter the container image, resulting in better filtering performance and a noise-removed container image.

[0140] Step 106: Verify the intersections of multiple edge lines and multiple contour vertices to determine the target vertices of the target container.

[0141] After edge detection and contour detection obtain multiple edge intersections and contour vertices of the target container, the corresponding edge intersections and contour vertices should coincide. However, due to different detection methods, there are often errors between the obtained edge intersections and contour vertices. Therefore, it is necessary to verify multiple edge intersections and contour vertices. The verification process involves cross-verifying the edge intersections and contour vertices at the same location. Through cross-verification, a more accurate target vertex is obtained. Cross-verification can be achieved by averaging the positions of edge intersections and contour vertices at the same location, or by selecting the point whose pixel information more closely matches the actual situation as the target vertex based on the pixel information of edge intersections and contour vertices at the same location.

[0142] In one implementation of this invention, step 106 can be implemented as follows: determine the intersection of the first edge lines and the first contour vertex located at the same position; obtain the pixel information of each first pixel point within a preset range of the intersection of the first edge lines and the pixel information of each second pixel point within a preset range of the first contour vertex; determine the target vertex of the target container based on the pixel information of each first pixel point and the pixel information of each second pixel point.

[0143] When validating multiple edge line intersections and multiple contour vertices, the first edge line intersection and the first contour vertex located at the same position are first identified. Then, with the first edge line intersection as the center, the pixel information of each first pixel within a preset range of the first edge line intersection is obtained (usually, the pixel information of 8 pixels near the first edge line intersection can be obtained). With the first contour vertex as the center, the pixel information of each second pixel within a preset range of the first contour vertex is obtained. The pixel information mentioned here refers to the attribute data of the pixel, which can be pixel value, grayscale value, etc. The pixel information of each first pixel and each second pixel represents the pixel distribution characteristics around the first edge line intersection and the first contour vertex. In actual scenarios, the pixel distribution of the target vertex of the container has a certain pattern. Based on this pattern, the target vertex of the target container is determined according to the pixel information of each first pixel and each second pixel.

[0144] There are several ways to determine the target vertex. One approach is to determine which of the pixel information of each first pixel and each second pixel conforms to the aforementioned rule, thus determining whether to identify the intersection of the first edge line or the first contour vertex as the target vertex. Another approach is to perform a weighted sum of the pixel information of each first pixel and each second pixel, and then determine whether the weighted result conforms to the aforementioned rule. If not, the positions of the intersection of the first edge line and the first contour vertex are adjusted until the weighted result conforms to the aforementioned rule, thereby determining the target vertex.

[0145] In one implementation of this invention, the step of determining the first edge line intersection point and the first contour vertex located at the same position can be implemented as follows: identifying a preset identifier in the container image, wherein the preset identifier is set in advance at each vertex of the target container; calculating a first distance between multiple edge line intersection points and the preset identifier, and a second distance between multiple contour vertices and the preset identifier; and determining the first edge line intersection point and the first contour vertex located at the same position based on the first distance and the second distance.

[0146] To more accurately locate the intersection of the first edge lines and the first contour vertex at the same position, in this embodiment of the invention, preset markers can be pre-set at each vertex of the target container. These preset markers are geometric markers, such as... Figure 2 As shown, Figure 2 The diagram illustrates a schematic outline of the front end of a target container according to an embodiment of the present invention. Circular markers are positioned at the four vertices of the target container. Of course, the circular markers are merely one example; the preset markers can also be rectangles, triangles, etc., and are not specifically limited here. The preset markers are fixed at each vertex of the target container, maintaining a fixed position relative to the target container, simplifying detection.

[0147] When performing vertex lookup, the preset markers in the container image can be identified first. Then, the first distance between multiple edge line intersections and the preset markers, and the second distance between multiple contour vertices and the preset markers can be calculated. Under normal circumstances, the difference between the first distance between the edge line intersection and a preset marker and the second distance between the contour vertex and the preset marker is very small (less than a certain threshold), and the first distance and the second distance are also small. This indicates that the edge line intersection and the contour vertex are the edge line intersection and the contour vertex corresponding to the vertex where the preset marker is located. That is, based on the first distance and the second distance, the first edge line intersection and the first contour vertex located at the same position can be determined, and the actual vertex position can also be determined.

[0148] In one implementation of this invention, the step of determining the target vertex of the target container based on the pixel information of each first pixel and the pixel information of each second pixel can be implemented as follows: Based on the pixel information of each first pixel, determine the grayscale distribution of the first pixel within a preset range of the intersection of the first edge lines; based on the pixel information of each second pixel, determine the grayscale distribution of the second pixel within a preset range of the first contour vertex; from the grayscale distribution of the first pixel and the grayscale distribution of the second pixel, determine the target pixel grayscale distribution that conforms to a preset distribution rule; based on the target pixel grayscale distribution, determine the target vertex of the target container from the intersection of the first edge lines and the first contour vertex.

[0149] Based on the pixel information of each first pixel, the grayscale distribution of the first pixel within a preset range of the intersection of the first edge lines can be calculated. Similarly, based on the pixel information of each second pixel, the grayscale of the second pixel within a preset range of the first contour vertex can be calculated. Specifically, the pixel information can be grayscale values. By extracting the grayscale values ​​of each first pixel and each second pixel, the grayscale distribution of the first pixel and the grayscale distribution of the second pixel can be obtained. If it is an ideal vertex, the grayscale distribution around it should be such that three-quarters of the area has grayscale values ​​within a certain threshold range, and the remaining quarter has grayscale values ​​within another threshold range. Therefore, after obtaining the grayscale distribution of the first pixel and the grayscale distribution of the second pixel, the target pixel grayscale distribution that conforms to the preset distribution rules can be determined. That is, it is determined which of the first pixel grayscale distribution and the second pixel grayscale distribution satisfies the above distribution rules and is taken as the target pixel grayscale distribution. Then, based on the target pixel grayscale distribution, the target vertex of the target container is determined from the intersection of the first edge line and the first contour vertex. That is, if the grayscale distribution of the first pixel corresponding to the intersection of the first edge line satisfies the above distribution rule, the intersection of the first edge line is taken as the target vertex; if the grayscale distribution of the second pixel corresponding to the first contour vertex satisfies the above distribution rule, the first contour vertex is taken as the target vertex.

[0150] Step 108: Determine the positioning result of the target container based on the target vertex.

[0151] After obtaining the target vertex of the target container, the localization result of the target container can be determined based on the target vertex. The localization result of the target container can be determined by using the PnP (Perspective-n-Point) localization algorithm or other localization algorithms based on vision sensors. The localization result refers to the actual localization information of the target container relative to the vision sensor. The localization result can include information such as the size and pose of the target container.

[0152] In one implementation of this invention, the outline shape of the target container is a quadrilateral. Accordingly, step 108 can be implemented as follows: obtain the vertex information of any three target vertices from the four target vertices of the target container; and calculate the pose information of the target container relative to the visual sensor using a preset perspective projection algorithm based on the vertex information of the three target vertices.

[0153] Since the target container has a quadrilateral shape, the P3P localization algorithm can be used for quadrilateral localization. That is, when performing localization, the vertex information of any three of the four target vertices of the target container can be obtained. The vertex information can be vertex coordinates, vertex pixels, etc. Of course, the width and height prior information of the target container can also be obtained. Then, based on the vertex information of these three target vertices, the pose information of the target container relative to the vision sensor can be calculated by using a preset perspective projection algorithm (i.e., the P3P algorithm).

[0154] like Figure 3 As shown, Figure 3 The diagram shows a projected view of a target container according to an embodiment of the present invention, where ABCD are the four vertices of the front contour of the target container, O is the viewpoint of the vision sensor, and abcd are the four vertices obtained after the target container is projected onto the projection surface.

[0155] The calculation is performed using three vertices A, B, and C. The modulus of BC, AC, and AB (i.e., the prior information of the width and height of the target container) is known, as well as ∠AOC = ∠aOc, ∠BOC = ∠bOc, ∠AOB = ∠aOb, and a = (u0, v0, 1), b = (u1, v1, 1), c = (u2, v2, 1).

[0156] The three angles can be calculated according to formula (7), as shown in formula (8):

[0157]

[0158]

[0159] According to the Law of Cosines, the equation shown in formula (9) holds true:

[0160]

[0161] To simplify the calculation, variable substitutions are made as shown in formula (10) (all variables are transformed to be related to OC, thus eliminating variables):

[0162]

[0163] After variable substitution, it can be transformed into formula (11):

[0164]

[0165] Substituting v into the first two equations for further elimination, we obtain a system of two quadratic equations in two variables, as shown in formula (12):

[0166]

[0167] Where p, q, r, a, b are known quantities, and x, y are unknown quantities. Solving the equation using Wu's elimination method yields x and y, thus obtaining the coordinates of points A, B, and C in the coordinate system of the visual sensor. Establishing a three-dimensional world coordinate system based on the surface of the target container, the world coordinates of points A, B, and C are also known. The pose information of the target container relative to the visual sensor can then be calculated using the ICP algorithm. This pose information can be represented by the pose R matrix and the T matrix.

[0168] In one implementation of this invention, after calculating the pose information of the target container relative to the visual sensor using a preset perspective projection algorithm based on the vertex information of the three target vertices, the following steps can be performed: Based on the pose information, project the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image to obtain the projection coordinates of the remaining target vertices; obtain the vertex coordinates of the remaining target vertices in the container image; calculate the error value based on the projection coordinates and vertex coordinates; update the pose information based on the error value, and return to execute the step of projecting the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image based on the pose information to obtain the projection coordinates of the remaining target vertices; if the error value is less than or equal to a preset threshold, determine that the updated pose information is the target pose information of the target container relative to the visual sensor.

[0169] Based on the vertex information of any three target vertices, a preset perspective projection algorithm is used to calculate the pose information of the target container relative to the visual sensor. However, errors are inevitably introduced during the calculation process, leading to inaccurate pose information. Therefore, it is necessary to verify the pose information using the remaining target vertices (excluding the three target vertices). The specific pose verification process is as follows: Based on the pose information, the remaining target vertices are projected onto the container image to obtain the projected coordinates of the remaining target vertices. The error between the projected coordinates and the actual detected vertex coordinates of the remaining target vertices is calculated to obtain the error value. Then, using the principle of backpropagation of the error gradient, the pose information is updated based on the error value. This process is repeated until pose information with an error value less than or equal to a preset threshold is obtained. By using the remaining target vertices (excluding the three target vertices) to verify the pose information, the final determined pose information is more accurate, improving the accuracy of target container localization.

[0170] This invention employs an embodiment to acquire a container image obtained by a vision sensor capturing an image of a target container. Edge detection is performed on the container image to determine multiple intersection points of the target container's edge lines, and contour detection is performed on the container image to determine multiple contour vertices of the target container. The intersection points and contour vertices are then verified to determine the target vertices of the target container. Based on these target vertices, the positioning result of the target container is determined. By performing edge detection and contour detection on the container image acquired by the vision sensor, multiple intersection points of the target container's edge lines and multiple contour vertices are obtained. Verification of these intersection points and vertices leads to a more accurate determination of the target vertices, resulting in a more precise positioning result and improved positioning accuracy.

[0171] The target container in the above embodiments can specifically be a cargo box. For ease of understanding, the container positioning method provided in this embodiment of the invention will be described below in conjunction with the application scenario of cargo box retrieval and placement. In this embodiment of the invention, the container storage and retrieval device includes a cargo box retrieval and placement device, a 2D vision sensor is installed at the bottom of the retrieval and placement device, and the container storage device also includes a processor, which executes functions such as... Figure 4 The container positioning method shown is as follows: Figure 4 A flowchart of another container positioning method provided by an embodiment of the present invention is shown, which specifically includes the following steps.

[0172] The first step is to acquire images of the cargo box. A 2D vision sensor acquires images of the target cargo box, and the field of view of the 2D vision sensor must include the entire front outline of the cargo box.

[0173] The second step is image filtering. Bilateral filtering is performed on the cargo box image acquired by the 2D vision sensor in the first step to obtain image 1 with noise removed.

[0174] The third step is gradient calculation and adaptive binarization. The horizontal and vertical gradients of image 1 obtained in the second step are calculated to obtain image 2; and adaptive binarization is simultaneously performed on image 1 in parallel to obtain image 3.

[0175] The fourth step is edge detection and contour detection. Edge detection is performed on image 2 obtained in step three to obtain image 4, and contour detection is performed on image 3 to obtain image 5. Edge detection involves identifying pixels in image 2 where the gradient strength after gradient calculation exceeds a certain threshold, as the grayscale changes are more pronounced at edges, resulting in a larger gradient response. Contour detection involves recognizing the boundary contours of image 3.

[0176] Step 5: Line fitting and quadrilateral fitting. Line fitting is performed on the edge lines in image 4 obtained in step 4, resulting in image 6. Quadrilateral fitting is then performed on the quadrilateral contours in image 5, resulting in image 7. For details on the specific line fitting and quadrilateral fitting methods, please refer to [link to details]. Figure 1 The embodiments shown are not described in detail here.

[0177] Step 6: Determine the intersection of the lines and the vertices of the quadrilateral. This is based on the results obtained in step 5. Figure 6 ,right Figure 6 Find the intersection points of the straight lines in the image, and obtain multiple intersection points. Then, find the vertices of the quadrilateral contour in the image 7, and obtain multiple quadrilateral vertices.

[0178] Step 7: Verify the top of the cargo box.

[0179] For the multiple line intersections and quadrilateral vertices obtained in step six, firstly, determine the first line intersection and the first quadrilateral vertex located at the same position. Then, using the first line intersection as the center, obtain the grayscale value of each first pixel within a preset range of the first line intersection. Using the first contour vertex as the center, obtain the grayscale value of each second pixel within a preset range of the first contour vertex. Extract the grayscale values ​​of each first pixel and each second pixel to obtain the first pixel grayscale distribution and the second pixel grayscale distribution. If it is an ideal vertex, the grayscale distribution around it should be such that three-quarters of the area has grayscale values ​​within a certain threshold range, and the remaining quarter of the area has grayscale values ​​within another threshold range. Therefore, after obtaining the first pixel grayscale distribution and the second pixel grayscale distribution, the target pixel grayscale distribution that conforms to the preset distribution rule can be determined. That is, determine which of the first pixel grayscale distribution and the second pixel grayscale distribution can satisfy the above distribution rule and use it as the target pixel grayscale distribution. Then, based on the target pixel grayscale distribution, the target vertex of the cargo box is determined from the intersection of the first straight line and the vertex of the first quadrilateral. That is, if the grayscale distribution of the first pixel corresponding to the intersection of the first straight line satisfies the above distribution rule, the intersection of the first straight line is taken as the target vertex; if the grayscale distribution of the second pixel corresponding to the vertex of the first quadrilateral satisfies the above distribution rule, the vertex of the first quadrilateral is taken as the target vertex.

[0180] Step 8: Calculate the pose of the cargo box relative to the 2D vision sensor using the P3P algorithm based on three of the vertices. For the four vertices of the cargo box obtained in Step 7, calculate the pose of the cargo box relative to the 2D vision sensor using the P3P algorithm based on three of the vertices and the prior information of the cargo box's width and height. See [link to algorithm implementation details] for more information. Figure 1 and Figure 3 The embodiments shown are not described in detail here.

[0181] The ninth step is to use the remaining vertices for pose verification.

[0182] For the cargo container pose obtained in step eight, the remaining vertices are used for pose verification. Specifically, based on the pose information, the remaining vertices are projected onto the image to obtain their projected coordinates. An error value is calculated by comparing the projected coordinates with the actually detected vertex coordinates of the remaining vertex. Then, using the principle of backpropagation of error gradients, the pose information is updated based on the error value. This process is repeated until a pose information with an error value less than or equal to a preset threshold is obtained. By using the remaining vertices from the four target vertices to verify the pose information, the final determined pose information is more accurate, improving the accuracy of target container localization.

[0183] Step 10: Output the pose of the cargo box relative to the 2D vision sensor.

[0184] This invention uses a common 2D vision sensor, which does not rely on a depth camera, thus reducing costs and improving the versatility of cargo box materials. Furthermore, it can directly locate the cargo box without relying on the cargo location identification code, avoiding inaccurate actual cargo box position caused by changes in the relative position between the cargo box and the cargo location identification code, and can obtain information such as the actual angle and depth of the cargo box.

[0185] Corresponding to the above method embodiments, the present invention also provides a container positioning device embodiment. Figure 5 A schematic diagram of a container positioning device according to an embodiment of the present invention is shown. Figure 5 As shown, the device includes:

[0186] The acquisition module 520 is configured to acquire a container image obtained by the vision sensor from image acquisition of the target container;

[0187] The detection module 540 is configured to perform edge detection on the container image to determine the intersection points of multiple edge lines of the target container, and to perform contour detection on the container image to determine multiple contour vertices of the target container.

[0188] The verification module 560 is configured to verify multiple edge line intersections and multiple contour vertices to determine the target vertices of the target container;

[0189] The positioning result determination module 580 is configured to determine the positioning result of the target container based on the target vertex.

[0190] This invention employs an embodiment to acquire a container image obtained by a vision sensor capturing an image of a target container. Edge detection is performed on the container image to determine multiple intersection points of the target container's edge lines, and contour detection is performed on the container image to determine multiple contour vertices of the target container. The intersection points and contour vertices are then verified to determine the target vertices of the target container. Based on these target vertices, the positioning result of the target container is determined. By performing edge detection and contour detection on the container image acquired by the vision sensor, multiple intersection points of the target container's edge lines and multiple contour vertices are obtained. Verification of these intersection points and vertices leads to a more accurate determination of the target vertices, resulting in a more precise positioning result and improved positioning accuracy.

[0191] Optionally, the detection module 540 can be further configured to perform edge detection on the container image to obtain a target edge detection image, wherein the target edge detection image includes multiple edge lines of the target container in the container image; and identify the intersection points of the multiple edge lines in the target edge detection image as the multiple edge line intersection points of the target container.

[0192] Optionally, the detection module 540 can be further configured to perform edge detection on the container image to obtain an initial edge detection image; and to fit each edge line in the initial edge detection image to obtain a target edge detection image.

[0193] Optionally, the detection module 540 can be further configured to perform gradient calculation on the container image to obtain an initial edge detection image.

[0194] Optionally, the detection module 540 can be further configured to perform contour detection on the container image to obtain a target contour detection image, wherein the target contour detection image includes the contour graphic of the target container in the container image; and identify the vertices of the contour graphic in the target contour detection image as multiple contour vertices of the target container.

[0195] Optionally, the detection module 540 can be further configured to perform contour detection on the container image to obtain an initial contour detection image; and to fit the contour pattern in the initial contour detection image to obtain a target contour detection image.

[0196] Optionally, the detection module 540 can be further configured to binarize the container image to obtain an initial contour detection image.

[0197] Optionally, the verification module 560 can be further configured to determine the intersection of the first edge lines and the first contour vertex located at the same position; obtain the pixel information of each first pixel point within a preset range of the intersection of the first edge lines and the pixel information of each second pixel point within a preset range of the first contour vertex; and determine the target vertex of the target container based on the pixel information of each first pixel point and the pixel information of each second pixel point.

[0198] Optionally, the verification module 560 can be further configured to identify a preset identifier in the container image, wherein the preset identifier is set in advance at each vertex of the target container; calculate a first distance between multiple edge line intersections and the preset identifier, and a second distance between multiple contour vertices and the preset identifier; and determine a first edge line intersection and a first contour vertex located at the same position based on the first distance and the second distance.

[0199] Optionally, the verification module 560 can be further configured to determine the grayscale distribution of the first pixel within a preset range of the intersection of the first edge lines based on the pixel information of each first pixel, and to determine the grayscale distribution of the second pixel within a preset range of the first contour vertices based on the pixel information of each second pixel; to determine the grayscale distribution of the target pixel that conforms to the preset distribution rules from the grayscale distribution of the first pixel and the grayscale distribution of the second pixel; and to determine the target vertex of the target container from the intersection of the first edge lines and the first contour vertices based on the grayscale distribution of the target pixel.

[0200] Optionally, the outline shape of the target container is quadrilateral;

[0201] Accordingly, the positioning result determination module 580 can be further configured to obtain the vertex information of any three target vertices from the four target vertices of the target container; and calculate the pose information of the target container relative to the visual sensor based on the vertex information of the three target vertices and a preset perspective projection algorithm.

[0202] Optionally, the localization result determination module 580 can also be configured to: project the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image based on the pose information to obtain the projected coordinates of the remaining target vertices; obtain the vertex coordinates of the remaining target vertices in the container image; calculate the error value based on the projected coordinates and vertex coordinates; update the pose information based on the error value, and return to the step of projecting the remaining target vertices (excluding the three target vertices) from the four target vertices onto the container image based on the pose information to obtain the projected coordinates of the remaining target vertices; and determine the updated pose information as the target pose information of the target container relative to the visual sensor if the error value is less than or equal to a preset threshold.

[0203] Optionally, the device further includes:

[0204] The filtering module is configured to filter the container image to obtain a container image with noise removed.

[0205] Optionally, the target container is a cargo box.

[0206] The above is a schematic scheme of a container positioning device according to this embodiment. It should be noted that the technical solution of this container positioning device and the technical solution of the container positioning method described above belong to the same concept. For details not described in detail in the technical solution of the container positioning device, please refer to the description of the technical solution of the container positioning method described above.

[0207] Figure 6 This diagram illustrates a structural block diagram of a container access device according to an embodiment of the present invention. The components of the container access device 600 include, but are not limited to, a vision sensor 610, a memory 620, and a processor 630. The processor 630 is connected to the vision sensor 610 and the memory 620 via a bus 640, and a database 660 is used to store data.

[0208] Container access device 600 also includes access device 650, which enables container access device 600 to communicate via one or more networks 670. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 650 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0209] In one embodiment of the present invention, the above-mentioned components of the container access device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the container access device shown is for illustrative purposes only and is not intended to limit the scope of the invention. Those skilled in the art can add or replace other components as needed.

[0210] The vision sensor 610 is used to acquire images and transmit them to the processor 630; the processor 630 is used to execute the following computer-executable instructions, which are implemented when executed by the processor:

[0211] Acquire container images obtained by the vision sensor 610 from image acquisition of the target container;

[0212] Edge detection is performed on the container image to determine the intersection points of multiple edge lines of the target container, and contour detection is performed on the container image to determine multiple contour vertices of the target container.

[0213] Verify multiple edge line intersections and multiple contour vertices to determine the target vertices of the target container;

[0214] Based on the target vertex, determine the positioning result of the target container.

[0215] After obtaining the location results, the container access device can be controlled to retrieve the target container based on the location results.

[0216] The above is an illustrative scheme of a container access device according to this embodiment. It should be noted that the technical solution of this container access device and the technical solution of the container positioning method described above belong to the same concept. For details not described in detail in the technical solution of the container access device, please refer to the description of the technical solution of the container positioning method described above.

[0217] An embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the container positioning method described above.

[0218] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the container positioning method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the container positioning method described above.

[0219] The foregoing has described specific embodiments of the invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0220] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0221] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present invention.

[0222] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0223] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of the present invention. These embodiments are selected and specifically described to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A container positioning method, characterized in that, include: Acquire container images obtained by a vision sensor from image acquisition of the target container; Edge detection is performed on the container image to determine the intersection points of multiple edge lines of the target container, and contour detection is performed on the container image to determine multiple contour vertices of the target container; The intersection points of the multiple edge lines and the multiple contour vertices are verified to determine the target vertices of the target container, wherein the contour shape of the target container is a quadrilateral. Determining the positioning result of the target container based on the target vertices includes: obtaining vertex information of any three target vertices from the four target vertices of the target container; and calculating the pose information of the target container relative to the visual sensor using a preset perspective projection algorithm based on the vertex information of the any three target vertices. The positioning result refers to the actual positioning information of the target container relative to the visual sensor, and the positioning result includes pose information.

2. The method according to claim 1, characterized in that, The step of performing edge detection on the container image to determine the intersection points of multiple edge lines of the target container includes: Edge detection is performed on the container image to obtain a target edge detection image, wherein the target edge detection image includes multiple edge lines of the target container in the container image; The intersection points of the multiple edge lines in the target edge detection image are identified as the multiple edge line intersection points of the target container.

3. The method according to claim 2, characterized in that, The step of performing edge detection on the container image to obtain a target edge detection image includes: Edge detection is performed on the container image to obtain an initial edge detection image; Each edge line in the initial edge detection image is fitted to obtain the target edge detection image.

4. The method according to claim 3, characterized in that, The step of performing edge detection on the container image to obtain an initial edge detection image includes: Gradient calculation is performed on the container image to obtain an initial edge detection image.

5. The method according to claim 1, characterized in that, The step of performing contour detection on the container image to determine multiple contour vertices of the target container includes: Contour detection is performed on the container image to obtain a target contour detection image, wherein the target contour detection image includes the contour graphic of the target container in the container image; The vertices of the contour graphic in the target contour detection image are identified as multiple contour vertices of the target container.

6. The method according to claim 5, characterized in that, The step of performing contour detection on the container image to obtain a target contour detection image includes: Perform contour detection on the container image to obtain an initial contour detection image; The contour pattern in the initial contour detection image is fitted to obtain the target contour detection image.

7. The method according to claim 6, characterized in that, The step of performing contour detection on the container image to obtain an initial contour detection image includes: The container image is binarized to obtain an initial contour detection image.

8. The method according to any one of claims 1-7, characterized in that, The step of verifying the intersection points of the plurality of edge lines and the plurality of contour vertices to determine the target vertices of the target container includes: Determine the intersection of the first edge lines and the first contour vertex located at the same position; Obtain pixel information of each first pixel point within a preset range of the intersection of the first edge line and pixel information of each second pixel point within the preset range of the first contour vertex; The target vertex of the target container is determined based on the pixel information of each first pixel and the pixel information of each second pixel.

9. The method according to claim 8, characterized in that, Determining the intersection of the first edge lines and the first contour vertex located at the same position includes: Identify a preset identifier in the container image, wherein the preset identifier is set in advance at each vertex of the target container; Calculate the first distance between the intersection points of the plurality of edge lines and the preset identifier, and the second distance between the plurality of contour vertices and the preset identifier; Based on the first distance and the second distance, determine the intersection of the first edge lines and the first contour vertex located at the same position.

10. The method according to claim 8, characterized in that, Determining the target vertex of the target container based on the pixel information of each first pixel and the pixel information of each second pixel includes: Based on the pixel information of each first pixel, the grayscale distribution of the first pixel within the preset range of the intersection of the first edge lines is determined, and based on the pixel information of each second pixel, the grayscale distribution of the second pixel within the preset range of the first contour vertex is determined. From the first pixel grayscale distribution and the second pixel grayscale distribution, determine the target pixel grayscale distribution that conforms to the preset distribution rules; Based on the target pixel grayscale distribution, the target vertex of the target container is determined from the intersection of the first edge line and the first contour vertex.

11. The method according to claim 1, characterized in that, After calculating the pose information of the target container relative to the visual sensor using a preset perspective projection algorithm based on the vertex information of any three target vertices, the method further includes: Based on the pose information, the remaining target vertices, excluding any three of the four target vertices, are projected onto the container image to obtain the projected coordinates of the remaining target vertices; Obtain the vertex coordinates of the remaining target vertices in the container image; Calculate the error value based on the projected coordinates and the vertex coordinates; Based on the error value, update the pose information, and return to the step of projecting the remaining target vertices (excluding any three target vertices) from the four target vertices onto the container image based on the pose information to obtain the projected coordinates of the remaining target vertices; If the error value is less than or equal to a preset threshold, the updated pose information is determined to be the target pose information of the target container relative to the visual sensor.

12. The method according to any one of claims 1-7, characterized in that, Before performing edge detection on the container image to obtain multiple intersection points of the edge lines of the target container, and performing contour detection on the container image to obtain multiple contour vertices of the target container, the method further includes: The container image is filtered to obtain a noise-removed container image.

13. The method according to claim 1, characterized in that, The target container is a cargo box.

14. A container positioning device, characterized in that, include: The acquisition module is configured to acquire container images obtained by the vision sensor from image acquisition of the target container; The detection module is configured to perform edge detection on the container image to determine multiple intersection points of edge lines of the target container, and to perform contour detection on the container image to determine multiple contour vertices of the target container. The verification module is configured to verify the intersection points of the plurality of edge lines and the plurality of contour vertices to determine the target vertices of the target container, wherein the contour shape of the target container is a quadrilateral. The positioning result determination module is configured to determine the positioning result of the target container based on the target vertex; The positioning result determination module is further configured to obtain vertex information of any three target vertices from the four target vertices of the target container, and calculate the pose information of the target container relative to the visual sensor based on the vertex information of the any three target vertices and a preset perspective projection algorithm. The positioning result refers to the actual positioning information of the target container relative to the visual sensor, and the positioning result includes pose information.

15. A container storage and retrieval device, characterized in that, include: Visual sensors, memory, and processors; The vision sensor is used to acquire images of the target container and transmit the container images to the processor. The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the container positioning method according to any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the container positioning method according to any one of claims 1-13.

Images