Pattern code positioning and connected domain identification method and robot

Abstract

The embodiment of the application discloses a graphic code positioning method, a connected domain identification method and a robot. The graphic code positioning method comprises the following steps: in the process of collecting an image containing a graphic code by an image sensor, performing connected domain identification on the collected image to determine position information of the circumscribed rectangle of a plurality of connected domains in a field of view coordinate system of the image sensor; wherein the collected image comprises a body image of the graphic code and an image of at least one boundary element located at the periphery of the body image; determining at least one target connected domain conforming to the characteristics of the boundary element according to the size characteristics and / or position distribution characteristics of the circumscribed rectangle of the plurality of connected domains; and acquiring positioning information of the graphic code according to the position information of the circumscribed rectangle of the at least one target connected domain. Through the embodiment of the application, more efficient graphic code positioning can be realized, and the application range is wider.

Description

Technical Field

[0001] This application relates to the field of graphic code technology, and in particular to graphic code localization, connected component recognition methods, and robots. Background Technology

[0002] Image codes are graphic symbols that record data information, using specific geometric shapes arranged in a planar pattern with alternating black and white lines. For example, two-dimensional barcodes (QR codes) are a commonly used type, widely applied due to their advantages such as large information storage capacity, high reliability, flexibility, practicality, and low cost. For instance, information can be obtained by decoding image codes (further enabling website redirection, account login, mobile payments, etc.), and in the field of robotics, they can be used for robot navigation, and so on.

[0003] In utilizing graphic codes for various applications, the process typically begins by scanning an image containing the graphic code using an image sensor. The graphic code is then located, determining its center point position and / or tilt angle within the image. If decoding is required, the tilt angle of the graphic code image can be corrected before decoding. For robot navigation based on graphic codes, the position and tilt angle of the graphic code's center point can be used to determine if the robot's position deviates from the target location. If deviation occurs, timely attitude adjustments can be made to prevent error accumulation during subsequent movement, and so on.

[0004] Therefore, image code localization is crucial, especially in scenarios such as robot navigation. Since the efficiency of the image code localization algorithm directly impacts the robot's travel speed and operational efficiency, real-time performance requirements for image code localization are paramount. Developing more efficient image code localization algorithms remains a constant pursuit for those skilled in the art. Summary of the Invention

[0005] This application provides a method for graphic code localization and connected component recognition, as well as a robot, which can achieve more efficient graphic code localization and has a wider range of applications.

[0006] This application provides the following solution:

[0007] A method for locating images using a graphic code, comprising:

[0008] During the process of acquiring an image containing a graphic code using an image sensor, connected component identification is performed on the acquired image to determine the position information of the circumscribed rectangles of multiple connected components in the field of view coordinate system of the image sensor; wherein, the acquired image includes the body image of the graphic code, and the image of at least one boundary element located outside the body image, and the image of the boundary element is not connected to the body image;

[0009] Based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components, at least one target connected component that conforms to the characteristics of the boundary elements is determined;

[0010] The positioning information of the graphic code is obtained based on the position information of the outer rectangle of the at least one target connected region.

[0011] The process of performing connected component identification on the acquired images includes:

[0012] After receiving each row of image acquisition results, the pixels in the current row are binarized, and connected line segments are identified from them.

[0013] After comparing the connected segment identification results in the current row with those in the previous row, the connected component identifier and the position information of the bounding rectangle of the connected component are updated.

[0014] The process of performing connected component identification on the acquired images includes:

[0015] The image sensor's pixel stream is received line by line via a hardware programmable chip, and after each line of image acquisition results is received, the following processing is performed on a line-by-line basis:

[0016] Binarize each pixel in the current row;

[0017] After identifying connected segments in the current row after binarization, and comparing them with the connected segment identification results in the previous row stored in the register of the hardware programmable chip, the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register are updated.

[0018] Save the connected segment identification result of the current row to the register, and delete the connected segment identification result of the previous row from the register.

[0019] The hardware programmable chip identifies multiple connected components and determines the position information of the bounding rectangles of each connected component, then saves the information to the memory of the main control system. This information is then used by the software positioning module running in the main control system to perform the steps of determining the target connected components and locating the graphic code.

[0020] The boundary element includes a single rectangular border element;

[0021] The step of determining at least one target connected component that conforms to the characteristics of the boundary elements based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components includes:

[0022] The characteristics of the boundary elements are determined in advance based on the side length and / or aspect ratio of the rectangular border;

[0023] Based on the side length and / or aspect ratio of the circumscribed rectangle of each connected component, determine a single target connected component that conforms to the characteristics of the boundary element.

[0024] Wherein, obtaining the positioning information of the graphic code based on the position information of the circumscribed rectangle of the at least one target connected component includes:

[0025] The center position of the outer rectangle of the single target connected region is determined as the center position of the graphic code.

[0026] Wherein, obtaining the positioning information of the graphic code based on the position information of the circumscribed rectangle of the at least one target connected component includes:

[0027] The tilt angle of the graphic code relative to the image sensor's field-of-view coordinate system is determined based on the ratio between the side length of the outer rectangle of the single target connected region and the side length of the rectangle frame.

[0028] The process of connecting component identification further includes:

[0029] Determine and save the location information of multiple circumscribed points of each connected component and its respective circumscribed rectangle;

[0030] The step of obtaining the positioning information of the graphic code based on the position information of the circumscribed rectangle of the at least one target connected component includes:

[0031] Based on the positions of multiple circumscribed points of the single target connected region and its circumscribed rectangle, the magnitude and direction of the tilt angle of the graphic code relative to the field-of-view coordinate system of the image sensor are determined.

[0032] The boundary elements include: multiple boundary elements distributed at corresponding positions of vertices of the graphic code body image, and the multiple boundary elements are distributed in a matrix manner;

[0033] The step of obtaining the positioning information of the graphic code based on the position of the outer rectangle of the at least one target connected component includes:

[0034] Based on the positions of the multiple bounding rectangles corresponding to the multiple target connected components, the center position of the graphic code and its tilt angle relative to the field of view coordinate system of the image sensor are determined.

[0035] The step of determining at least one target connected component that conforms to the characteristics of the boundary elements based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components includes:

[0036] Based on the size and / or positional distribution characteristics of the circumscribed rectangles of the multiple connected components, and the size and / or positional distribution characteristics of the boundary elements, the confidence level of each connected component belonging to the boundary elements is determined, so that at least one connected component with the highest confidence level is determined as the target connected component, and the determined position of the center point of the graphic code and the confidence level information are used as the positioning result of the graphic code.

[0037] The step of obtaining the positioning information of the graphic code includes:

[0038] Based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangle of the target connected region and the confidence level, it is determined whether the graphic code image acquired by the image sensor is complete;

[0039] If incomplete, the outer rectangle is completed based on the size characteristics of the outer rectangle of at least one target connected component and the size characteristics of the boundary element. Then, the center point position of the graphic code is determined based on the center point position of the completed rectangle.

[0040] A method for connected component identification of graphic codes includes performing the following steps via a hardware programmable chip:

[0041] During the process of receiving the pixel stream acquired by the image sensor line by line, the following processing is performed on a line-by-line basis:

[0042] Binarize each pixel in the current row;

[0043] After identifying connected segments in the current row after binarization, and comparing them with the connected segment identification results in the previous row stored in the register of the hardware programmable chip, the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register are updated.

[0044] Save the connected segment identification result of the current row to the register, and delete the connected segment identification result of the previous row from the register;

[0045] After completing the connected segment identification of the last row of images acquired by the image sensor, and updating the position information of the connected component identifier and the circumscribed rectangle of the connected component, the connected component identification result is output from the register.

[0046] A type of robot,

[0047] The robot includes an image sensor, a hardware programmable chip, and a main control system, in which a software positioning module runs.

[0048] The image sensor is used to acquire images containing graphic codes line by line;

[0049] The hardware programmable chip is used to perform connected component identification on the image acquired by the image sensor by analyzing the image acquisition results line by line during the process of receiving the pixel stream acquired by the image sensor, and write the position information of the identified multiple connected components into the memory of the main control system.

[0050] The software positioning module in the main control system is used to determine at least one target connected component that meets the target conditions from the plurality of connected components, and to determine the positioning information of the graphic code based on the position information of the at least one target connected component, so as to adjust the navigation posture of the robot according to the positioning information of the graphic code.

[0051] Specifically, the hardware programmable chip is used to: after receiving each row of image acquisition results, perform binarization processing on each pixel in the current row, identify connected line segments, compare the identification results of connected line segments in the previous row stored in the register of the hardware programmable chip, update the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register, save the identification results of connected line segments in the current row to the register, and delete the identification results of connected line segments in the previous row from the register.

[0052] The image acquired by the image sensor includes the body image of the graphic code and the image of at least one boundary element located outside the body image, wherein the image of the boundary element is not connected to the body image;

[0053] The software positioning module in the main control system is specifically used to: determine at least one target connected region that conforms to the characteristics of the boundary element based on the size characteristics and / or position distribution characteristics of the circumscribed rectangles of the plurality of connected regions; and obtain the positioning information of the graphic code based on the position information of the circumscribed rectangles of the at least one target connected region.

[0054] A graphic code positioning device, comprising:

[0055] A connected component identification unit is used to identify connected components in an image during the acquisition of an image containing a graphic code by an image sensor, so as to determine the position information of the circumscribed rectangles of multiple connected components in the field of view coordinate system of the image sensor; wherein the acquired image includes the body image of the graphic code and the image of at least one boundary element located outside the body image, and the image of the boundary element is not connected to the body image.

[0056] The target connected component determination unit is used to determine at least one target connected component that conforms to the characteristics of the boundary element based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components.

[0057] A positioning unit is used to obtain the positioning information of the graphic code based on the position information of the outer rectangle of the at least one target connected region.

[0058] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0059] An electronic device, comprising:

[0060] One or more processors; and

[0061] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the preceding descriptions.

[0062] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0063] In this embodiment, for cases where the body image of a graphic code also has boundary element images for auxiliary positioning, during the acquisition of an image containing the graphic code using an image sensor, connected component identification can be performed on the acquired image to determine the position information of the circumscribed rectangles of multiple connected components in the field-of-view coordinate system of the image sensor. Then, based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the multiple connected components, at least one target connected component conforming to the characteristics of the boundary elements can be determined. Since the center point and tilt angle of such boundary elements can represent the center and tilt angle of the graphic code, the positioning information of the graphic code can be obtained based on the position information of the circumscribed rectangles of the at least one target connected component. Because this method does not directly identify the boundary elements themselves, but rather identifies the circumscribed rectangles of connected components, it is not limited by the shape of the specific boundary elements themselves. Whether it is a rectangular border, a circular border, an L-shaped border, or multiple solid dots or solid rectangles distributed in a matrix, the graphic code can be positioned using the solution provided in this embodiment. Therefore, the algorithm has a wide range of applications. Furthermore, since the implementation process does not require scanning the edge lines of boundary elements in both horizontal and vertical directions, it is more efficient.

[0064] Furthermore, in a preferred embodiment of this application, a method is provided to identify connected segments row by row and compare the results with those of the previous row to determine if there are overlapping regions, thereby identifying connected components and updating the specific connected component identifiers and the positions of the circumscribed rectangles. Therefore, the positioning process does not require separate scanning in the horizontal and vertical directions; it can be performed row by row, thus improving efficiency.

[0065] In the above-described method of line-by-line connected component identification, since each comparison only needs to be made with the connected segment identification result of the previous row, only the connected segment identification result of the most recent row needs to be saved. Furthermore, since the position of the connected component is identified by the position of the circumscribed rectangle, only the position information of the circumscribed rectangle (e.g., only the coordinates of the upper left and lower right corners of the circumscribed rectangle) needs to be saved and continuously updated. Therefore, the amount of information that needs to be saved during the entire connected component identification process is relatively small, and the requirements for storage resources are also low. Based on the above characteristics, the solution provided in this application embodiment is also more suitable for performing the connected component identification part of the calculation using a hardware programmable chip. In this way, the image acquired line by line by the image sensor can first be provided to the hardware programmable chip, which can then perform connected segment identification, comparison, connected component identification, and updating of the circumscribed rectangle position line by line. Thus, connected component identification can be completed almost simultaneously with the image sensor completing image acquisition. Moreover, data storage during the identification process can be completed in the hardware registers; therefore, no software intervention is required, achieving true real-time hardware connected component identification.

[0066] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of the graphic code positioning method provided in the embodiments of this application;

[0069] Figure 2 This is a schematic diagram of a QR code with boundary elements provided in an embodiment of this application;

[0070] Figure 3 This is a schematic diagram of the connected component identification process provided in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of the circumscribed rectangle of a connected region provided in an embodiment of this application;

[0072] Figure 5 This is a schematic diagram illustrating the method for determining the location information of the graphic code provided in the embodiments of this application;

[0073] Figure 6This is a schematic diagram illustrating the positioning process under incomplete image acquisition conditions provided in this application embodiment;

[0074] Figure 7 This is a flowchart of the connected component identification method provided in the embodiments of this application;

[0075] Figure 8 This is a schematic diagram of the robot provided in an embodiment of this application;

[0076] Figure 9 This is a schematic diagram of the graphic code positioning device provided in the embodiments of this application;

[0077] Figure 10 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0078] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0079] To facilitate understanding of the technical solutions provided in the embodiments of this application, the concept of graphic code positioning and its application scenarios in the embodiments of this application are briefly introduced below. In practical applications, an image containing a graphic code can first be acquired by an image sensor in a barcode scanning device, and then the graphic code can be decoded, or the barcode scanning device (e.g., a robot) can be navigated. However, before decoding the graphic code or navigating the robot, it is usually necessary to first locate the graphic code. This is because when the image sensor acquires the graphic code image, there may be tilting or other issues, causing the graphic code to be tilted in the image sensor's field of view coordinate system. Before decoding the graphic code, it is usually necessary to determine the specific tilt angle, and then use software to straighten the graphic code image in the field of view before decoding. When navigating a robot, after determining the position of the center point of the graphic code in the image and the tilt angle, this information can be used to determine whether the robot's posture deviates from the route. If so, posture adjustments can be made in time to avoid excessive error accumulation.

[0080] In robot navigation applications, robots typically operate within a spatial environment. For example, in a warehouse using a "shelf-to-person" picking solution, a transport robot (usually a disc-shaped structure) can move inventory shelves to the picking station, allowing pickers to complete the picking process directly at their workstations, completely revolutionizing the traditional "person-to-goods" model. During the "shelf-to-person" transport process, after identifying the shelf to be moved, a route is planned based on the shelf's location and the robot's starting position. The robot then automatically travels to the shelf's location, uses its lifting device to lift the shelf, and then transports it to the picker's workstation according to the planned route.

[0081] During the robot's journey along a planned route, multiple graphic codes are typically pre-set on the floor of spaces such as warehouses. For example, a grid is drawn on the ground, and a graphic code is placed within each grid. These graphic codes carry the specific location information of their position within the space. Furthermore, the distances between each graphic code can be equal (or unequal, but the distances between different graphic codes can be determined through a map system). Thus, when navigating the robot, it only needs to know how many graphic codes it has traversed in a certain direction and how to turn, for example, after passing 10 graphic codes forward, turn 90 degrees to the left, and then pass 5 more graphic codes to reach the shelf location, and so on. During the journey, the robot doesn't even need to decode each graphic code; it only needs to record how many graphic codes it has traversed, thus improving efficiency. However, in practical applications, the robot may deviate from its course during its journey. For example, a robot's image sensor is typically positioned at the center of its chassis. Normally, when the robot travels over a graphic marker, the center of the graphic marker coincides with the center of the image sensor's field of view (usually rectangular), and the graphic marker has no tilt angle in the image sensor's field of view coordinate system. However, due to uneven ground, the robot may deviate from the center of the graphic marker when it travels over it. This will cause the center point of the graphic marker in the image captured by the image sensor to not coincide with the center point of the image sensor's field of view, and the graphic marker may also have a tilt angle in the field of view coordinate system. Conversely, if the center point of the graphic marker does not coincide with the center point of the image sensor's field of view in the image captured by the image sensor when the robot passes over it, or if the graphic marker has some tilt angle, it indicates that the robot has deviated from the center of the graphic marker, and the robot's posture needs to be adjusted.

[0082] In addition, in the aforementioned "shelf-to-person" transport scenario using the aforementioned handling robot, an upward-facing image sensor is typically installed at the center of the robot pallet to capture images of the graphic code at the center of the shelf's bottom. By locating this graphic code, the robot determines whether the shelf's center is aligned with its center. If there is any deviation or tilt, adjustments are made before lifting the shelf. Furthermore, the shelf's graphic code can be further decoded to determine if the specific shelf is correct, and so on.

[0083] Therefore, graphic code positioning involves determining the position of the graphic code's center point within the image sensor's field-of-view coordinate system, as well as the graphic code's tilt angle within that system. This positioning information can then be used to decode the graphic code, or to adjust the posture of scanning devices such as robots during navigation, and so on.

[0084] One approach to locating graphic codes is to directly use the image features of the graphic code itself. For example, the graphic code's image typically includes square positioning markers at the top left, top right, and bottom left corners, which can be used for positioning. However, in scenarios like robot navigation, where real-time positioning is crucial (the positioning time needs to match the robot's speed), boundary elements are often added around the graphic code's image to facilitate faster positioning. These boundary elements are not connected to the graphic code's image to distinguish them. For example, a rectangular or circular border can be added around the graphic code. Alternatively, four solid dots or small solid rectangles can be added at the corresponding positions of the four vertices (e.g., the envelope of the graphic code's image is usually a rectangular area, so four solid dots can be placed on the extension of the diagonal of this rectangular area). Other irregular shapes can also be used. Since these boundary elements are usually connected themselves, but not connected to the graphic code body image, graphic code localization can be performed by identifying these boundary elements, without needing to identify more complex patterns in the graphic code body image. In other words, boundary elements are not part of the graphic code body image; they mainly serve to assist in the rapid localization of the graphic code.

[0085] In existing technologies, when using the aforementioned boundary elements to locate graphic codes, the location of these boundary elements is mainly identified from the acquired image. For example, when using a rectangular border as the boundary element, after the image sensor acquires a complete frame of image, the original image is binarized, and then the four edge lines of the rectangle are scanned in both horizontal and vertical directions. The position of the rectangle is then found using these edge lines, allowing the calculation of the rectangle's position and its tilt angle in the image sensor's field-of-view coordinate system. However, this method can only identify rectangular boundary elements; it cannot identify boundary elements such as circles or solid dots, thus failing to facilitate rapid graphic code location.

[0086] Based on the above, this application first provides a graphic code localization method, which is mainly used for rapid localization of graphic codes with boundary elements. Specifically, since the type of boundary elements of a graphic code is known in scenarios such as robot navigation, prior knowledge about the characteristics of the boundary elements can be obtained in advance. In this application embodiment, the characteristics of the boundary elements may include: not being connected to the graphic code body image; and the circumscribed rectangle of such a boundary element having certain characteristics in terms of size and / or positional distribution. That is, in the image acquired by the image sensor, the boundary elements of the graphic code will form one or more separate connected regions, and the circumscribed rectangle of such connected regions conforms to the known characteristics of the boundary elements in terms of size and / or positional distribution.

[0087] Therefore, in this embodiment of the application, specifically when locating the graphic code, during the process of acquiring an image containing the graphic code using an image sensor, connected component identification can be performed on the acquired image to determine the position information of the circumscribed rectangles of multiple connected components in the field-of-view coordinate system of the image sensor. Then, based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the multiple connected components, at least one target connected component that conforms to the characteristics of the boundary element can be determined, and the positioning information of the graphic code can be obtained based on the position information of the circumscribed rectangles of the at least one target connected component.

[0088] Specifically, when identifying connected components, connected segments can be identified row by row and compared with those identified in the previous row. If overlapping areas exist, they can be merged into a single connected component, and the circumscribed rectangles of each connected component can be marked. This way, connected component identification can be completed simply by identifying them in the horizontal direction, eliminating the need for vertical identification. Furthermore, since boundary elements of any shape have circumscribed rectangles, the implementation is no longer limited by the shape of the boundary elements themselves; a uniform approach can be used for connected component identification, and the circumscribed rectangles of the connected components can be used for marking.

[0089] In the process of connected component identification, a method can be adopted to identify connected segments row by row and compare them with the connected segments in the previous row, forming connected components from the overlapping connected segments. In a preferred embodiment, to further improve positioning efficiency, a hardware programmable chip can also be used to process the connected component identification. For example, a hardware programmable chip can be added to the robot, and the image data collected by the image sensor can be provided to the hardware programmable chip to complete the connected component identification in hardware. Since the hardware programmable chip can process data on a pixel-by-pixel basis, and the image sensor can also collect images on a row-by-row basis, the connected component identification can be completed almost simultaneously with the image acquisition. In addition, since in this embodiment, only the identification results of the most recent row of connected segments, the identifiers of each connected component (e.g., numbers), and the coordinates of the upper left and lower right corners of the bounding rectangle of the connected component need to be stored during the connected component identification process, the amount of information that needs to be stored during the processing is also very small. This better adapts to the limited storage space of the registers in the hardware chip, thereby realizing the identification of connected components in a fully hardware-based manner.

[0090] After completing the connected component identification, the target connected component that meets the conditions can be determined from the identified connected components using software, and the graphic code can be located.

[0091] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0092] Example 1

[0093] First, this embodiment provides a graphic code localization method, which can be applied to robot systems, or other systems or devices with graphic code localization requirements. See also Figure 1 The method may include:

[0094] S101: During the process of acquiring an image containing a graphic code through an image sensor, connected component identification is performed on the acquired image to determine the position information of the circumscribed rectangles of multiple connected components in the field of view coordinate system of the image sensor; wherein, the acquired image includes the body image of the graphic code, and the image of at least one boundary element located outside the body image, and the image of the boundary element is not connected to the body image.

[0095] The specific image sensor can be an image acquisition sensor equipped in a barcode scanning device. For example, in a preferred embodiment, it could be a CMOS (Complementary Metal-Oxide-Semiconductor) camera, etc. Such a CMOS camera can acquire images line by line. In specific applications, this could involve a robot navigating by using an image sensor deployed at the center of its chassis to acquire images of barcodes placed on the ground in spaces such as warehouses. Alternatively, it could be used to acquire images of barcodes when decoding them, and so on.

[0096] Image sensors typically have a rectangular field of view. When analyzing the acquired images, a field-of-view coordinate system can be established with the top-left corner of this field of view as the origin. The positions of each pixel can then be determined based on this coordinate system. In the aforementioned robot navigation scenario, the image sensor's shooting parameters (including viewing angle and pixel count) are known. Furthermore, since the image sensor is deployed on the robot chassis, pointing downwards at the ground, and the graphic code is placed on the ground, the image sensor's shooting distance (i.e., the height of the robot chassis from the ground) is also known. With both the shooting parameters and the shooting distance known, the specific size of the field of view is also known.

[0097] In this application embodiment, a fast positioning implementation scheme is provided mainly for cases where the graphic code body image has boundary elements around it. For example, such as Figure 2 As shown, the graphic code body image is the part specifically used to carry information, while the boundary elements do not belong to the graphic code body image. Instead, they are elements added around the body image to assist in the positioning of the graphic code. Specifically, they can be as follows: Figure 2 The rectangular black frame shown, etc. That is to say, in this embodiment of the application, prior knowledge about the characteristics of its boundary elements can be obtained beforehand, including dimensional features such as side length and aspect ratio, as well as positional distribution features, etc. Furthermore, since these boundary elements are themselves connected and have the characteristic of being disconnected from the main area of ​​the graphic code, connected component identification can be used. Based on the known characteristics of the boundary elements, the confidence level of each connected component belonging to the boundary element can be determined, thereby identifying the connected components belonging to the boundary element.

[0098] In this context, a connected component refers to a connected region, which is a set of points in a plane (i.e., a connected component includes multiple points located in the same plane), and any two points in the set can be connected by a broken line that completely belongs to the set. In practice, there are various methods for identifying connected components, and this application proposes a preferred implementation. Specifically, each row of data collected by the image sensor can be binarized (i.e., 0 and 1 can be used to represent black and white pixels in the image, respectively). Then, connected segment identification can be performed on each row of pixels. A connected segment is part of a connected component. It is called a connected segment because a row of pixels has only one pixel in the vertical direction; if N pixels in a row are connected together, these N pixels form a line segment. After identifying a connected segment from the current row, its start and end points can be compared with those of the connected segments identified in the previous row to determine if the connected segments in the current row overlap with those in the previous row. If so, they can be merged into the same connected component. Furthermore, in this embodiment of the application, the position of each connected component can be recorded by the position of its circumscribed rectangle.

[0099] For example, such as Figure 3 As shown in Figure (a), after binarizing the data stream received from the first row, no connected components were obtained. After binarizing the data stream received from the second row, two connected segments were obtained, for example, labeled as connected segment 1 and connected segment 2. In this embodiment, for the connected components obtained in real-time analysis, the bounding rectangle information of each connected component can be recorded simultaneously, for example, the coordinate information of the two vertices of the upper left and lower right corners of the bounding rectangle of each connected component can be recorded.

[0100] During the row-by-row analysis, if a connected segment in a row overlaps with a connected segment in the previous row, they can be merged into the same connected component and marked as such. For example, after analyzing multiple subsequent pixel rows based on Figure (a), the algorithm described above yields the three connected components shown in Figure (b): Connected Component 1, Connected Component 2, and Connected Component 3. After identifying and merging the connected segments in each row, the identifier and the position of the bounding rectangle of each connected component can be updated. Further analysis of the next row of pixels based on Figure (b), as shown in Figure (c), reveals that the connected segments in this row are connected to all three connected components in Figure (b), meaning they overlap. Therefore, these three connected components in Figure (b) can be merged and marked as the same connected component, for example, all marked as Connected Component 1. Simultaneously, the position of the bounding rectangle of Connected Component 1 can be updated, and information about Connected Components 2 and 3 in Figure (b) is no longer stored.

[0101] After all rows have been analyzed, the final connected components can be obtained based on the merged connected component results. Since the bounding rectangle information of each connected component is marked during the connected component analysis process, this bounding rectangle information is updated accordingly as the connected components are merged, ultimately yielding the bounding rectangles of each connected component.

[0102] In specific implementation, the above-mentioned scheme of identifying connected segments row by row and comparing them with the identification results of the previous row to identify connected components can be implemented in software; or, in order to obtain higher efficiency, in the preferred embodiment of this application, the image acquisition results provided by the image sensor can be received row by row by a hardware programmable chip (e.g., FPGA (Field Programmable Gate Array) chip), and the image acquisition results can be analyzed row by row by the hardware programmable chip to identify connected components in the acquired image.

[0103] For example, a programmable hardware chip can be added to the robot. Image data collected by the image sensor can be provided to this chip. After connected component identification is performed in the hardware, the position information of the bounding rectangles of each connected component is provided to the memory of the robot's main control system. The positioning module implemented in software in the main control system only needs to perform the subsequent determination of the target connected components and the specific positioning information of the graphic code. In other words, the algorithm can be divided into two parts: the connected component identification part can be completed by the programmable hardware chip, and the subsequent parts can be completed in software.

[0104] The above solution improves efficiency because if the image sensor directly writes the acquired image data into the main control memory, and the software module in the main control system performs connected component identification and subsequent positioning information acquisition, the software module can only process on a frame-by-frame basis. That is, it needs to wait until a complete frame of image is received before starting connected component identification and other processing. However, the image sensor can acquire and transmit images on a line-by-line basis, and the hardware programmable chip can also process images directly on a line-by-line basis. Furthermore, in this embodiment, connected component identification can be performed by identifying connected segments line by line. Therefore, each line of image data acquired by the image sensor can be transmitted to the hardware programmable chip. After receiving the data, the hardware programmable chip can process the current line of data. Thus, as the image sensor completes the last line of image data, the hardware programmable chip only needs to complete the connected segment identification and comparison for the last line of image data to complete the entire connected component identification process. In other words, the connected component identification process can be completed almost simultaneously with the image acquisition process. Therefore, the subsequent software only needs to perform further processing based on the connected component identification results, thus significantly improving processing efficiency. If the software part still needs to perform decoding and other processing on the image code, since the target connected components have already been identified by the hardware part, the software part only needs to analyze and calculate the image within the bounding rectangle of the target connected component after acquiring an image frame, instead of analyzing all the images in a frame. This also helps to improve the efficiency of subsequent decoding processes. In addition, since the connected component identification process only needs to save the identification results of the most recent row of connected segments, as well as the identifiers such as the numbers of each identified connected component and the coordinates of the upper left and lower right corners of the bounding rectangle, the amount of information that needs to be saved is very small. It can be implemented directly in the registers of the hardware programmable chip, thus realizing a fully hardware-based connected component identification scheme.

[0105] Specifically, in this embodiment, the hardware programmable chip can binarize each pixel in the current row after receiving each row of image acquisition results, identify connected segments, and compare the results with the connected segment identification results of the previous row stored in the register of the hardware programmable chip. Then, it updates the connected component identifiers and the position information (mainly the coordinates of the top-left and bottom-right corners) of the bounding rectangles of the connected components stored in the register. After completing the above update, the connected segment identification results of the current row can be saved to the register, and the connected segment identification results of the previous row can be deleted from the register. Thus, after receiving the next row of image acquisition results, the same process of binarization, connected segment identification, and comparison with the connected segment identification results of the most recent row stored in the register can be performed. This process continues until the connected segment identification and comparison of the last row of image acquisition results are completed, yielding the final connected component identification results and the position information of their respective bounding rectangles.

[0106] S102: Based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected domains, determine at least one target connected domain that conforms to the characteristics of the boundary elements.

[0107] After obtaining the bounding rectangles of multiple connected components, at least one target connected component that conforms to the characteristics of the boundary elements can be determined based on the size and / or positional distribution characteristics of the bounding rectangles. In other words, since the characteristics of the boundary elements of a specific graphic code are known—they can be rectangular borders, circular borders, L-shaped borders, rectangular connected component matrices, circular connected component matrices, etc.—even if there is a tilt during image acquisition, resulting in a tilt angle for the boundary elements in the image sensor's field of view, the bounding rectangles of these boundary elements still possess certain characteristics. These specific characteristics can include size and / or positional distribution characteristics.

[0108] The size features can include the side length of the circumscribed rectangle or its aspect ratio. The positional distribution features mainly refer to the case where a solid circle or similar boundary element is placed at each of the four vertices of the graphic code. In this case, there are multiple boundary elements, all of equal size, distributed in a matrix. Furthermore, if the graphic code has a tilt angle, its boundary elements will also be tilted. Correspondingly, the side length of the circumscribed rectangle will change at different tilt angles, but this variation is within a certain range. For example, in most cases, the boundary elements of a graphic code are typically square, circular, or L-shaped borders. The aspect ratio of the circumscribed rectangles of these borders is close to 1. Therefore, the closer the aspect ratio of the circumscribed rectangle of a connected component is to 1, the higher the confidence that it belongs to the target connected component. In other words, the degree to which the aspect ratio of the circumscribed rectangle of a connected component is close to 1 can be used as one factor in determining the confidence that it belongs to a boundary element.

[0109] Of course, the body image of the graphic code may also contain connected components with square bounding rectangles. Therefore, the aspect ratio of the bounding rectangle alone may not be sufficient to definitively determine whether it belongs to a boundary element. A comprehensive judgment must be made by considering the specific side lengths. Specifically, since boundary elements are located outside the graphic code body image, the side length of the bounding rectangle of the connected component they form is usually greater than the side length of the bounding rectangle of the connected component in the graphic code body image. Furthermore, even with a tilt angle, the side length of the bounding rectangle of a boundary element, when its side length is known, will vary within a certain range. For example, as... Figure 4 (a) When using a rectangular border as the boundary element, the smallest side length of the circumscribed rectangle occurs when the graphic code is aligned. In this case, the side length of the circumscribed rectangle can be calculated based on the side length of the rectangular border, as well as the image sensor's shooting parameters and shooting distance (these parameters remain constant). For example, let's assume it's L. The largest side length of the circumscribed rectangle occurs when the graphic code is rotated 45 degrees. In this case, the side length of the circumscribed rectangle is L. Therefore, we can determine whether the side length of the bounding rectangle of the connected component is multiples of 1. Within the range, this is one of the factors used to determine the confidence level of whether the connected component belongs to the rectangular bounding box.

[0110] In addition, such as Figure 5As shown in (b), for the case where a circular border and four solid dots arranged in a matrix are used as boundary elements, the circular border and these solid dots can be identified as multiple different connected components and marked by the circumscribed rectangles of each connected component. Furthermore, since it is a circle, the side length of its circumscribed rectangle is equal to the diameter of the circle; even if the graphic code is tilted, the side length of the circumscribed rectangle of this circular border or dots will not change. Regarding the four dots, they also exhibit a matrix distribution characteristic, except that the rows / columns of the matrix may form a certain angle with the coordinate axes of the image sensor's field of view coordinate system, and so on. Therefore, these four dots can be identified based on the side length information of the circumscribed rectangles, and subsequently, the tilt angle of the graphic code can be determined based on the angle between the rows / columns of the matrix formed by these four dots and the coordinate axes of the image sensor's field of view coordinate system.

[0111] In summary, by using both the aspect ratio and side length information of the circumscribed rectangle, the confidence score for a connected component belonging to a boundary element can be determined. Alternatively, using only one factor can also determine the confidence score. Furthermore, the above factors can be combined with other factors to determine the confidence score. These other factors could include the parallelism of opposite sides of the shape formed by the connected component, the perpendicularity of adjacent sides, and so on. For example, ideally, opposite sides of a rectangular border should be parallel, and adjacent sides should be perpendicular. Further enumeration of other factors is not provided here.

[0112] Additionally, when determining the target connected components, the aspect ratio and / or side length information of the circumscribed rectangles of each connected component can be used to filter them. For example, connected components with an aspect ratio greater than a certain percentage (such as 2) or less than a certain percentage (such as 1 / 2) are usually not boundary elements of the graphic code and are directly filtered out. As another example, when the boundary elements are known to be rectangular or circular borders, connected components with circumscribed rectangles whose side lengths are less than a certain length are usually not boundary elements of the graphic code and are therefore directly filtered out.

[0113] S103: Obtain the positioning information of the graphic code based on the position information of the outer rectangle of the at least one target connected region.

[0114] After identifying the target connected region that meets the boundary element characteristics, the positioning information of the graphic code can be determined based on the position information of the bounding rectangle of this target connected region in the field of view coordinate system. That is, the center point position of the graphic code and its tilt angle relative to the image sensor's field of view coordinate system can be determined.

[0115] Specifically, as mentioned above Figure 4As shown in (a), for cases where a single rectangular frame is used as the boundary element, the center position of the outer rectangle of the single target connected component can be directly determined as the center position of the graphic code. Furthermore, since the side length of the rectangular frame is known, and the side length of the outer rectangle is related to the tilt angle of the rectangular frame, the tilt angle of the graphic code relative to the image sensor's field-of-view coordinate system can be calculated by using the ratio between the side length of the outer rectangle of the single target connected component and the side length of the rectangular frame. For example, assuming the side length of a rectangular frame is L, if the side length of its outer rectangle is also L, it proves that the rectangular frame is perfectly aligned and has no tilt angle. If the side length of the outer rectangle is... This proves that the rectangular border is tilted at 45 degrees, and so on. Since the rectangular border is tilted along with the graphic code image, the tilt angle of the rectangular border also represents the tilt angle of the graphic code.

[0116] Of course, judging solely from the side length of the circumscribed rectangle, one can usually only determine the magnitude of the tilt angle of the graphic code, but not the direction of the tilt. Therefore, in an optional implementation, if it is necessary to know the direction of the tilt angle, the position information of multiple circumscribed points of each connected component and its respective circumscribed rectangle can be recorded during connected component identification. In this way, the direction of the tilt angle can be determined based on the positions of multiple circumscribed points of the single target connected component and its circumscribed rectangle. For example, as... Figure 5 As shown in (a), assuming the four intersection points of the connected component and the circumscribed rectangle are determined to be A, B, C, and D, then in addition to determining the magnitude of the tilt angle of the graphic code in the field of view coordinate system based on the side length of the circumscribed rectangle (for example, the angle between the graphic code and the horizontal direction is 15° shown in this figure), the direction of the tilt angle can also be determined, for example, in the... Figure 5 In the example shown in (a), based on the relative position of the line connecting points A and B with the horizontal side of the circumscribed rectangle, it can be determined that the graphic code is tilted to the right, and so on.

[0117] Furthermore, if the boundary elements of a specific graphic code are multiple boundary elements distributed at corresponding positions of the vertices of the graphic code body image, and arranged in a matrix, then the center position of the graphic code and its tilt angle relative to the image sensor's field-of-view coordinate system can be directly determined based on the positions of the multiple bounding rectangles corresponding to the determined multiple target connected components. For example, Figure 5 In the example shown in (b), after identifying the positions of the bounding rectangles of the connected domains corresponding to the four surrounding realization points, the positions of the center points of these four connected domains can be found. Then, these four center point positions can be connected into a rectangle. The center point of this rectangle can be used as the center position of the graphic code. The angles between the four sides of this rectangle and the coordinate axes of the field of view coordinate system can be used as the tilt angle information of the graphic code.

[0118] It should be noted that in practical applications, there may be situations where the image sensor does not capture the graphic code image completely, such as... Figure 6 As shown, in this case, the connected components A, B, C, and D in the diagram will still be identified as the target connected components (because calculations show that their confidence in belonging to the edit element is still the highest). However, at this point, the center of this target connected component actually deviates from the center of the graphic code, and this deviation can be reflected by the confidence of the target connected component. In other words, the algorithm's positioning result for the center point of the graphic code can include both the specific center point coordinates and the confidence level. Subsequent applications such as robot navigation and pose adjustment can then be performed based on these coordinates and confidence information.

[0119] Alternatively, in another optional implementation, the completeness of the graphic code image acquired by the image sensor can be determined based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangle of the target connected component and the confidence level. For example, when the image is complete, the confidence level of the identified target connected component belonging to the boundary element is often relatively high, such as above 95%. However, if after identifying multiple connected components, it is found that the highest confidence level among each connected component is only 80%, then it can be proven that the image acquisition may be incomplete. In this case, the circumscribed rectangle can be completed based on the size characteristics of the circumscribed rectangle of at least one identified target connected component and the size characteristics of the boundary element, and then the center point position of the graphic code can be determined based on the center point position of the completed rectangle. For example, in Figure 6 In the example shown, assuming the boundary element of the graphic code is a square black frame, the sides of its circumscribed rectangle should also be of equal length. However, if the captured graphic code image is incomplete, the circumscribed rectangle of the target connected component will be a rectangle. In this case, the shorter side can be padded to a square based on the length of the longer side of the circumscribed rectangle of the target connected component. Then, the center point of this square can be determined as the center point of the graphic code.

[0120] Of course, even with incomplete image acquisition, as long as the missing portion is not significant, the tilt angle information of the graphic code can still be determined using simple geometric algorithms based on the positional information of the intersection points A, B, C, and D of the target connected component and the circumscribed rectangle, or directly based on information such as the side length of the circumscribed rectangle, and so on. In other words, incomplete image acquisition has a relatively small impact on recognizing the tilt angle information of the graphic code.

[0121] In summary, through the embodiments of this application, when the body image of a graphic code also has boundary element images for auxiliary positioning, the image containing the graphic code can be acquired by a sensor during the acquisition of the image. Connected component identification can be performed on the acquired image to determine the position information of the circumscribed rectangles of multiple connected components in the field-of-view coordinate system of the image sensor. Then, based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the multiple connected components, at least one target connected component conforming to the characteristics of the boundary element can be determined. Since the center point and tilt angle of this boundary element can represent the center and tilt angle of the graphic code, the positioning information of the graphic code can be obtained based on the position information of the circumscribed rectangles of the at least one target connected component. In this way, since the boundary element itself is not directly identified, but rather the circumscribed rectangles of the connected components are identified, this scheme is not limited by the shape of the specific boundary element itself. Whether it is a rectangular border, a circular border, an L-shaped border, or multiple solid dots or solid rectangles distributed in a matrix, the graphic code can be positioned using the scheme provided in the embodiments of this application.

[0122] Furthermore, in a preferred embodiment of this application, a method is provided to identify connected segments row by row and compare the results with those of the previous row to determine if there are overlapping regions, thereby identifying connected components and updating the specific connected component identifiers and the positions of the circumscribed rectangles. Therefore, the positioning process does not require separate scanning in the horizontal and vertical directions; it can be performed row by row directly.

[0123] In the above-described method of line-by-line connected component identification, since each comparison only needs to be made with the connected segment identification result of the previous row, only the connected segment identification result of the most recent row needs to be saved. Furthermore, since the position of the connected component is identified by the position of the circumscribed rectangle, only the position information of the circumscribed rectangle (e.g., only the coordinates of the upper left and lower right corners of the circumscribed rectangle) needs to be saved and continuously updated. Therefore, the amount of information that needs to be saved during the entire connected component identification process is relatively small, and the requirements for storage resources are also low. Based on the above characteristics, the solution provided in this application embodiment is also more suitable for using a hardware programmable chip to perform the connected component identification. In this way, the image acquired line by line by the image sensor can first be provided to the hardware programmable chip, which can then perform line-by-line connected segment identification, comparison, connected component identification, and updating of the circumscribed rectangle position. Thus, connected component identification can be completed almost simultaneously with the image sensor completing image acquisition. Moreover, data storage during the identification process can be completed in the hardware registers, therefore, no software intervention is required, achieving true real-time hardware connected component identification.

[0124] Example 2

[0125] Regarding the connected component identification scheme using a hardware programmable chip mentioned in Embodiment 1, this scheme can also be applied to connected component identification processes in other scenarios in practical applications. That is, the acquired image may not necessarily contain a graphic code, and the purpose of connected component identification may not be for graphic code localization. As long as connected component identification of the acquired image is required, the method provided in this application embodiment can be used. Therefore, Embodiment 2 of this application provides a method for connected component identification of graphic codes. For details, see [link to documentation]. Figure 7 The method may include the following steps performed via a hardware-programmable chip:

[0126] S701: During the process of receiving the pixel stream acquired by the image sensor line by line, the following processing is performed on a line-by-line basis:

[0127] S7011: Binarize each pixel in the current row;

[0128] S7012: After identifying connected segments from the current row after binarization, and comparing them with the connected segment identification results from the previous row stored in the register of the hardware programmable chip, update the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register.

[0129] S7013: Save the connected segment identification result of the current row to the register, and delete the connected segment identification result of the previous row from the register;

[0130] S702: After completing the connected segment identification of the last row of images acquired by the image sensor, and updating the position information of the connected component identifier and the circumscribed rectangle of the connected component, the connected component identification result is output from the register.

[0131] Example 3

[0132] This third embodiment also provides a robot, see [link / reference] Figure 8 The robot includes an image sensor 801, a hardware programmable chip 802, and a main control system 803, wherein a software positioning module 8031 ​​runs in the main control system.

[0133] The image sensor 801 is used to acquire images containing graphic codes line by line;

[0134] The hardware programmable chip 802 is used to perform connected component identification on the image acquired by the image sensor by analyzing the image acquisition results line by line during the process of receiving the pixel stream acquired by the image sensor, and write the position information of the identified multiple connected components into the memory of the main control system.

[0135] The software positioning module 8031 ​​in the main control system 803 is used to determine at least one target connected component that meets the target conditions from the plurality of connected components, and to determine the positioning information of the graphic code based on the position information of the at least one target connected component, so as to adjust the navigation posture of the robot according to the positioning information of the graphic code.

[0136] In a specific implementation, the hardware programmable chip can be used to: after receiving each row of image acquisition results, perform binarization processing on each pixel in the current row, identify connected line segments, compare the identified connected line segments in the previous row stored in the register of the hardware programmable chip, update the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register, save the identified connected line segments of the current row to the register, and delete the identified connected line segments of the previous row from the register.

[0137] The image acquired by the image sensor includes the body image of the graphic code and the image of at least one boundary element located outside the body image, wherein the image of the boundary element is not connected to the body image;

[0138] The software positioning module in the main control system is specifically used to: determine at least one target connected region that conforms to the characteristics of the boundary element based on the size characteristics and / or position distribution characteristics of the circumscribed rectangles of the plurality of connected regions; and obtain the positioning information of the graphic code based on the position information of the circumscribed rectangles of the at least one target connected region.

[0139] For the parts not described in detail in Embodiments 2 and 3 above, please refer to the description in Embodiment 1, which will not be repeated here.

[0140] Corresponding to the aforementioned Embodiment 1, this application also provides a graphic code positioning device, see [link to embodiment 1]. Figure 9 The device may include:

[0141] The connected component identification unit 901 is used to identify connected components in the acquired image during the process of acquiring an image containing a graphic code through an image sensor, so as to determine the position information of the circumscribed rectangles of multiple connected components in the field of view coordinate system of the image sensor; wherein the acquired image includes the body image of the graphic code, and the image of at least one boundary element located outside the body image, and the image of the boundary element is not connected to the body image.

[0142] The target connected component determination unit 902 is used to determine at least one target connected component that conforms to the characteristics of the boundary element based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components.

[0143] The positioning unit 903 is used to obtain the positioning information of the graphic code based on the position information of the outer rectangle of the at least one target connected region.

[0144] Specifically, the connected component identification unit can be used for:

[0145] After receiving each row of image acquisition results, the pixels in the current row are binarized, and connected line segments are identified from them.

[0146] After comparing the connected segment identification results in the current row with those in the previous row, the connected component identifier and the position information of the bounding rectangle of the connected component are updated.

[0147] More specifically, the connected component identification unit can be used for:

[0148] The image sensor's pixel stream is received line by line via a hardware programmable chip, and after each line of image acquisition results is received, the following processing is performed on a line-by-line basis:

[0149] Binarize each pixel in the current row;

[0150] After identifying connected segments in the current row after binarization, and comparing them with the connected segment identification results in the previous row stored in the register of the hardware programmable chip, the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register are updated.

[0151] Save the connected segment identification result of the current row to the register, and delete the connected segment identification result of the previous row from the register.

[0152] The hardware programmable chip, after identifying multiple connected components and determining the position information of the circumscribed rectangles of each connected component, can save the information to the memory of the main control system. This information can then be used by the software positioning module running in the main control system to perform the steps of determining the target connected component and locating the graphic code.

[0153] The boundary element includes a single rectangular border element;

[0154] At this point, the target connected component determination unit can specifically be used for:

[0155] The characteristics of the boundary elements are determined in advance based on the side length and / or aspect ratio of the rectangular border;

[0156] Based on the side length and / or aspect ratio of the circumscribed rectangle of each connected component, determine a single target connected component that conforms to the characteristics of the boundary element.

[0157] Specifically, the positioning unit can be used for:

[0158] The center position of the outer rectangle of the single target connected region is determined as the center position of the graphic code.

[0159] In addition, the positioning unit can also be used for:

[0160] The tilt angle of the graphic code relative to the image sensor's field-of-view coordinate system is determined based on the ratio between the side length of the outer rectangle of the single target connected region and the side length of the rectangle frame.

[0161] Alternatively, the connected component identification unit can also be used to determine and save the position information of multiple circumscribed points of each connected component relative to its respective circumscribed rectangle.

[0162] At this time, the positioning unit can be specifically used for:

[0163] Based on the positions of multiple circumscribed points of the single target connected region and its circumscribed rectangle, the magnitude and direction of the tilt angle of the graphic code relative to the field-of-view coordinate system of the image sensor are determined.

[0164] In addition, the boundary elements include: multiple boundary elements distributed at corresponding positions of the vertices of the graphic code body image, and the multiple boundary elements are distributed in a matrix manner;

[0165] At this time, the positioning unit can be specifically used for:

[0166] Based on the positions of the multiple bounding rectangles corresponding to the multiple target connected components, the center position of the graphic code and its tilt angle relative to the field of view coordinate system of the image sensor are determined.

[0167] In addition, the target connected component determination unit can specifically be used for:

[0168] Based on the size and / or positional distribution characteristics of the circumscribed rectangles of the multiple connected components, and the size and / or positional distribution characteristics of the boundary elements, the confidence level of each connected component belonging to the boundary elements is determined, so that at least one connected component with the highest confidence level is determined as the target connected component, and the determined position of the center point of the graphic code and the confidence level information are used as the positioning result of the graphic code.

[0169] In a specific implementation, the positioning unit can also be used for:

[0170] Based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangle of the target connected region and the confidence level, it is determined whether the graphic code image acquired by the image sensor is complete;

[0171] If incomplete, the outer rectangle is completed based on the size characteristics of the outer rectangle of at least one target connected component and the size characteristics of the boundary element. Then, the center point position of the graphic code is determined based on the center point position of the completed rectangle.

[0172] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0173] And an electronic device, comprising:

[0174] One or more processors; and

[0175] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0176] in, Figure 10 An exemplary architecture of an electronic device is shown, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020 can communicate with each other via a communication bus 1030.

[0177] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0178] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system 1021 for controlling the operation of the electronic device 1000, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 1000. Additionally, it can store a web browser 1023, a data storage management system 1024, and a graphic code positioning processing system 1025, etc. The aforementioned graphic code positioning processing system 1025 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0179] Input / output interface 1013 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0180] The network interface 1014 is used to connect the communication module (not shown in the figure) to enable communication and interaction between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0181] Bus 1030 includes a pathway for transmitting information between various components of the device (e.g., processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020).

[0182] It should be noted that although the above-described device only shows the processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, memory 1020, bus 1030, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0183] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0184] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0185] The foregoing has provided a detailed description of the graphic code localization, connected component recognition method, and robot provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for locating images using graphic codes, characterized in that, include: During the acquisition of images containing graphic codes using an image sensor, connected component identification is performed on the acquired images to determine the position information of the circumscribed rectangles of multiple connected components in the field-of-view coordinate system of the image sensor. The acquired images include the body image of the graphic code and images of at least one boundary element located outside the body image; the boundary element images are not connected to the body image. The connected component identification process includes: after receiving each row of image acquisition results, binarizing each pixel in the current row and identifying connected segments; comparing the connected segment identification results in the current row with those in the previous row, and then updating the connected component identifiers and the position information of the circumscribed rectangles of the connected components. Based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components, at least one target connected component that conforms to the characteristics of the boundary elements is determined; The positioning information of the graphic code is obtained based on the position information of the outer rectangle of the at least one target connected region.

2. The method according to claim 1, characterized in that, The process of performing connected component identification on the acquired images includes: The image sensor's pixel stream is received line by line via a hardware programmable chip, and after each line of image acquisition results is received, the following processing is performed on a line-by-line basis: Binarize each pixel in the current row; After identifying connected segments in the current row after binarization, and comparing them with the connected segment identification results in the previous row stored in the register of the hardware programmable chip, the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register are updated. Save the connected segment identification result of the current row to the register, and delete the connected segment identification result of the previous row from the register.

3. The method according to claim 2, characterized in that, After identifying multiple connected components and determining the position information of the bounding rectangles of each connected component, the hardware programmable chip saves the information to the memory of the main control system so that the software positioning module running in the main control system can execute the steps of determining the target connected component and locating the graphic code.

4. The method according to claim 1, characterized in that, The boundary element includes a single rectangular border element; The step of determining at least one target connected component that conforms to the characteristics of the boundary elements based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components includes: The characteristics of the boundary elements are determined in advance based on the side length and / or aspect ratio of the rectangular border; Based on the side length and / or aspect ratio of the circumscribed rectangle of each connected component, determine a single target connected component that conforms to the characteristics of the boundary element.

5. The method according to claim 4, characterized in that, The step of obtaining the positioning information of the graphic code based on the position information of the circumscribed rectangle of the at least one target connected component includes: The center position of the outer rectangle of the single target connected region is determined as the center position of the graphic code.

6. The method according to claim 4, characterized in that, The step of obtaining the positioning information of the graphic code based on the position information of the circumscribed rectangle of the at least one target connected component includes: The tilt angle of the graphic code relative to the field-of-view coordinate system of the image sensor is determined based on the ratio between the side length of the outer rectangle of the single target connected region and the side length of the rectangle's border element.

7. The method according to claim 4, characterized in that, The process of performing connected component identification also includes: Determine and save the location information of multiple circumscribed points of each connected component and its respective circumscribed rectangle; The step of obtaining the positioning information of the graphic code based on the position information of the circumscribed rectangle of the at least one target connected component includes: Based on the positions of multiple circumscribed points of the single target connected region and its circumscribed rectangle, the magnitude and direction of the tilt angle of the graphic code relative to the field-of-view coordinate system of the image sensor are determined.

8. The method according to claim 1, characterized in that, The boundary elements include: multiple boundary elements distributed at corresponding positions of the vertices of the graphic code body image, and the multiple boundary elements are distributed in a matrix manner; The step of obtaining the positioning information of the graphic code based on the position of the outer rectangle of the at least one target connected component includes: Based on the positions of the multiple bounding rectangles corresponding to the multiple target connected components, the center position of the graphic code and its tilt angle relative to the field of view coordinate system of the image sensor are determined.

9. The method according to any one of claims 1 to 8, characterized in that, The step of determining at least one target connected component that conforms to the characteristics of the boundary elements based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components includes: Based on the size and / or positional distribution characteristics of the circumscribed rectangles of the multiple connected components, and the size and / or positional distribution characteristics of the boundary elements, the confidence level of each connected component belonging to the boundary elements is determined, so that at least one connected component with the highest confidence level is determined as the target connected component, and the determined position of the center point of the graphic code and the confidence level information are used as the positioning result of the graphic code.

10. The method according to claim 9, characterized in that, The step of obtaining the location information of the graphic code includes: Based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangle of the target connected region and the confidence level, it is determined whether the graphic code image acquired by the image sensor is complete; If incomplete, the outer rectangle is completed based on the size characteristics of the outer rectangle of at least one target connected component and the size characteristics of the boundary element. Then, the center point position of the graphic code is determined based on the center point position of the completed rectangle.

11. A method for connected component identification of graphic codes, characterized in that, This includes performing the following steps via a hardware-programmable chip: During the process of receiving the pixel stream acquired by the image sensor line by line, the following processing is performed on a line-by-line basis: Binarize each pixel in the current row; After identifying connected segments in the current row after binarization, and comparing them with the connected segment identification results in the previous row stored in the register of the hardware programmable chip, the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register are updated. Save the connected segment identification result of the current row to the register, and delete the connected segment identification result of the previous row from the register; After completing the connected segment identification of the last row of images acquired by the image sensor, and updating the position information of the connected component identifier and the circumscribed rectangle of the connected component, the connected component identification result is output from the register.

12. A robot, characterized in that, The robot includes an image sensor, a hardware programmable chip, and a main control system, in which a software positioning module runs. The image sensor is used to acquire images containing graphic codes line by line; The hardware programmable chip is used to perform connected component identification on the image acquired by the image sensor by analyzing it line by line during the process of receiving the pixel stream acquired by the image sensor line by line, and write the position information of the identified multiple connected components into the memory of the main control system. The software positioning module in the main control system is used to determine at least one target connected component that meets the target conditions from the plurality of connected components, and to determine the positioning information of the graphic code based on the position information of the at least one target connected component, so as to adjust the navigation posture of the robot according to the positioning information of the graphic code.

13. The robot according to claim 12, characterized in that, The hardware programmable chip is specifically used to: after receiving each row of image acquisition results, perform binarization processing on each pixel in the current row, identify connected line segments, compare the identified connected line segments in the previous row stored in the register of the hardware programmable chip, update the connected component identifier and the position information of the bounding rectangle of the connected component stored in the register, save the identified connected line segments in the current row to the register, and delete the identified connected line segments in the previous row from the register.

14. The robot according to claim 13, characterized in that, The image acquired by the image sensor includes the body image of the graphic code and the image of at least one boundary element located outside the body image, wherein the image of the boundary element is not connected to the body image; The software positioning module in the main control system is specifically used to: determine at least one target connected region that conforms to the characteristics of the boundary element based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected regions; and obtain the positioning information of the graphic code based on the positional information of the circumscribed rectangles of the at least one target connected region.

15. A graphic code positioning device, characterized in that, include: A connected component identification unit is used to perform connected component identification on the acquired image during the acquisition of an image containing a graphic code by an image sensor, so as to determine the position information of the circumscribed rectangles of multiple connected components in the field-of-view coordinate system of the image sensor; wherein the acquired image includes the body image of the graphic code and the image of at least one boundary element located outside the body image, and the image of the boundary element is not connected to the body image; the connected component identification of the acquired image includes: after receiving each row of image acquisition results, performing binarization processing on each pixel in the current row, and identifying connected line segments from it; comparing the connected line segment identification results in the current row with the connected line segment identification results in the previous row, and updating the connected component identifiers and the position information of the circumscribed rectangles of the connected components; The target connected component determination unit is used to determine at least one target connected component that conforms to the characteristics of the boundary element based on the size characteristics and / or positional distribution characteristics of the circumscribed rectangles of the plurality of connected components. A positioning unit is used to obtain the positioning information of the graphic code based on the position information of the outer rectangle of the at least one target connected region.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 10.

17. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 10.

Images