Graphical fiducial marker recognition for augmented reality, virtual reality, and robotics

Abstract

The present invention discloses graphical fiducial marker recognition suitable for augmented reality, virtual reality, and robotics. In various examples, image data representing an image is received. Corner detection can be used to identify pixels that may be candidate corner points. The image data can be converted from a higher dimensional color space to a converted image in a lower dimensional color space, and boundaries within the converted image can be identified. A set of candidate corner points within a threshold distance to one of the boundaries is determined, and the set of candidate corner points can be analyzed to determine a subset of candidate corner points representing corners of a polygon. Using the subset of candidate corner points, one or more polygons can be identified, and a filter can be applied to the polygons to identify polygons corresponding to the fiducial marker boundaries of a fiducial marker.

Description

Background Art

[0001] Fiducial markers, such as AprilTags, ARTags, ARToolkit, ARToolkitPlus, RUNE-Tags, reacTIVision, QR codes, etc., have been used in the fields of virtual reality, augmented reality, robotics, and other object positioning technologies (e.g., robot-to-robot positioning), object recognition, detecting the position of an object, detecting the orientation of an object, testing virtual reality headsets, tracking objects in an environment, simultaneous localization and mapping (SLAM) algorithm estimation, camera calibration, and other purposes. Typically, fiducial markers are deployed as graphical data patterns in a predetermined arrangement within a polygon, each pattern uniquely mapping to a corresponding data record (user account, unit, product, message, etc.). In order to use fiducial markers for these purposes, specialized algorithms are used to detect and identify fiducial markers in a scene or environment.

[0002] Some conventional methods for detecting fiducial markers rely on graph-based image segmentation algorithms to identify lines in an input image and group them into polygons. These methods result in a large number of identified polygons (e.g., quadrilaterals), which results in a consumption of computational resources when filtering the polygons to identify actual fiducial markers in the image. In addition, some conventional methods use segmentation to identify boundaries in an image and then analyze each pixel along the boundary to determine the corners of the polygon. However, analyzing each pixel along the boundary is inefficient and results in significant computational, time, and energy costs.

[0003] As described above, each of these conventional approaches results in a significant consumption of computational and energy resources. This consumption is exacerbated by their reliance on a central processing unit (CPU) to identify the fiducial markers. For example, due to CPU processing limitations, these conventional approaches may be able to operate at a frame rate of 30 frames per second (fps) for an input image having a resolution of 640x480 (e.g., 480p), for example, but may only be able to operate at a frame rate of 10fps for an input image having a resolution of 1920x1080 (e.g., 1080p). Such low frame rates cannot support the functionality required for many uses of fiducial markers, particularly as the resolution of input images continues to increase (e.g., to 3840x2160 (e.g., 4k), 7680x4320 (e.g., 8k), or higher). Summary of the invention

[0004] Embodiments of the present disclosure relate to graphical fiducial marker recognition. More specifically, systems and methods are disclosed that use computer vision processing, which in some examples is at least partially implemented on a graphics processing unit (GPU), to recognize fiducial markers using image data representing an environment containing the fiducial markers.

[0005] Unlike conventional systems such as those described above, the present system can implement filtering and segmentation on an input image to identify boundaries in the input image. By identifying boundaries in this manner, the disadvantages of conventional methods associated with identifying a large number of quadrilaterals or other polygons in an input image are greatly reduced. Additionally, in some examples in parallel with the image thresholding and segmentation processes, the present system can implement corner detection for identifying pixels corresponding to corners of objects in the input image. The identified corners are then filtered so that only corners that are within a threshold distance from one of the boundaries in the input image are retained. By identifying and filtering corners in this manner, only a small number of pixels are required to detect and process polygons in the input image, thereby greatly reducing the computational cost of identifying fiducial markers.

[0006] In addition, further compared to conventional systems, the present system can implement at least some of the processes on one or more GPUs. In doing so, the efficiency of executing the processes can be improved, especially when two or more processes are executed in parallel (e.g., image thresholding, image segmentation, and / or corner detection). In addition, by offloading some of the processing to one or more GPUs, one or more central processing units (CPUs) only require a small amount of processing, thereby improving the overall efficiency and effectiveness of the system while also reducing computational and energy requirements. For example, the processes described herein can enable the system to effectively perform at higher image resolutions (e.g., 1080p, 4k, etc.), such as effectively identifying fiducial markers in an input image at a frame rate of 30 frames per second (fps) or greater. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present system and method for graphical fiducial marker recognition will be described in detail below with reference to the accompanying drawings, in which:

[0008] Figure 1A is a system diagram of a fiducial marker recognition system according to some embodiments of the present disclosure;

[0009] Figure 1B According to some embodiments of the present disclosure, the Figure 1A An example data flow diagram of a process implemented by a fiducial marker recognition system in;

[0010] Figure 2A is an example data flow diagram for fiducial marker identification according to some embodiments of the present disclosure;

[0011] Figure 2B is another example data flow diagram for fiducial marker identification according to some embodiments of the present disclosure;

[0012] Figure 2C is another example data flow diagram for fiducial marker identification according to some embodiments of the present disclosure;

[0013] Figure 3 An example description of a portion of a method for identifying a reference mark according to some embodiments of the present disclosure;

[0014] Figure 4 is a flowchart illustrating a method of benchmark mark recognition according to some embodiments of the present disclosure;

[0015] Figure 5 is a flow chart illustrating another method of fiducial marker identification according to some embodiments of the present disclosure; and

[0016] Figure 6 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Systems and methods for using a graphics processing unit for fiducial marker recognition are disclosed. Figure 1A , Figure 1A 1 is an example system diagram of a fiducial marker recognition system 100 according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are described in detail only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and certain elements may be omitted altogether. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in combination with other components, and in any suitable combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in memory.

[0018] The fiducial marker identification system 100 may include one or more computing devices 102, one or more computing devices 130, and / or one or more networks 128. Although one or more computing devices 102 and one or more computing devices 130 are shown in FIG. 1 , this is not intended to be limiting. In any embodiment, there may be any number of computing devices 102 and / or one or more computing devices 130. The fiducial marker identification system 100 (and its components and / or features) may be implemented using one or more computing devices, such as Figure 6 The computing device 600 will be described in detail herein.

[0019] The components of the fiducial marker identification system 100 may communicate via one or more networks 128. The one or more networks may include a wide area network (WAN) (e.g., the Internet, a public switched telephone network (PSTN), etc.), a local area network (LAN) (e.g., Wi-Fi, ZigBee, Z-wave, Bluetooth, Bluetooth low energy (BLE), Ethernet, etc.), a low power wide area network (LPWAN) (e.g., LoRaWAN, Sigfox, etc.), a global navigation satellite system (GNSS) network (e.g., a global positioning system (GPS), and / or other network types. In any example, each component of the fiducial marker identification system 100 may communicate with one or more other components via one or more networks 128.

[0020] One or more computing devices 102 (and / or one or more computing devices 130) may include a smartphone, a laptop, a tablet, a desktop computer, a wearable device, a gaming console, a virtual reality system (e.g., a headset, a computer, a vehicle, a gaming console, one or more remote controls, one or more controllers, and / or other components), an augmented reality system, a smart home device (which may include an intelligent personal assistant, a robotic device, a smart or IoT camera), and / or any other device capable of fiducial marker recognition (e.g., capable of analyzing an input image to identify one or more fiducial markers in the input image).

[0021] One or more computing devices 102 may include an image manager 104, a thresholder 106, an image segmenter 108, a corner detector 110, a corner filter 112, a quad fitter 114, a decoder 116, a fiducial marker manager 118, one or more cameras 120, one or more graphics processing units (GPUs) 122, one or more central processing units (CPUs) 124, one or more data stores 126, and / or additional or alternative components. In some examples, one or more components or features may be implemented by the first computing device 102, and one or more other components or features may be implemented by the second computing device 102 and / or 130. For example, at least some of the processes 132 may be implemented by the first computing device 102, and at least some of the processes 132 may be implemented by the second computing device 102 and / or 130. In other examples, each process block in the process 132 may be implemented by a single computing device.

[0022] In one non-limiting example, a first computing device 102 may capture one or more images using one or more cameras 120, and another computing device 102 and / or 130 may process one or more images according to a process 132. In another non-limiting example, the same computing device 102 may capture one or more images using one or more cameras 120, and may process one or more images according to a process 132. For example, the first computing device 102 may be located in a robot, and the second computing device 102 and / or 130 may be one or more servers and / or in another robot, such that the robot may capture one or more images and / or perform some processing, while other processing is offloaded to one or more servers and / or other robots. In some examples, one or more computing devices 102 may be a virtual reality system, and the system may include a headset, a computer, and / or communicate with one or more servers via one or more networks 128. Thus, the process 132 may be performed by the headset, the computer, and / or the one or more servers. Other examples are contemplated without departing from the scope of this disclosure. In other words, any combination of one or more computing devices 102 and / or 130 may be used to implement the fiducial marker identification system and execute process 132 for fiducial marker identification and / or other processes.

[0023] The fiducial marker identification system 100 may be used to perform fiducial marker identification according to any of a number of different methods and / or processes. In other words, although reference may be made herein to Figure 1B The data flow diagram of system 100 depicts the components and functionality of process 132, illustrating fiducial marker identification, but this is not intended to be limiting. For example, while adaptive thresholding 136 and image segmentation 138 are shown as being performed in parallel with corner detection 140, this is not intended to be limiting. In some examples, additional or alternative process blocks may be performed in parallel, or none of the process blocks in process 132 may be performed in parallel. As another example, except for Figure 1B In addition to the process blocks shown in , other or alternative process blocks may be implemented by the fiducial marker identification system 100 .

[0024] Reference now Figure 1B , Figure 1B According to some embodiments of the present disclosure, the Figure 1A100. The process 132 may be implemented on one or more GPUs 122 and one or more CPUs 124 on one or more computing devices 102 (but not limited to). For example, a process block of the process 132 with a large computational overhead may be implemented on one or more GPUs 122 to improve efficiency. In addition, by implementing at least some of the process blocks on one or more GPUs 122, two or more process blocks may be implemented in parallel (e.g., using NVIDIA's CUDA parallel processing). For example, since one or more GPUs 122 are capable of executing a large number of threads simultaneously, and one or more GPUs 122 are more suitable for processing image data than one or more CPUs 124, the computation time for identifying a fiducial marker in an image may be greatly reduced by implementing at least some of the process blocks on one or more GPUs 122 compared to the conventional methods described herein. In addition, as described herein, even if implemented on one or more CPUs 124, the process 132 itself includes process blocks that reduce computation time compared to conventional methods. Therefore, by creating a more efficient process 132 and implementing the process 132 at least in part on one or more GPUs 122, the overall processing requirements, computing power, energy, and bandwidth resources required to identify fiducial markers in an image may be significantly reduced.

[0025] In some examples, without limitation, Figure 1B The process blocks of process 132 shown in the middle dashed line 150 can be executed at least partially on one or more GPUs 102 of one or more computing devices 102, while the process blocks shown below the dashed line 150 can be executed at least partially on one or more CPUs 124 of one or more computing devices 102.

[0026] The process 132 may include input image receiving 134. For example, one or more input images 202 ( Figure 2A ) may be received and / or managed by the image manager 104 (e.g., in response to one or more input images 202 being captured by one or more cameras 120). For example, the one or more input images 202 may be stored by the image manager 104 in the one or more data stores 126 upon input image reception 134, and / or may be retrieved by the image manager 104 from one or more data stores 126 in order to pass the input images 202 to another process block (e.g., adaptive thresholding 136, corner detection 140, etc.).

[0027] In some examples, input image 202 may have a higher dimensional color space than a converted image generated as a result of adaptive thresholding 136 (described in greater detail herein). For example, input image 202 may be a color image, such as a red, green, and blue (RGB) color image, a cyan, magenta, yellow, and black (CMYK) color image, an indexed color image, a hue, saturation, and brightness (HSB) color image, and / or other image types.

[0028] like Figure 2A As shown, input image 202 may include any number of fiducial markers 204 (e.g., fiducial marker 204A, fiducial marker 204B, fiducial marker 204C, etc.). Figure 2A-2C and Figure 3 The fiducial markers 204 described include AprilTags, but are not intended to be limiting. For example, similar processes and / or process blocks of process 132 may be performed using ARKit fiducial markers, ARToolkit, ARToolKitPlus, and / or other fiducial marker types without departing from the scope of the present disclosure.

[0029] For example, one or more of the input images 202 may also include other objects, such as the table 206 and / or one or more other objects (not illustrated) in the background 208 of the input image 202 .

[0030] The image manager 104 may transmit, send, and / or pass the input image 202 to the thresholder 106 for adaptive thresholding 136 of the input image 202, and / or transmit, send, and / or pass the input image 202 to the corner detector 110 for corner detection 140. By way of example and not limitation, the thresholder 106 may receive the input image 202 and perform adaptive thresholding 136 on the input image 202. However, in other examples, the thresholder 106 may perform other types of thresholding. For example, the thresholder may perform global thresholding, local thresholding, adaptive thresholding, and / or combinations thereof in addition to or in lieu of adaptive thresholding.

[0031] When implementing the adaptive threshold 136, the thresholder 106 may convert the input image 202 into a grayscale version of the input image 202 (e.g., a grayscale input image). Once the grayscale input image is generated, the adaptive threshold 136 may be performed to find the minimum and maximum values ​​in the area around each pixel of the grayscale input image. However, in some examples, the grayscale input image may be divided into pixel tiles (e.g., 4x4 pixels in some examples), rather than calculating the exact extreme values ​​(e.g., maximum and minimum values) around each pixel of the grayscale input image. The extreme values ​​(e.g., maximum and minimum values) in each tile may then be calculated. In order to prevent artifacts from being generated between tile boundaries where there are large differences in extreme values, when calculating the extreme values ​​of adjacent pixels in the input grayscale image, the extreme values ​​in a set of surrounding tiles (e.g., 3x3 adjacent tiles in some examples) may be used (e.g., to ensure that at least one tile overlap is considered when calculating the extreme values). Therefore, each pixel may obtain the maximum and minimum values ​​from the surrounding tiles as its maximum and minimum values. If the maximum and minimum values ​​in a given tile are too close, the tile can be identified as not having sufficiently high contrast, and the pixels in the tile can be assigned pixel values ​​associated with gray (or one or more other colors) and omitted from further processing. When the tile has sufficient contrast, a white or black value can be assigned to each pixel using the average value as a threshold (e.g., (maximum value + minimum value) / 2).

[0032] The result of the adaptive thresholding 136 may include a converted image 210 (also referred to herein as the thresholded image 210). In some examples, the converted image 210 may be a binary image and / or referred to as a binary image (e.g., because the only pixels used in further processing may be black pixels and white pixels). Therefore, as described herein, the converted image 210 may have a lower dimensional color space than the input image 202. As shown in the converted image 210, due to the adaptive thresholding 136, areas 212 where black pixels are adjacent to white pixels and white pixels are adjacent to black pixels (e.g., white areas 212A and black areas 212B) may remain black and white after the adaptive thresholding 136, while other portions 214 of the converted image 210 (e.g., pixels therein) may be gray (e.g., illustrated by closely spaced diagonal lines). For example, an area where a tile of black pixels is surrounded by multiple tiles of black pixels may not have sufficient contrast and, therefore, may be converted to gray. In some examples, any pixels assigned a pixel value representing grayscale (or another color indicating unimportant pixels used to identify a fiducial marker) may be excluded from any future processing blocks of process 132 (e.g., image segmenter 108 may exclude such pixels from processing when performing image segmentation 138).

[0033] When identifying fiducial markers, such as AprilTags, ARTags, and / or similar markers, the pixel color associated with the fiducial marker in one or more images is only black or white (e.g., because the fiducial marker is black and white), and therefore, this process of adaptive thresholding 136 can allow consistent differentiation between white pixels and black pixels that make up the fiducial marker while ignoring the remaining pixels that are changed to another color (e.g., gray).

[0034] The thresholder 106 may transmit, send, and / or pass the converted image 210 to the image segmenter 108. The image segmenter 108 may perform image segmentation 138 to group continuous white regions 212A and black regions 212B of the converted image 210 and extract boundaries that segment the regions 212.

[0035] Conventional methods have identified pixels with opposite color neighboring pixels (e.g., white pixels adjacent to black pixels), and then formed connected groups of these identified pixels (e.g., continuous groups of white pixels, each of which is adjacent to black pixels) as edges. However, these conventional methods may not work when the white pixel grouping is only one pixel wide (e.g., due to a mark that is far away, or a physically small mark). In these methods, a grouping of black pixels adjacent to a single pixel wide grouping of white pixels may be incorrectly identified as a single edge, resulting in the fiducial marker not being detected.

[0036] In order to overcome these shortcomings of conventional methods, the image segmenter 108 can use union-find-based region clustering and region boundary extraction. The region clustering based on the union-find may include segmenting or grouping the converted image 210 into continuous black regions and continuous white regions (e.g., white regions 212A and black regions 212B). In order to segment the converted image 210 in this way, adjacent white pixels in the converted image 210 may be grouped together, and adjacent black pixels in the converted image 210 may be grouped together. This process may include local merging, boundary processing, and global union merging. Each process may be implemented on a separate kernel using one or more GPUs 122 (e.g., local merging on a first CUDA kernel, boundary processing on a second CUDA kernel, and global union merging on a third CUDA kernel).

[0037] For the local merging process of the regional clustering based on merging search, the converted image 210 can be divided into pixel blocks (e.g., 4x4 pixels, 8x8 pixels, etc.). In some examples, each block can be assigned to different threads on one or more GPUs 122, and different threads can use shared memory to cooperate with each other and / or can synchronize with each other. Row scanning and column scanning can be performed to each pixel block so that each pixel is linked to its left neighboring pixel and upper neighboring pixel. In some examples, only when adjacent pixels have the same intensity (e.g., black or white), they can be linked. After local merging, there may be no link between the pixel blocks, only between the pixels in the same block.

[0038] For the boundary processing of the regional clustering based on merge search, the cells (or pixels) along the boundary of the pixel block can be linked to the cells (or pixels) along the boundary of the adjacent pixel block. For example, another row scan and column scan can be performed on the boundary of each pixel block for the adjacent boundaries of the adjacent pixel blocks. After the local merge processing and boundary processing, each pixel in the converted image 210 can have a link to the adjacent pixel of the same intensity (e.g., white or black).

[0039] For the global joint merging process of the merge-find based region clustering, each pixel in the continuous black region or the continuous white region can be linked to the same representative parent pixel. Then, each continuous black region and each continuous white region can be assigned a unique region identifier (ID).

[0040] For region boundary extraction, the boundary between continuous black regions and continuous white regions can be extracted. Region boundary extraction can be performed on one or more GPUs 122, and more specifically, can be performed using a processing kernel (e.g., a CUDA kernel). For example, the boundary can be represented as an unordered set of points marked by a region ID. The boundary can be stored in a two-dimensional boundary grid, where the cell position within the pixel block can encode the boundary coordinates, and the cell value can indicate the region ID. Then the boundary points between each black pixel and the adjacent or neighboring white pixels can be identified. For example, each black pixel with a region ID of "ID1" with an image coordinate of "P1" and an adjacent white pixel with a region ID of "ID2" with an image coordinate of "P2" can be identified. Then, the boundary point can be indexed by the image coordinate (e.g., (P1+P2) / 2) and a boundary identifier (ID) value assigned based on the ID of one or more adjacent regions (e.g., by combining two 32-bit region IDs by moving one of them by 32 bits and adding another ID to obtain a unique 64-bit boundary ID).

[0041] For example, for the converted image 210, the boundary between the white region 212A and the black region adjacent to or adjacent to the white region 212A can be identified as a boundary using a region boundary extraction process. Similarly, for each black region extending adjacent to the white region, other boundaries can be identified, and each boundary can be assigned a boundary ID, wherein the boundary ID indicates a single continuous boundary.

[0042] Thus, compared to the conventional system described above, due to image segmentation 138, a single pixel wide white region and a second black region adjacent to a first black region (e.g., on either side of the white region) will result in the identification of two different boundaries (e.g., a first boundary between the first black region and the white region and a second boundary between the second black region and the white region).

[0043] The image segmenter 108 may transmit, send, and / or pass the boundary information to the corner filter 112. In addition, as described in more detail below, the corner filter 112 may also receive an output from the corner detector 110 after corner detection 140. Since the adaptive threshold 136 and image segmentation 138 process blocks are not required to perform corner detection 140, in some examples (e.g., using the parallel processing capabilities of one or more GPUs 122) these processes may be performed in parallel and then handed off to the corner filter 112 for boundary corner filtering 142.

[0044] As described herein, conventional systems may analyze each or a large number of pixels along the identified boundaries in the image to determine the corners of the polygon. However, analyzing each pixel along the boundaries is inefficient and can result in significant computational and energy costs. By using corner detection 140 in the subject technique, only relatively few points are identified as candidate corners, most of which represent actual corners 218 (such as corners 218A and 218B), compared to conventional systems. Although a few points may be temporarily misidentified as candidate corners 218 (such as candidate corners 220A and 218D), processing, computational, and energy requirements are reduced as fewer overall pixels (e.g., pixels representing candidate corners 218) are processed. In addition, compared to conventional systems, since the boundaries in the input image do not need to have been identified prior to corner detection 140, corner detection 140 can be performed in parallel with adaptive thresholding 136 and / or image segmentation 138 (e.g., the process step of identifying boundaries), thereby reducing processing time and improving the efficiency of system 100.

[0045] The corner detector 110 may be configured to detect a grayscale version of the input image 202 (eg, a grayscale input image), the input image 202, and / or the converted image 210 (eg, Figure 2B ) performs corner detection 140 to detect candidate corners 218 in the grayscale input image, the input image 202, and / or the transformed image 210 (e.g., at Figure 2B202). Visualization 216, which shows an example result of corner detection 140 represented on input image 202, may include fiducial marker 204, table 206, background, and multiple candidate corners 218. Candidate corners 218 (e.g., corners 218A-218C and unlabeled corners 218) are not intended to be limiting, as additional or alternative candidate corners 218 have been identified as a result of corner detection 140 without departing from the scope of the present disclosure. In some examples, corner detection 140 may be implemented on one or more GPUs 122, as described herein.

[0046] In some examples, corner detection 140 may result in certain pixels being marked as candidate corners even when the pixels do not represent actual corners (e.g., Figure 2B 218). For example, corner detection 140 may be sensitive to shadows, noise, and / or image resolution issues, and therefore, some pixels (e.g., misidentified pixels 220A-220D) may be identified as candidate corners 218 even though they are not actual corners. However, even with misidentified pixels 220, the result of using corner detection 140 still identifies fewer pixels (e.g., corners 218) that need to be analyzed than conventional methods. In any example, corners 218 and misidentified pixels 220 may be collectively referred to as a set of candidate corners or a set of identified candidate corners.

[0047] In some examples, additional processing may be performed to account for shadows, noise, and / or image resolution issues. For example, in an example where corner detection 140 is scale-variable (e.g., at a single scale, using only a grayscale input image and / or a converted image 210), additional processing may be performed to cover different scale spaces in the image, such as by using a variable Gaussian convolution kernel. For example, in addition to the grayscale input image and / or the converted image 210, at least one additional scale space image may be generated. In some non-limiting examples, three additional scale space images may be generated, and in addition to or instead of the grayscale input image and / or the converted image 210, the three additional scale space images may be analyzed to determine candidate corners. In some examples, each scale space image may be executed using a different kernel specific to the scale space of the image. In any example, the corners detected in one or more different scale space images and the grayscale input image and / or the converted image 210 may be compared to determine the relationship between the detected candidate corners. For example, the candidate corners detected in each scale-space image may be compared to the grayscale input image and / or the converted image 210 to determine whether the corner is supported (e.g., identified) in each image. Thus, at least some misidentified pixels detected due to noise, shadows, and / or image resolution issues (e.g., detected as candidate corners in one image but not detected as candidate corners in another image in a different scale space) may be filtered out, resulting in fewer pixels being analyzed as corners 218.

[0048] As described above, the detected candidate corners 218 (which may include the misidentified pixels 220) may then be transmitted, sent, and / or passed to the corner filter 112 for boundary corner filtering 142 (eg, Figure 2C ). Corner filter 112 may also receive boundary 308 (e.g., boundary between white pixel region 212A and black pixel region 212B) from image segmentation 138, as described above. Thus, corner filter 112 may perform boundary corner filtering 142 using corner 218, misidentified pixels 220, and boundary 308. As described above, since corner detection 140 may result in misidentified pixels 220, such as misidentified pixels 220A and 220B that are not located on boundary 308, boundary corner filtering 142 may be used to filter out at least some of misidentified pixels 220.

[0049] like Figure 3 As shown, the first pixel 302 may have been identified as the corner 218 by the corner detector 110, the second pixel 304 may have been identified as being on or near the boundary 208 but is a misidentified pixel 220, and the third pixel 306 ( Figure 3308) may have been identified as not being a corner 218, not being located on or near a boundary 208 (e.g., being a threshold distance to the boundary 308, such as two pixels, four pixels, etc. from the boundary 308), and being a misidentified pixel 220. Then, the boundary corner filter 142 is used to filter out the third pixel 306, which is a misidentified pixel 220 and is not within the threshold distance from the boundary 308. In some examples, the threshold distance may be the number of pixels from the boundary 308 (e.g., a pixel associated with the boundary 308). For example and not limitation, the threshold distance may be two pixels, three pixels, seven pixels, ten pixels, etc. Therefore, if pixels in the image are not included in the candidate corners (e.g., not included in the corner 218 and / or the misidentified pixel 220) and are not within the threshold distance from one of the boundaries 308, these pixels in the image may be filtered out. The resulting pixels (e.g., the first pixel 302 and the second pixel 304) may be collectively referred to as a subset of the candidate corner set (e.g., the pixel set may include the corner 218 and the misidentified pixel 220).

[0050] Boundary corner filtering 142 can be performed by one or more GPUs 122, and as described above, a kernel (e.g., a CUDA kernel) can be used to analyze each pixel (e.g., first pixel 302, second pixel 304, and third pixel 306) identified as a corner 218 and / or misidentified pixel 220 in parallel (e.g., each pixel can be applied to a separate thread on one or more GPUs 122).

[0051] The corner filter 112 may then transmit, send, and / or pass the pixels (e.g., the first pixel 302 and the second pixel 304) to the quad fitter 114 to perform quad fitting 144. In some examples, the quad fitting 144 is performed by the one or more CPUs 124, and thus, the pixels may be copied from a memory associated with the one or more GPUs 122 to a memory associated with the one or more CPUs 124. For example, the pixels (e.g., the location of the pixels in the image) may be retrieved from the memory of the one or more GPUs 122 and transmitted to and stored by the memory of the one or more CPUs 124. In some examples, the one or more data stores 126 may include a memory associated with the one or more GPUs 122 and / or a memory associated with the one or more CPUs 124. Unlike conventional systems where a large number of edge pixels (e.g., each pixel along a boundary identified in an image) are analyzed by one or more CPUs, system 100 may require only one or more CPUs 124 to analyze a subset of a set of pixels (e.g., first pixel 302 and second pixel 304) to determine whether a pixel represents a corner of a polygon (e.g., a quadrilateral, such as the quadrilaterals forming a fiducial marker boundary for AprilTags, ARTags, etc.). Thus, a significant amount of computational and processing time may be saved, resulting in a more efficient and robust system 100 that is capable of processing higher image resolutions (e.g., 1080p, 4k, etc.) at the same or higher frame rates (e.g., 30fps, 60fps, etc.) than conventional systems while still accurately identifying fiducial markers.

[0052] The quadrilateral fitter 114 may perform quadrilateral fitting 144 to identify the corners 218 of the polygon. Thus, although the quadrilateral fitter 114 refers to quadrilaterals or quadrilaterals, this is not intended to be limiting. For example, the quadrilateral fitter 114 may also be referred to as the polygon fitter 114, and the quadrilateral fitter 114 may also be referred to as the polygon fitter 144 (e.g., when the fiducial marker boundary is not quadrilateral).

[0053] Each corner 218 and misidentified pixel 220 may be stored in a hash table. In the hash table, each corner 218 (e.g., first pixel 302) and each misidentified pixel 220 (e.g., misidentified pixel 220 associated with boundary 308, such as second pixel 304) and a corresponding boundary ID (e.g., based on the boundary 308 associated with the pixel determined by image segmentation 138) for each boundary 308 (e.g., a boundary extending between adjacent or neighboring white pixel regions 212A and black pixel regions 212B determined during image segmentation 138) may be stored. Thus, each corner 218 and / or misidentified pixel 220 associated with the same boundary ID may be grouped together. Then, quadrilateral fitter 114 may analyze each of the separate groups to identify the best corner candidate (e.g., best corner 218 and / or misidentified pixel 220) for representing the fiducial marker boundary of fiducial marker 204.

[0054] In some examples, the pixel groups may first be analyzed based on the number of pixels associated with each boundary. For example, for a fiducial marker with a known shape boundary, the number of corner points may also be known (e.g., a square or rectangle has four corners, a triangle has three corners, etc.). Therefore, once pixel groups are formed for each boundary, the pixel groups may be analyzed based on the number of pixels that should be present for the fiducial marker type. For example, if the fiducial marker boundary is known to be a square and therefore has four corner points, any pixel groups with less than four pixels may be filtered out and not included in any further processing. In such an example, any pixel groups with four corner points may not require additional corner filtering to identify only four candidate corners, but a quadrilateral formed by four pixels may still undergo filtering to determine whether the quadrilateral is a candidate for the fiducial marker boundary (as described in more detail below). In addition, when the pixel group is greater than 4, filtering may be performed to determine four pixels that may be candidate corner points of the quadrilateral, as described in more detail below.

[0055] For each set of candidate angles, edges 310 (e.g., edge 310 of first boundary 308A, edge 310B of second boundary 308B, etc.) may be generated that extend from each candidate angle to the next closest candidate angle (e.g., the first candidate angle is clockwise relative to the centroid of the candidate angle, the second candidate angle is counterclockwise relative to the candidate angle), until a continuous string of edges 310 may be formed. In some examples, the candidate angles may be sorted according to the centroid (e.g., clockwise, counterclockwise, etc.), and edges 310 may be generated from each candidate angle to the next candidate angle. For example, for Figure 3In the quadrilateral fitting 144 in FIG. 1 , the candidate corners associated with the boundary 308A may be ordered clockwise starting from the centroid. In such an example, the candidate corners may be ordered as the first pixel 302C, the second pixel 304B, the first pixel 302D, the first pixel 302E, the second pixel 304A, the first pixel 302F, the second pixel 304C, the first pixel 302C. Thus, a first edge 310A extending between the first pixel 302C and the second pixel 304B may be generated, a second edge 310A extending between the second pixel 304B and the first pixel 302D may be generated, a third edge 310A extending between the first pixel 302D and the first pixel 302E may be generated, and so on until a continuous loop or grouping of edges extending through each candidate corner is formed.

[0056] In some examples, due to image aliasing or noise, one or more pixels may be detected as candidate corners that are not actual corners, but are misidentified pixels along an edge (e.g., an edge that forms part of a polygon). Therefore, to filter out these pixels that are misidentified as corners, once a continuous ring or grouping of edges is formed, the angle formed between each two adjacent edges 312 may be calculated so as to identify the actual corner of the polygon (e.g., a quadrilateral) from the candidate corners. In some examples, the candidate corners may be sorted in a clockwise or counterclockwise direction by their centroids. For each candidate corner, the angle 312 between a first vector (e.g., a first edge) from the candidate corner to the immediately preceding candidate corner and a second vector (e.g., a second edge) from the candidate corner to the immediately succeeding candidate corner may be calculated. The candidate corner associated with the least colinear (e.g., relative to its two adjacent corners) edge (or vector) may be considered an actual corner. In other words, the smallest angle 312 or the angle closest to 90 degrees (e.g., an example in which the polygon is a quadrilateral) can be used to identify the actual angle (e.g., the first pixel 302C) and / or filter out the candidate angles on the side of the polygon (e.g., the second pixel 304A). Therefore, the smallest angle 312 or the angle 312 closest to 90 degrees can be identified as the actual angle of the polygon (e.g., relative to the Figure 3 As another example, if the polygon is an equilateral triangle, the angle 312 closest to 60 degrees can be determined as the actual angle. Similar determinations can be made based on the shape corresponding to the boundary of the reference mark to be identified.

[0057] For example, the angle 312A formed between the edge 310A extending between the first pixel 302F and the second pixel 304C and the edge 310A extending between the first pixel 302F and the second pixel 304A may include an angle of about 90 degrees. In such an example, the angle formed between the edge 310A extending between the first pixel 302F and the second pixel 304A and the edge 310A extending between the second pixel 304A and the first pixel 302E may be about 0 degrees. Similarly, the angle 312 formed by the edge 310A extending from the first pixel 302C, 302D and 302E may be about 90 degrees, while the angle formed by the edge 310A extending from the second pixel 304B and 304C may be about 0 degrees. Therefore, the first pixel 302C, 302D, 302E and 302F can be determined as the actual corner point of the boundary 308A (e.g., it can ultimately be determined as the fiducial marker boundary).

[0058] As another example, the candidate corners associated with edge 308B may undergo similar processing as described with respect to edge 308A. Thus, four corner points may be identified (e.g., the four corner points between which edge 310B extends). Edge 310B is an identified polygon (or Figure 3 ), which ultimately will not be included as a candidate for a fiducial marker boundary (e.g., boundary 308A). Figure 2C Boundaries 222B and 22D in represent identified polygons as a result of quadrilateral fitting 144 that may not ultimately be included as candidates for fiducial marker boundaries, while boundaries 222A and 222C represent fiducial marker boundaries. Therefore, additional processing (e.g., filtering) may be employed to identify fiducial marker boundaries.

[0059] The quadrilateral fitter 114 may perform one or more filters to filter out identified polygons that may not be expected to be used as candidates for fiducial marker boundaries. For example, a first filter may be applied that finds the length of an edge 310 and / or compares the length of edges 310 of the same boundary 308 to determine a ratio or ratio. For example, some fiducial markers may require a minimum number of pixels to effectively encode data. Therefore, any edge 310 that is less than the minimum number of pixels in length may cause the polygon associated with the edge 310 to be filtered out. Another example is that some fiducial markers may include known shapes such as squares, rectangles, triangles (e.g., equilateral triangles), etc., and therefore, the ratios between different edges may indicate a polygon candidate as a fiducial marker boundary. For a specific example, when the fiducial marker boundary is known to be a square, edges 310 with different ratios (e.g., outside a threshold when considering image distortion or fiducial marker orientation in the environment) may be filtered out.

[0060] As another example, the detected angles inside polygons can be used to filter out polygons that are candidates for fiducial marker boundaries. For example, for a fiducial marker of known shape, the angles formed by the sides of the fiducial marker may be known. For example, for a square fiducial marker boundary, taking into account some variance based on the orientation of the fiducial marker in the image, image distortion, etc., the interior angles should all be approximately 90 degrees. Thus, Figure 3 For example, if the fiducial marker boundary is known to be a square, the angle 312B formed between the sides 310B of the polygon can be used to filter polygons that are candidates for the fiducial marker boundary (e.g., because the angle may be approximately 45 degrees and therefore may be outside the threshold of angles of polygons detected in the image).

[0061] The result of quadrilateral fitting 144 is a set of one or more polygons that may be candidates for fiducial marker boundaries. Quadrilateral fitter 114 may transmit, send, and / or pass the candidate polygons to decoder 116 for fiducial marker decoding 146. Decoder 116 may decode the candidate quadrilaterals using input image 202, a grayscale version of input image 202, and / or converted image 210, depending on the embodiment. Due to the use of fiducial markers that encode data (e.g., pixel data) in a pattern of pixel values, decoder 116 may decode the pattern of pixels within the candidate fiducial marker boundary (e.g., pixels within the polygon identified by quadrilateral fitter 114) and compare the decoded data (e.g., representing pixel values) with an existing or known pixel value mask or template (e.g., black and white pixel value code or fiducial marker data) of the fiducial marker. For example, the decoded data representing the code of the polygon may be compared to a code table (e.g., a query may be performed on the code table, which may be a lookup table), which may include a list of codes representing known fiducial markers (e.g., fiducial marker data). When the code associated with the polygon is included in the code table, the fiducial marker can be successfully identified (e.g. Figure 2C , and when the code is not included in the code table, the polygon can be considered a false positive and filtered out for further processing.

[0062] For example, for AprilTags, AprilTags are typically square and are typically NxN black or white cells, where N is predefined according to the AprilTag family or class. Therefore, each AprilTag in the family or class can be encoded using an NxN bit binary code, where 0 and 1 represent black and white, respectively. The decoder 116 can perform a decoding algorithm to determine the data represented by the polygons in the image (e.g., as represented by Figure 3After decoding the pixel values ​​from within boundary 308A, the decoded pixel values ​​(e.g., codes representing a fiducial tag such as AprilTag) may be compared to a lookup table and / or code table that includes a known pattern mask or template for AprilTag. Thus, the pixels within boundary 308A may be determined to be within a threshold similar to one of the codes in the code table (e.g., within a threshold-weighted average distance, such as 2, 3, 4, etc.). Thus, boundary 308A may be determined to be a successfully decoded fiducial tag (e.g., AprilTag) and may be used by fiducial tag manager 118 to perform one or more functions, execute one or more commands, or trigger one or more commands by computing device 102 and / or another component or feature of other computing devices 102 and / or 130.

[0063] Reference Figure 3 The image portions marked as “white” and “black” for corner detection 140 are also white and black in other process blocks (e.g., boundary corner filtering 142, quadrilateral fitting 144, and fiducial marker decoding 146), but are not included in the other process blocks for clarity. Figure 3 The black and white areas in the image can form fiducial markers, such as AprilTag, ARKit, ARToolkit, etc.

[0064] Reference now Figure 4 and Figure 5 , each block of the method 400 and method 500 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in a memory. The method 400 and method 500 may also be embodied as computer usable instructions stored on a computer storage medium. The method 400 and method 500 may be provided by a stand-alone application, a service or a hosted service (stand-alone or in combination with other hosted services), or a plug-in to another product, to name a few. In addition, the reference marker recognition system 100 ( Figure 1A ) and process 132( Figure 1B ) method 400 and method 500. However, these methods may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0065] Figure 4Flowchart of a method 400 for fiducial marker recognition is shown in accordance with some embodiments of the present disclosure. At block B402, the method 400 includes receiving image data representing an image. For example, the image manager 104 may receive image data representing an image (e.g., at an input image receiving 134 process block of the process 132). The input image has been captured by one or more cameras 120 of one or more computing devices 102. In some examples, the one or more input images may be stored in one or more data stores 126 and managed by the image manager 104.

[0066] At block B404, method 400 includes detecting pixels representing candidate corner points. For example, corner detector 110 may detect pixels representing candidate corner points (e.g., Figure 2B In some examples, detecting pixels representing candidate corner points may be performed by corner detector 110 at a corner detection 140 process block of process 132 .

[0067] At block B406, method 400 includes converting the image data into a converted image. For example, the image can be converted from a higher dimensional color space to a converted image of a lower dimensional color space. This can include thresholding of the image, such as adaptive threshold 136. In some examples, a grayscale version of the image can be generated before adaptive threshold 136 (and / or global threshold). In some examples, thresholder 106 can perform the conversion of the image into a converted image.

[0068] At block B408, method 400 includes identifying boundaries in the converted image. For example, separating adjacent white pixel regions (e.g., Figure 2A region 212A) and a black pixel region (e.g. Figure 2A The boundaries of the region 212B) may be identified by the image segmenter 108 as a result of the image segmentation 138 .

[0069] At block B410, method 400 includes determining a set of candidate corner points that are within a threshold distance to a boundary. For example, corner filter 112 may filter out pixels from the set of candidate corner points that are not within a threshold distance to one of the boundaries identified at block B408. In some examples, corner filter 112 may determine the set of candidate corner points when performing boundary corner filtering 142 of process 132.

[0070] At block B412, method 400 includes analyzing the set of candidate corner points to determine a subset of candidate corner points that represent corners of a polygon. For example, quadrilateral fitter 114 may analyze the set of candidate corner points to determine a subset of candidate corner points that are corners of a polygon. For example, edges extending between adjacent candidate pixels may be generated, and the angles between the adjacent edges may be analyzed to determine whether the candidate corner points are suitable as corner points. The analysis of the set of candidate corner points may be performed by quadrilateral fitter 114 when performing quadrilateral fitting 144 of process 132.

[0071] At block B414, the method 400 includes applying at least one filter to the polygons to identify polygons corresponding to the fiducial marker boundaries of the fiducial marker. For example, the polygons may be filtered based on side lengths, ratios of side lengths, angles formed by the sides, and / or based on comparing decoded data within the polygons to a code table of known masks or templates representing fiducial markers. In some examples, filtering may be applied by the quadrilateral fitter 114 during quadrilateral fitting 144 and / or may be filtered by the decoder 116 during fiducial marker decoding 146 when executing the process 132. As a result of the filter and / or decoding, the remaining polygons may be determined to be fiducial markers, and the polygons may be determined to be fiducial marker boundaries.

[0072] Figure 5 Flowchart showing another method 500 for fiducial marker identification according to some embodiments of the present disclosure. At block B502, method 500 includes determining pixels in an image that represent candidate corner points within a threshold distance to at least one boundary in the identified image. For example, pixels in a grayscale version of the converted image 210 and / or the input image 202 that are within a threshold distance to one of the identified boundaries in the grayscale version of the converted image 210 and / or the input image 202 may be determined. In some examples, the candidate corner points may be determined after input image reception 134, adaptive thresholding 136, image segmentation 138, corner detection 140, and / or boundary corner filter 142 of process 132. In some examples, each of these steps may be performed on one or more GPUs 122, and one or more process blocks may be performed in parallel, as described herein.

[0073] At block B504, process 500 includes copying pixels from GPU memory to CPU memory. For example, once one or more GPUs 122 complete boundary corner filtering 142, process 132 can continue on one or more CPUs 124. Therefore, the pixels identified at block B502 can be copied from GPU memory to CPU memory for further processing by one or more CPUs 124. However, as shown by the dashed line, block B504 is optional, and in examples where both one or more GPUs 122 and one or more CPUs 124 are not used, or in examples where the next process block is not executed on one or more CPUs 124, block B504 can be removed.

[0074] At block B506 , the process 500 includes determining a set of candidate corner points associated with the same boundary. For example, the candidate corner points associated with the same boundary ID are determined by the quadrilateral fitter 114 during quadrilateral fitting 144 .

[0075] At block B508, process 500 includes generating edges extending between members of the set of candidate corner points. For example, quadrilateral fitter 114 may generate edges extending between each candidate corner point and each adjacent candidate corner point.

[0076] At block B510, process 500 includes calculating the angle formed by the adjacent sides of the side. For example, quadrilateral fitter 114 may calculate the angle formed by the adjacent sides of the side.

[0077] At block B512, process 500 includes identifying a subset of the candidate corner point set associated with the set of angles having the least magnitude as corners of the polygon. For example, quadrilateral fitter 114 may identify a subset of the candidate corner point set associated with the set of angles having the least magnitude (e.g., Figure 3 The first pixels 302C, 302D, 302E and 302F in FIG.

[0078] At block B514, process 500 includes determining whether the polygon corresponds to a fiducial marker boundary of the fiducial marker. For example, quadrilateral fitter 114 and / or decoder 116 may determine whether one or more polygons detected in the image correspond to a fiducial marker boundary of the fiducial marker. This process may be similar to that described above. Figure 4 The process of block B414 of method 400 is similar.

[0079] Reference now Figure 6 , Figure 66 is a block diagram of an example computing device 600 suitable for implementing at least some embodiments of the present disclosure. The computing device 600 may include a bus 602 that directly or indirectly couples the following devices: a memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communication interface 610, an input / output (I / O) port 612, an input / output component 614, a power supply 616, and one or more presentation components 618 (e.g., one or more displays).

[0080] although Figure 6 The various blocks of are shown as being connected to the lines via bus 602, but this is not intended to be limiting and is provided for clarity. For example, in some embodiments, presentation component 618 (e.g., a display device) may be considered to be I / O component 614 (e.g., if the display is a touch screen). Another example is that CPU 606 and / or GPU 608 may include memory (e.g., memory 604 may represent a storage device in addition to the memory of GPU 608, CPU 606, and / or other components). In other words, Figure 6 The computing devices in the Term are illustrative only. No distinction is made between categories such as “workstations,” “servers,” “laptops,” “desktops,” “tablets,” “client devices,” “mobile devices,” “handheld devices,” “gaming consoles,” “electronic control units (ECUs),” “virtual reality systems,” “robotic devices,” and / or other device or system types, as all of these categories are in the Figure 6 is considered within the context of computing devices.

[0081] The bus 602 may represent one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. The bus 602 may include one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or other types of buses.

[0082] Memory 604 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 600. Computer-readable media may include volatile and non-volatile media and removable and non-removable media. For example, and without limitation, computer-readable media may include computer storage media and communication media.

[0083] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 604 may store computer-readable instructions (e.g., instructions representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 600. As used herein, computer storage media itself does not include signals.

[0084] Communication media may contain computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in a manner that encodes information into the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

[0085] One or more CPUs 606 may be configured to execute computer-readable instructions to control one or more components of the computing device 600 to perform one or more methods and / or processes described herein. Each of the one or more CPUs 606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of processing multiple software threads simultaneously. The one or more CPUs 606 may include any type of processor according to the type of computing device 600 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers), and may include different types of processors. For example, depending on the type of computing device 600, the processor may be an ARM processor implemented using reduced instruction set computing (RISC), or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or auxiliary coprocessors, such as a math coprocessor, the computing device 600 may also include one or more CPUs 606.

[0086] The computing device 600 may use one or more GPUs 608 to render graphics (e.g., 3D graphics). One or more GPUs 608 may include hundreds or thousands of cores that can process hundreds or thousands of software threads simultaneously. One or more GPUs 608 may generate pixel data of an output image in response to a rendering command (e.g., a rendering command received from one or more CPUs 606 via a host interface). One or more GPUs 608 may include a graphics memory, such as a display memory, for storing pixel data. The display memory may be included as part of the memory 604. One or more GPUs 608 may include two or more GPUs running in parallel (e.g., by linking). When combined together, each GPU 608 may generate pixel data for different parts of an output image or different output images (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0087] In examples where computing device 600 does not include one or more GPUs 608 , one or more CPUs 606 may be used to render graphics.

[0088] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 610 may include components and functionality to enable communication via any number of different networks, such as wireless networks (e.g., Wi-Fi, Z-wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet, etc.

[0089] The I / O port 612 may enable the computing device 600 to be logically coupled to other devices, including an I / O component 614, one or more presentation components 618, and / or other components, some of which may be built into (e.g., integrated into) the computing device 600. Illustrative I / O components 614 include microphones, mice, keyboards, joysticks, game keyboards, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. The I / O component 614 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, the input may be transmitted to an appropriate network element for further processing. The NUI may implement voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition (described in more detail below) associated with the display of the computing device 600. The computing device 600 may include a depth camera (e.g., a stereo camera system), an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof for gesture detection and recognition. In addition, computing device 600 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) for detecting motion. In some examples, the output of the accelerometer or gyroscope may be used by computing device 600 to render immersive augmented reality or virtual reality.

[0090] The power supply 616 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 616 may provide power to the computing device 600 to enable the components of the computing device 600 to operate.

[0091] One or more presentation components 618 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. One or more presentation components 618 may receive data from other components (e.g., one or more GPUs 608, one or more CPUs 606, etc.) and output data (e.g., images, videos, sounds, etc.).

[0092] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions (such as program modules) executed by a computer or other machine (e.g., a personal data assistant or other handheld device). Typically, a program module (including routines, programs, objects, components, data structures, etc.) refers to code that performs a specific task or implements a specific abstract data type. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network.

[0093] The elements described in the embodiments of the present disclosure (such as systems, components, features, machines, interfaces, functions, sequences, grouping of functions, functionality and / or arrangement) are only described as examples. In addition to or in place of the elements described herein, other elements other than the elements described herein can be used, and some elements can be omitted entirely. In addition, many of the elements described herein are functional entities, which can be implemented as discrete or distributed components, can also be used in combination with other components, and can be implemented in any suitable combination and position. The various functions performed by the entity described herein can be performed by hardware, firmware and / or software. For example, various functions can be performed by a processor that executes instructions stored in a memory.

[0094] As used herein, the recitation of "and / or" of two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. In addition, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0095] The subject matter of the present disclosure is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to the steps described in this document in conjunction with other existing or future technologies. In addition, although the terms "step" and / or "block" may be used herein to refer to different elements of the method employed, unless the order of the various steps is explicitly described, the terms should not be interpreted as implying any particular order between or among the various steps disclosed herein.

Claims

1. A method for identifying a graphic reference mark, comprising: receiving image data representing an image; converting the image data from a higher dimensional color space to a converted image in a lower dimensional color space; identifying a first pixel corresponding to a boundary of the transformed image; detecting a second pixel representing a candidate corner point; determining the set of candidate corner points within a threshold pixel distance to at least one of the boundaries based at least in part on a comparison of the first pixel and the second pixel; analyzing the set of candidate corner points to determine a subset of the candidate corner points representing corners of at least one polygon; as well as At least one filter is applied to the at least one polygon to identify a polygon of the at least one polygon that corresponds to a fiducial marker boundary of the fiducial marker. 2 . The method of claim 1 , wherein converting the image data into the converted image comprises at least one of global thresholding or adaptive thresholding.

3. The method of claim 1 , wherein identifying a first pixel corresponding to the boundary comprises: identifying a first contiguous cluster of pixels of a first color and a second contiguous cluster of pixels of a second color in the converted color image; as well as A first pixel corresponding to the boundary is determined based at least in part on identifying a location where at least one of the first contiguous clusters and at least one of the second contiguous clusters extend adjacent to each other. 4 . The method of claim 1 , wherein the fiducial marker is one of an AprilTag, an ARTag, an ARToolkit fiducial marker, or an ARToolkitPlus fiducial marker.

5. The method according to claim 1, wherein at least one of the detecting the second pixel representing the candidate corner point in the image, the converting the image data into the converted image, the identifying the first pixel of the converted image corresponding to the boundary, or the determining that at least one of the candidate corner points in the boundary is within the threshold pixel distance is performed by a graphics processing unit (GPU).

6. The method of claim 1 , wherein converting the image data into the converted image comprises: converting the image data into grayscale image data representing a grayscale version of the image; thresholding the grayscale image data to generate binary image data representing a binary version of the image, Wherein the converted image is the binary version of the image.

7. The method of claim 1, wherein identifying a first pixel corresponding to the boundary of the converted image is performed in parallel with detecting a second pixel in the image representing the candidate corner point.

8. The method of claim 1 , wherein analyzing the set of candidate corner points to determine the subset of the candidate corner points representing the corners of the at least one polygon comprises: Determining members of the candidate corner point set associated with the same boundary ID; generating edges extending between the members of the candidate corner point set; calculating the angle formed by the adjacent sides of the side; as well as The member of the set of candidate corner points associated with the corner set having the smallest amount is identified as the subset of the candidate corner points.

9. The method of claim 1 , wherein applying the at least one filter to the at least one polygon comprises: decoding pixel values ​​within the at least one polygon to generate pixel data representing the at least one polygon; comparing the pixel data to fiducial marker data representing a fiducial marker; as well as A determination is made based at least in part on the comparison whether any one of the at least one polygon corresponds to any one of the fiducial markers.

10. A method for identifying a graphic reference mark, comprising: receiving data representing a boundary in a first image and an associated boundary identifier ID of said boundary; determining pixels in at least one of the first image or the second image representing at least one candidate corner point within a threshold pixel distance to the boundary; For each of the boundaries: determining the set of candidate corner points associated with the same boundary ID based at least in part on the boundary ID; generating edges extending between members of the candidate corner point set; calculating the angle formed by the adjacent sides of the side; as well as identifying a subset of the set of candidate corner points associated with the set of corners having a minimum amount as corners of a quadrilateral; as well as At least one filter is applied to the quadrilateral to determine whether the quadrilateral corresponds to a fiducial marker boundary of a fiducial marker. 11 . The method according to claim 10 , wherein at least a first part of the method is executed on a central processing unit (CPU), and a second part of the method is executed on a graphics processing unit (GPU).

12. The method of claim 10, wherein applying at least one filter to the quadrilateral comprises: decoding pixel values ​​within the quadrilateral to determine pixel data representing the quadrilateral; comparing the pixel data to fiducial marker data representing a fiducial marker; as well as Based at least in part on the comparison, a determination is made as to whether the quadrilateral corresponds to any of the fiducial markers.

13. The method of claim 10, wherein generating the edges extending between the members of the candidate corner point set comprises, for each of the members: generating a first edge extending from the member to a first neighboring member of the member; and A second edge is generated extending from the member to a second adjacent member of the member.

14. The method of claim 10, wherein: Based at least in part on said generating the edge, a first edge extends from a first member of the members of the candidate corner points to a second member of the members that is adjacent to the first member; Based at least in part on the generating the edge, a second edge extends from the first one of the members of the candidate corner points to a third one of the members that is adjacent to the first member; The first side and the second side are included in the adjacent sides; as well as The calculating the angle formed by the adjacent sides of the side includes calculating an angle formed by the first side and the second side.

15. The method of claim 10, wherein the fiducial marker is one of an AprilTag, an ARTag, an ARToolkit fiducial marker, or an ARToolkitPlus fiducial marker.

16. A method for identifying a graphical reference mark, comprising: Determining, by one or more graphics processing units (GPUs), a first pixel in the image representing a candidate corner point; determining, by the one or more GPUs, a second pixel in the image that represents a boundary of the image; determining, by the one or more GPUs, the set of candidate corner points corresponding to the first pixel within a threshold pixel distance from one or more of the second pixels corresponding to the boundary; Copying, by the one or more GPUs, the set of candidate corner points from a GPU memory to a central processing unit CPU memory, the CPU memory being associated with one or more CPUs; For each of the boundaries: determining, by the one or more CPUs, a subset of the set of candidate corner points associated with a same boundary of at least one boundary; generating, by the one or more CPUs, edges extending between members of a subset of the set of candidate corner points; calculating, by the one or more CPUs, an angle formed by adjacent sides of the side; as well as identifying, by the one or more CPUs, the candidate corner point as a corner of a quadrilateral from a subset of the set of candidate corner points associated with the set of corners having a minimum amount; as well as A determination is made, by the one or more CPUs, whether the quadrilateral corresponds to a fiducial marker boundary of a fiducial marker.

17. The method of claim 16, wherein determining whether the quadrilateral corresponds to the fiducial marker boundary of the fiducial marker comprises: decoding pixel values ​​within the quadrilateral to determine pixel data representing the quadrilateral; comparing the pixel data to fiducial marker data representing a fiducial marker; as well as Based at least in part on the comparison, a determination is made as to whether the quadrilateral corresponds to any of the fiducial markers.

18. The method of claim 16, wherein generating the edges extending between the members of the subset of the candidate corner point set comprises, for each of the members: generating a first edge extending from the member to a first neighboring member of the member; and A second edge is generated extending from the member to a second adjacent member of the member.

19. The method according to claim 16, wherein Based at least in part on said generating the edge, a first edge extends from a first member of the members of the candidate corner points to a second member of the members that is adjacent to the first member; Based at least in part on the generating the edge, a second edge extends from the first one of the members of the candidate corner points to a third one of the members that is adjacent to the first member; The first side and the second side are included in the adjacent sides; as well as The calculating the angle formed by the adjacent sides of the side includes calculating an angle formed by the first side and the second side.

20. The method of claim 16, wherein the fiducial marker is one of an AprilTag, an ARTag, an ARToolkit fiducial marker, or an ARToolkitPlus fiducial marker.

Images