Visual markers

Abstract

Various implementations disclosed herein include apparatuses, systems, and methods that provide a visual marker including a plurality of markers arranged in a corresponding plurality of shapes. In some implementations, each marker is formed by a set of sub-markers separated by gaps and arranged according to a respective shape, and the gaps of the plurality of markers are configured to encode data and indicate an orientation of the visual marker. In some implementations, the plurality of markers are arranged in a plurality of concentric rings of increasing size. In some implementations, the orientation is encoded in a first set of the gaps in the plurality of markers, and data is encoded in a second set of the gaps.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to visual markers that convey information and to systems, methods, and devices that capture and interpret images of such visual markers to obtain and use the conveyed information. BACKGROUND

[0002] Today, visual markers exist in the form of barcodes, quick response (QR) codes, and other proprietary formats. QR codes encode binary data such as a string or other payload. SUMMARY

[0003] Various implementations disclosed herein include visual markers having a plurality of markings arranged in a plurality of shapes. In some implementations, the markings of the visual marker can be configured to both indicate an orientation of the visual marker and convey information. In some implementations, each marking is formed from a set of sub-markings that are separated by gaps and arranged according to a respective shape. Some of the gaps are positioned to uniquely indicate an orientation of the visual marker. In one example, each marking is a ring formed from ring segment sub-markings that are spaced to define a template of locations. Some of the locations in the template are selectively populated with other ring segment sub-markings (representing 1) or left as gaps (representing 0) to convey information. Other of the locations in the template are left as gaps to indicate the orientation of the visual marker. For example, the gaps at the template locations that indicate orientation can provide a combination of gap locations that is unique to a single orientation of the visual marker. The size, shape, number of locations, and other characteristics of the markings can be configured such that the gaps at certain locations provide a combination of gap locations that is unique to a single orientation of the visual marker. Various other implementations disclosed herein decode or otherwise interpret the visual marker to determine an orientation of the visual marker or obtain information conveyed by the visual marker based on the orientation.

[0004] In some implementations, the visual marker conveys a first set of information by selectively encoding (e.g., closing or opening) gaps between ring segment sub-markings that form a plurality of elements in each of a plurality of markings. In some implementations, the visual marker conveys a second set of information by selectively coloring a subset of the plurality of elements.

[0005] In some implementations, a visual marker that conveys information includes a plurality of markings arranged in a corresponding plurality of shapes, each marking formed from a set of sub-markings that are separated by gaps and arranged according to a respective shape, where the gaps of the plurality of markings are configured to convey information (e.g., encode data) and indicate an orientation of the visual marker.

[0006] In some implementations, at an electronic device with a processor, a method includes obtaining an image of a physical environment, the physical environment including a visual marker, the visual marker including a plurality of markers arranged in a corresponding plurality of shapes, each marker formed by a set of sub-markers separated by gaps and arranged according to a respective shape. An orientation of the visual marker is determined from a first set of gaps in at least two of the markers of the plurality of markers depicted in the image. Data encoded in a second set of gaps in the visual marker is then decoded based on the orientation of the visual marker.

[0007] In some implementations, at an electronic device with a processor, a method includes obtaining an image of a physical environment, the physical environment including a visual marker, the visual marker including a plurality of elements. Color characteristics of the visual marker are then determined based on the image. In some implementations, data values are determined for colors exhibited by the plurality of elements, the data values determined based on the determined color characteristics. Data encoded in the colors exhibited by the plurality of elements is then decoded based on the determined data values for the colors. BRIEF DESCRIPTION OF DRAWINGS

[0008] Accordingly, the disclosure can be understood in more detail with reference to some example implementations, some of which are illustrated in the appended drawings.

[0009] Figure 1 is an illustration of an example operating environment in accordance with some implementations.

[0010] Figure 2 is an illustration of an example electronic device in accordance with some implementations.

[0011] Figures 3-4 is an illustration of an example visual marker that illustrates indicating orientation and communicating information using gaps in a plurality of progressively larger markers in accordance with some implementations.

[0012] Figures 5A-5B is an illustration of an example configuration of two concentric rings of an example visual marker.

[0013] Figure 6 is an illustration of an example visual marker that illustrates indicating orientation and communicating information using gaps in a plurality of progressively larger markers in accordance with some implementations.

[0014] Figure 7 is an illustration of a visual marker that can be detected by an electronic device in a physical environment in accordance with some implementations.

[0015] Figure 8is a flow diagram illustrating an example method of decoding a visual marker that indicates orientation and conveys information, in accordance with some implementations.

[0016] Figure 9 is an example visual marker that conveys information using gaps in a plurality of progressively larger markers and conveys information in colored sub-markers in the plurality of progressively larger markers, in accordance with some implementations.

[0017] Figure 10 is a flow diagram illustrating an example method of decoding a visual marker that conveys information using colors in a plurality of elements that form a plurality of markers arranged in a corresponding plurality of shapes that increase in size, in accordance with some implementations.

[0018] Figure 11 is a flow diagram illustrating an example method of decoding a visual marker that indicates orientation and conveys information using gaps in a plurality of markers arranged in a corresponding plurality of shapes that increase in size, in accordance with some implementations.

[0019] Figure 12 is an illustration of another example visual marker that includes a plurality of markers arranged in a corresponding plurality of shapes that increase in size, in accordance with some implementations.

[0020] In accordance with common practice the various features can not be drawn to scale in the various figures. Therefore, the dimensions of the various features can be arbitrarily expanded or reduced for the clarity of presentation. In addition, some of the drawings can not depict all of the components of a given system, method or device. Finally, like reference numerals can be used to denote like features throughout the specification and figures. DETAILED DESCRIPTION

[0021] Many details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely illustrate some example aspects of the present disclosure and, therefore, should not be considered to be limiting in scope so as to limit the overall scope of possibilities. Those of ordinary skill in the art will appreciate that other effective aspects or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as to avoid obscuring aspects of the example implementations described herein.

[0022] Figure 1 An example operating environment 100 is shown that uses an electronic device 120 in a physical environment 105. A physical environment refers to the physical world that people are able to interact with and / or sense without aid of electronic systems. A physical environment such as a physical park includes physical articles such as physical trees, physical buildings, and physical people. People are able to directly sense and / or interact with a physical environment such as through sight, touch, hearing, taste, and smell.

[0023] In Figure 1 In the example of FIG. 1, device 120 is shown as a single device. Some implementations of device 120 are handheld. For example, device 120 can be a mobile phone, a tablet computer, a laptop computer, etc. In some implementations, device 120 is worn by a user. For example, device 120 can be a watch, a head-mounted device (HMD), a head-clip-on device (eyeglasses), etc. In some implementations, the functionality of device 120 is implemented via two or more devices, such as additionally including an optional base station. Other examples include a laptop computer, a desktop computer, a server, or other such device that includes additional capabilities in terms of power, CPU capability, GPU capability, storage capability, memory capability, etc. Multiple devices that can be used to implement the functionality of device 120 can communicate with each other via wired or wireless communication.

[0024] In some implementations, electronic device 120 is configured to capture, interpret, and use visual markers, for example, to present content to user 115. In some implementations, electronic device 120 captures one or more images of a physical environment, including one or more images of a visual marker. Electronic device 120 can identify the visual marker in the one or more images and use a corresponding portion of the one or more images to determine an orientation of the visual marker and interpret information conveyed by the visual marker based on the orientation.

[0025] Figure 2 is a block diagram of an example device 200. Device 200 illustrates an example device configuration of device 120. While some specific features are illustrated, one of skill in the art will from the present disclosure recognize that, for brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein, various other features are not illustrated. To that end, as non-limiting examples, in some implementations, electronic device 200 includes one or more processing units 202 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 206, one or more communication interfaces 208 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.1 lx, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, one or more displays 212, one or more inward- or outward-facing image sensor systems 214, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0026] In some implementations, the one or more communication buses 204 include circuitry that interconnects the system components and controls communications between the system components. In some implementations, the one or more I / O devices and sensors 206 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, or one or more depth sensors (e.g., structured light, time-of-flight, etc.), etc.

[0027] In some implementations, the one or more displays 212 are configured to present content to a user. In some implementations, the one or more displays 212 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS), or similar display types. In some implementations, the one or more displays 212 correspond to waveguide, diffractive, reflective, polarized, holographic, etc. displays. For example, the electronic device 200 can include a single display. As another example, the electronic device 200 includes a display for each eye of a user.

[0028] In some implementations, the one or more inward- or outward-facing sensor systems 214 include an image capture device or array that captures image data or an audio capture device or array (e.g., microphones) that captures audio data. The one or more image sensor systems 214 can include one or more RGB cameras (e.g., with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, or event-based cameras, etc. In various implementations, the one or more image sensor systems 214 also include an illumination source that emits light, such as a flash. In some implementations, the one or more image sensor systems 214 also include an on-camera image signal processor (ISP) configured to perform a number of processing operations on image data.

[0029] Memory 220 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some implementations, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 220 optionally includes one or more storage devices remotely located from the one or more processing units 202. Memory 220 comprises a non-transitory computer readable storage medium.

[0030] In some implementations, memory 220 or the non-transitory computer readable storage medium of memory 220 stores optional operating system 230 and one or more instruction sets 240. Operating system 230 includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, instruction set 240 includes executable software defined by binary information stored in charge form. In some implementations, instruction set 240 is software that is executable by the one or more processing units 202 to implement one or more of the techniques described herein.

[0031] In some implementations, instruction set 240 includes a visual marker reader 242 that is executable by processing unit 202 to identify a visual marker, determine an orientation of the visual marker, and interpret information conveyed by the visual marker based on the orientation. In some implementations, visual marker reader 242 is executed to detect and interpret visual markers present in one or more images of a physical environment captured, for example, by one or more interior- or exterior-facing sensor systems 214.

[0032] In some implementations, instruction set 240 includes a visual marker creator 244 that is executable by processing unit 202 to create a visual marker that indicates an orientation and conveys information in accordance with one or more of the techniques disclosed herein.

[0033] Although instruction set 240 is shown as residing on a single device, it will be appreciated that, in other implementations, any combination of the elements can be located in separate computing devices. Figure 2 More functionality is used as a functional description of the various features present in a particular implementation, as opposed to structural diagrams of implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately can be combined and some items can be separated. For example, the actual number of instructions and the division of particular functions between them, as well as how features are allocated among the instructions, will vary from one implementation to another and, in some implementations, depend in part on the particular combination of hardware, software, or firmware chosen to implement the

[0034] Figures 3-4 and Figure 6 This is an illustration of exemplary visual markers that use gaps in a plurality of progressively increasing marks to indicate orientation and convey data, according to some specific embodiments. In some embodiments, visual marker 300 is a template (e.g., an uncoded visual marker), and visual marker 400 is an instance of the visual marker 300 template (e.g., conveying information). In some embodiments, visual marker 600 is another instance of the visual marker 300 template (e.g., conveying information). In some embodiments, visual marker 300 includes a plurality (e.g., a series) of progressively increasing marks, wherein each of these marks has the same or different shapes. In some embodiments, visual marker 300 includes a plurality of progressively increasing marks, wherein at least one of these marks uses a different shape. In some embodiments, visual marker 300 includes a plurality of concentric marks. In some embodiments, visual marker 300 includes a plurality of progressively increasing marks (e.g., progressively increasing rings), wherein each of these marks has a different number of gaps for conveying information.

[0035] like Figure 3 As shown, the visual marker 300 includes a plurality of markers arranged in corresponding shapes (e.g., each marker is a ring of the same shape). In some embodiments, each marker of the plurality of rings 310A-310E is formed by sub-markers 330 arranged according to (e.g., along) a corresponding shape (e.g., one ring of rings 310A-310E) and positioned according to (e.g., along) a corresponding shape (e.g., one ring of rings 310A-310E). In some embodiments, each ring of the plurality of rings 310A-310E may include fixed sub-markers 330 that define a template (e.g., a grid) for conveying information in gaps 320 within the visual marker 300. In some embodiments, the template of gaps 320 is selectively filled to convey information or orientation. In some embodiments, the template of gaps 320 is selectively filled with sub-markers to convey at least one information bit. In some implementations, each of the plurality of rings 310A-310E may include a fixed sub-marker 330, which is defined in Figure 4 and Figure 6 Templates for the location of each visual marker of the visual marker type shown (e.g., before being decoded, unencoded, or without data).

[0036] In some embodiments, the template of gap 320 or position is selectively filled with sub-marks (e.g., representing a "1" bit) or left as a gap (e.g., representing a "0" bit) to convey information in instances of visual marker 300 templates (e.g., encoded visual marker 400). In some embodiments, the size of gap 320 in each ring of rings 310A-310E is the same. In some embodiments, the size of gap 320 in all rings of rings 310A-310E is the same. In some embodiments, a data sequence represented by multiple encoded adjacent gaps (e.g., encoded gaps 320 in visual marker 400) of encoded markers (e.g., of multiple rings 410A-410E) can indicate a data sequence, such as 0100101.

[0037] Figure 4 An exemplary visual marker 400 is illustrated, which, according to some specific embodiments, indicates orientation and conveys information through the spacing of a plurality of marks arranged in corresponding multiple shapes. In some embodiments, the visual marker 400 includes a plurality of progressively larger surrounding marks, wherein each of these marks is of the same shape. In some embodiments, the visual marker 400 includes a plurality of progressively larger marks, wherein at least one of these marks uses a different shape. In some embodiments, the plurality of marks of the visual marker 400 are equally spaced. In some embodiments, the plurality of marks of the visual marker 400 are not equally spaced.

[0038] like Figure 4 As shown, the visual marker 400 includes a plurality of concentric rings 410A-410E, each of which may have a different number of gaps for storing information. In some embodiments, each concentric ring increases in size from the innermost concentric ring 410A to the outermost concentric ring 410E. In some embodiments, each ring in rings 410A-410E, when encoded with information, includes a plurality of arcs 450 and gaps 440 located between two adjacent arcs 450. In some embodiments, the gaps 440 represent at least one binary digit (bit) of the information. In some embodiments, the gaps 440 represent "0", and the filled gaps (e.g., 320) forming the larger arcs in the arcs 450 represent at least one "1". In some embodiments, the gaps 440 in each ring in rings 410A-410E are of the same size. In some embodiments, the gaps 440 in all rings in rings 410A-410E are of the same size. In some implementations, the visual marker 400 uses gaps 440 and arcs 450 to encode 128 bits (e.g., including parity bits or error correction bits).

[0039] In some specific implementations, Figure 3The visual marker 300 template shown is considered to represent all 0s, while Figure 4 The visual marker 400 shown encodes some data (e.g., the template gaps 320 are selectively filled with sub-marks indicating the value of 1 for this data). In some implementations, the template gaps 320 are selectively filled with parametric graphical elements to encode more than 1 bit of data in the visual marker 400.

[0040] In some implementations, the visual marker 400 conveys information (e.g., indicating orientation and encoding data), and the visual marker 300 is able to use the gaps 320 to convey information (e.g., indicating orientation and encoding data payload). In some implementations, the visual marker 400 encodes metadata, variant metadata, or corresponding parity data in the template gaps 320 to form the gaps 440 and arcs 450. In some implementations, the visual marker 400 indicates the orientation of the visual marker 400 in the gaps 440 (e.g., gaps 320). In some implementations, the visual marker 400 (e.g., using the template gaps 320) encodes aesthetic data of the visual marker 400 in the gaps 440 and arcs 450.

[0041] In some implementations, the visual markers 400, 300 have a single detectable orientation. In some implementations, the visual markers 400, 300 use the gaps 320 to determine the single detectable orientation. In some implementations, the number of gaps 320 in each of the rings 310A-310E, 410A-410E is selected so that there is only one orientation in which all of the gaps 320 align in the visual marker 300, 400. In some implementations, the respective number of gaps 320 in each of the rings 310A-310E, 410A-410E is selected to have no common divisors, which ensures the single orientation of the visual marker 300, 400. As shown, Figures 3-4 As shown, the respective number of gaps 320 (e.g., filled and empty) in each of the rings 310A-310E, 410A-410E is 17, 23, 26, 29, 33 in the visual marker 300 template and the visual marker 400.

[0042] In some implementations, the orientation can be used to determine where to start reading, decoding, or otherwise interpreting information represented by the arcs 450 and gaps 440 present at the locations of the gaps 320 in the visual marker 400. For example, reading data in the oriented visual marker 400 can start at the 6 o’clock position and proceed counterclockwise in each of the rings 410A-410E to interpret information represented by the arcs 450 and gaps 440 present at the locations of the gaps 320, and the innermost ring 410A can be decoded as 11111011101010010.

[0043] Figures 5A-5B is a diagram showing an example of 2 concentric rings with different numbers of gaps. As Figure 5A shown, the inner ring has 2 gaps, and the outer ring has 4 gaps. In Figure 5A , the 2 concentric rings have an ambiguous orientation because the gaps in the 2 concentric rings are aligned twice (e.g., at 0° and at 180°). As Figure 5B shown, the inner ring has 2 gaps, and the outer ring has 3 gaps. In Figure 5B , the 2 concentric rings have an unambiguous orientation because the gaps in the 2 concentric rings are aligned in a single orientation. In Figure 5B , the number of gaps in the 2 concentric rings do not have a common divisor.

[0044] Figure 6 shows an example visual marker 600 that uses gaps in a plurality of markers arranged in a corresponding plurality of shapes to indicate orientation and convey information, in accordance with some implementations. As Figure 6 shown, in the visual marker 600 that conveys information (e.g., different information than the visual marker 400), the respective number of gaps 320 (e.g., filled and empty) in each of the rings 410A-410E is 17, 23, 26, 29, 33. As Figure 6 shown, at least 2 of the rings 410A-410E that each have 1 gap (e.g., 440) are used to unambiguously indicate the orientation of the visual marker 600. In some implementations, at least one gap (e.g., 440) in each of the rings 410A-410E is used to indicate the orientation of the visual marker 600. In some implementations, at least one of the rings 410A-410E includes more than one gap (e.g., 440) to determine the orientation of the visual marker 600.

[0045] In some implementations, the visual markers 300, 400, 600 include a first subset of gaps 320 that indicate the orientation of the visual markers 300, 400, 600. In some implementations, the visual markers 300, 400, 600 include a second subset of gaps 320 that convey information (e.g., encoded data) of the visual markers 400, 600.

[0046] In some embodiments, visual markers 300, 400, and 600 include a first subset of gaps 320, which includes at least one gap 320 in at least two rings of rings 310A-310E and 410A-410E. In some embodiments, the first subset of gaps 320, including at least one gap 320 in at least two rings of rings 310A-310E and 410A-410E, is filled (e.g., closed) unless the corresponding rings in rings 410A-410E become solid (e.g., completely filled without gaps) by encoding data (e.g., payload) of visual markers 400 and 600, and then at least one gap 320 in the first subset of gaps 440 is not filled (e.g., open).

[0047] In some embodiments, visual marker 400 includes a first color (e.g., foreground color) for arc 450 and a second color (e.g., background color) for gap 440. In some embodiments, the filled gap 320 is filled with the first color to form arc 450. In some embodiments, when the gap 440 corresponding to gap 320 is the background color, the gap 440 of visual marker 400 represents a "0" bit (e.g., empty). In some embodiments, when arc 450 is the foreground color and is larger in size than sub-marker 330, the gap 320 filled to form an arc in arc 450 represents a "1" bit (e.g., filled).

[0048] In some embodiments, the first color of multiple markers of the visual marker (e.g., rings 310A-310E, 410A-410E) and the second color of the background of the visual marker are selected anywhere within the color spectrum. In some embodiments, the first and second colors of the visual marker can be any colors, but typically the two colors are chosen based on detectability or aesthetics. In some embodiments, the detectability of the two colors is based on one or more of the following: interval in the 3D color space, lighting conditions, printing conditions, display conditions, image capture sensor, or aesthetic information. In some embodiments, the color of the visual marker 400 is not used for encoding data.

[0049] In some embodiments, visual marker 400 provides detectable orientation or conveys information without using excessively large features in the visual marker. In some embodiments, visual marker 400 provides detectable orientation or conveys information without using excessively small features in the visual marker. In some embodiments, visual marker 400 provides detectable orientation or conveys information without using coloring features in the visual marker.

[0050] Figure 7 This is an illustration showing visual markers that can be detected by electronic devices in a physical environment, according to some specific implementations. For example...Figure 7 As shown, visual marker 400 in physical environment 705 is detected by second electronic device 720. In some implementations, visual marker 400 is on a surface of object 710. In some implementations, object 710 is a first electronic device that includes a visual producing device such as a display or a projector.

[0051] As Figure 7 shown, visual marker 400 is a 2D / 3D object that encodes information in a preset format (e.g., binary format), such as a string or other payload for accessing remote-based experience 712. In some implementations, the link to remote-based experience 712 includes a link to initiate a payment (e.g., a licensed payment endpoint), a link to a website (e.g., a URL), or a link to launch a web-based experience. In some implementations, visual marker 400 is used to launch or link only to approved remote-based experiences 712 that are authorized by the creator of visual marker 400. In some implementations, the creator of a visual marker includes the entity that designed the visual marker, the entity that printed (e.g., fabricated) the visual marker (e.g., a developer), and the entity that manages / hosted the visual marker. In some implementations, visual marker 400 can not encode a URL.

[0052] As Figure 7 shown, in some implementations, an image of physical environment 705 is obtained using a sensor (e.g., camera 740) on electronic device 720. In some implementations, the sensor can be an RGB camera, a stereo camera, a depth sensor (e.g., time-of-flight, structured light), an RGB-D camera, a monochrome camera, one or more 2D cameras, an IR camera, a dynamic vision sensor (event camera), etc. In some implementations, a color image can be used. Alternatively, in some implementations, a grayscale image can be used. In some implementations, the captured image is a 2D image or a 3D image at electronic device 720. Figure 7 An electronic device is shown that can include some or all of the features of one or both of electronic devices 120, 200.

[0053] Figure 8 is a flowchart illustrating an exemplary method of decoding a visual marker that uses gaps in a plurality of markers arranged in a corresponding plurality of shapes that increase in size to indicate an orientation and convey information, in accordance with some implementations. In some implementations, the plurality of markers are arranged in a corresponding plurality of expanding concentric rings. In some implementations, a template gap in the plurality of markers is used to provide an orientation and convey data. In some implementations, a first set of gaps is used to provide an orientation and a second, different set of gaps in the plurality of markers is used to convey data. In some implementations, method 800 is performed by a device (e.g., Figures 1-2by the electronic device or by multiple devices in communication with one another. In some implementations, the method 800 is performed by processing logic that includes hardware, firmware, software, or a combination thereof. In some implementations, the method 800 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0054] At block 810, the method 800 obtains an image of a physical environment that includes a visual marker, the visual marker including a plurality of markers arranged in a corresponding plurality of shapes, each of the plurality of markers formed from a set of sub-markers arranged according to a respective shape and separated by a gap. In some implementations, the plurality of markers form a plurality of identical at least partially encircling circles, ellipses, rectangles, polygons, stars, or other shapes having different sizes (e.g., increasing, decreasing). In some implementations, the plurality of markers are concentric. In some implementations, a first marker corresponds to an inner ring, a second marker corresponds to a second ring encircling the first ring, a third marker corresponds to a third ring encircling the second ring, and so on. In some implementations, the gaps in the plurality of markers can have a uniform size. In some implementations, the visual marker has a unique detectable orientation.

[0055] In some implementations, at block 810, the visual marker is visible at a surface of an object in the physical environment. In some implementations, the visual marker is printed on the surface of the object. In some implementations, the visual marker is printed by a 2D or 3D printer. In some implementations, the visual marker is printed by a black-and-white printer or a color printer (e.g., RGB or CYMK). In some implementations, the visual marker is painted, etched, powdered, drawn, sprayed, etc. onto the surface of the object. In some implementations, the visual marker is displayed by a display or projected by a projector onto the object in the physical environment. In some implementations, the display or projector is self-emissive, emissive, transmissive, or reflective.

[0056] In some implementations, at block 810, an image sensor at the electronic device captures an image of the physical environment that includes the visual marker. In some implementations, the detecting electronic device (e.g., including the image sensor) detects the visual marker in the image of the physical environment. In some implementations, the visual marker is detected by finding a predetermined shape of a selected portion of the visual marker (e.g., one of the plurality of markers) in the image. In some implementations, the sensor can be an RGB camera, a depth sensor, an RGB-D camera, a monochrome camera, one or more 2D cameras, an event camera, an IR camera, etc. In some implementations, a combination of sensors is used. In some implementations, the sensor is used to generate an extended reality (XR) environment representing the physical environment. In some implementations, a color image can be used. Alternatively, in some implementations, a grayscale image can be used.

[0057] At block 820, the method 800 determines an orientation of the visual marker from a first set of gaps in the gaps in at least two of the markers of the plurality of markers depicted in the image. In some implementations, the orientation is determined using at least one of the first set of gaps in the gaps in two different markers of the plurality of markers depicted in the image. In some implementations, determining the orientation includes determining a unique orientation of the visual marker corresponding to the relative positioning of the first set of gaps. In some implementations, a respective number of template gaps in each of the plurality of markers is respectively selected to have no common divisor to provide a single detectable orientation of the visual marker. In some implementations, the image can be rectified to account for image capture conditions.

[0058] At block 830, the method 800 decodes data encoded in a second set of gaps in the gaps based on the orientation of the visual marker. In some implementations, the data is encoded in the second set of gaps in the template gaps in the plurality of markers. In some implementations, the second set of gaps and the first set of gaps are the same gaps in the plurality of markers. In some implementations, the data is encoded in a second set of gaps in the plurality of markers different from the first set of gaps.

[0059] In some implementations, at block 830, decoding includes clustering pixels of the plurality of markers into one of the corresponding plurality of markers. In some implementations, clustering classifies pixels of the plurality of markers into a plurality of classes each representing one of the corresponding plurality of markers and at least one other class (e.g., error, outlier, occlusion, etc.) using a data-driven learning segmentation method such as a semantic segmentation deep learning model. In some implementations, clustering classifies pixels of the plurality of markers into a plurality of classes each representing one of the corresponding plurality of shapes and at least one other class (e.g., error, outlier, occlusion, etc.) using k-means clustering and iterative matching.

[0060] In some implementations, at block 830, clustering includes randomly selecting a plurality of points from the plurality of labeled pixels as a set of points (e.g., after binarization or image segmentation), and hypothesizing a shape from the selected set of points as a modeling shape. For example, 5 randomly selected pixels from the plurality of labeled pixels form a set of points that are used to hypothesize a uniquely defined elliptical shape as a modeling shape. In another example, 3 randomly selected pixels from the plurality of labeled pixels form a set of points that are used to hypothesize a uniquely defined circular shape as a modeling shape. In some implementations, the random selection step and the hypothesis step are repeated for a prescribed number of iterations (e.g., 1000 times) or until at least one alternative stopping criterion (e.g., a number of detected inner bounding layers, a model fitting cost) is satisfied. In some implementations, a first shape of the corresponding plurality of shapes is determined from the hypothesized modeling shape that results in a largest set of points obtained during the iterations that are close to (e.g., up to a distance threshold) that shape. In some implementations, the best set of points (e.g., the set of points that results in the highest number of points that are close to the hypothesized modeling shape) is used to determine a shape of one concentric shape (e.g., one concentric ring or a first concentric ring of a plurality of concentric rings) of the plurality of concentric shapes of the visual marker. In one example, the remaining pixels of the plurality of labels are independently clustered into a corresponding set of each of the remaining concentric rings (e.g., as described above for the first concentric ring or the outermost concentric ring). Then, for each additional concentric ring, the best set of points that results in a largest set of points that are close to (e.g., up to a distance threshold) that shape is determined. In this example, the pixels clustered into each of the concentric rings can be removed from the analysis of the remaining concentric rings. Thus, in some implementations, the pixels of the plurality of labels are independently clustered into one of the corresponding plurality of labels. In some implementations, a pre-set relationship such as a size or a distance exists and is known between the plurality of concentric labels arranged in a corresponding plurality of shapes (e.g., a plurality of concentric rings) based on a shape of one of the labels (e.g., rings), and this information can be used to hallucinate the other labels (e.g., estimate the shape of the other or remaining concentric rings). In some implementations, once the remaining concentric labels are estimated, the best set of points (e.g., described above) is used to determine a shape of each of the remaining concentric labels. In some implementations, the clustered pixels in each of the plurality of labels are compared simultaneously to match gaps in a plurality of sets of template sub-labels of the plurality of labels to detect an orientation of the visual marker for decoding the visual marker.

[0061] In some implementations, at block 830, the method 800 further decodes the data of the visual marker from the starting position sequentially (e.g., by marker such as innermost marker to outermost marker and clockwise / counterclockwise ordering) among the plurality of markers based on the orientation of the visual marker. In some implementations, at block 830, the method 800 further decodes the data of the visual marker into binary data such as a string or other payload to initiate a payment, link to a website, link to a location-based or context-based experience, or launch to other web-based experience. In some implementations, the use of the visual marker can be arbitrary in terms of the user experience after decoding. For example, the visual marker can be displayed on a TV and when scanned, the decoded data can help the user select an option, obtain information about a movie being displayed on the TV, etc. In another example, the decoded data from the visual marker when scanned by a user can initiate an application on a scanning electronic device (e.g., a smartphone), such as a food delivery application. In some implementations, the visual marker can be displayed and when scanned, the decoded data delivers an audio message or music to the decoding electronic device.

[0062] In some implementations, the visual marker depicted in the image is binarized prior to orientation determination or data decoding. In some implementations, the pixels of the plurality of markers arranged in the corresponding plurality of shapes are changed to a first color (e.g., black) and the remaining pixels are changed to a second color (e.g., white).

[0063] In some implementations, the colors (e.g., two or more) of the visual marker can be any color, however, the colors are selected based on detectability or aesthetics. Thus, the first color for the plurality of markers of the visual marker and the second color for the background of the visual marker are selected anywhere within a color spectrum.

[0064] In some implementations, a version of the visual marker is encoded in a first portion of the plurality of markers (e.g., the first or innermost marker) and the orientation is indicated in a second portion of the plurality of markers (e.g., the remaining markers) with the encoded data of the visual marker. In some implementations, the version of the visual marker is encoded using a first encryption type and the second portion is encoded using a different second encryption type. In some implementations, the version encodes the number of the plurality of markers (e.g., 4, 5, 6 concentric rings, etc.) in the second portion or in the visual marker.

[0065] In some implementations, at block 810, the method 800 determines, based on the one or more images, a relative positioning between the detecting electronic device and the visual marker. In some implementations, the relative positioning determines a relative pose (position and orientation) of the visual marker with respect to the detecting electronic device. In some implementations, the relative positioning is determined using computer vision techniques (e.g., VIO or SLAM) or n-point perspective (PNP) techniques. In some implementations, the relative positioning is determined based on stereo image processing (e.g., based on different estimates). In some implementations, the relative positioning is determined based on deep learning (e.g., convolutional neural network, CNN). In some implementations, the relative positioning determines a distance or a direction from the detecting electronic device to the visual marker.

[0066] In some implementations, the relative positioning is determined at the detecting electronic device by identifying a size or scale of the detected visual marker in the captured image. In some implementations, a distance between the detecting electronic device and the detected visual marker can be determined based on the size of the visual marker. In some implementations, the size or shape of the visual marker can be encoded in the visual marker and then decoded directly from the image of the physical environment. In some implementations, the size or shape of the visual marker is preset and known by the detecting electronic device. In some implementations, the size or shape of the visual marker is determined at the detecting electronic device using VIO, SLAM, RGB-D image processing, etc.

[0067] Alternatively, a distance between the detecting electronic device and the detected visual marker can be determined based on a depth sensor at the detecting electronic device that detects the visual marker in the physical environment. In some implementations, the depth sensor at the detecting electronic device uses stereo-based depth estimation. In some implementations, the depth sensor at the detecting electronic device is a depth-specific sensor (e.g., time-of-flight, structured light).

[0068] Figure 9 An example visual marker 900 is shown that uses gaps in a plurality of markers arranged in a corresponding plurality of shapes that increase in size to indicate an orientation and convey information in colored sub-markers in the plurality of markers, in accordance with some implementations. In some implementations, the visual marker 900 includes a plurality (e.g., a series) of progressively larger encircling markers, where each of the markers is the same shape. In some implementations, the visual marker 900 includes a plurality of progressively larger markers, where at least one of the markers uses a different shape.

[0069] In some embodiments, two different techniques are used to convey information in multiple marks (e.g., rings 910A-910E) of visual marker 900. In some embodiments, a first technique (e.g., template gaps 920 between closed or unclosed template sub-marks 930) is used to convey information in multiple marks (e.g., rings 910A-910E) to form arcs 950 with gaps 940 therebetween, and a second technique (e.g., color decoding of a predetermined number of arcs 950) is used to convey information in visual marker 900 using the arcs 950.

[0070] In some specific implementations, a first technique is used to convey information in the visual marker 900, and then a second technique is used in the visual marker 900 to convey information.

[0071] like Figure 9 As shown, the visual marker 900 includes a plurality of concentric rings 910A-910E, wherein each of the concentric shapes has a different number of template gaps 920 for indicating orientation and conveying information. In some embodiments, each of the plurality of markers (e.g., rings 910A-910E) is formed by a set of template sub-markers 930 arranged according to a corresponding shape and separated by template gaps 920. In some embodiments, the visual marker 900 is another instance of the visual marker 300 template (e.g., for conveying information).

[0072] In some embodiments, each ring in rings 910A-910E includes multiple arcs 950 and gaps 940 between them when encoding information (e.g., using template gaps 920). In some embodiments, each template gap 920 represents at least one binary digit (bit) of information. In some embodiments of visual marker 900, empty template gaps 920 represent "0" and form gaps 940, and each filled template gap 920 represents "1" and forms a larger arc 950. In some embodiments, the size of the template gaps 920 in each ring in rings 910A-910E is the same. In some embodiments, the size of the template gaps 920 in all rings in rings 910A-910E is the same. In some embodiments, the visual marker 900 uses template gaps 920 between template sub-markers 930 to encode 128 bits (e.g., including parity).

[0073] In some implementations, the visual marker 900 has a single detectable orientation. In some implementations, the visual marker 900 uses the template gaps 920 to indicate the single detectable orientation. In some implementations, the number of template gaps 920 in each of the rings 910A-910E is selected so that there is only one orientation in which all of the template gaps 920 are aligned in the visual marker 900. In some implementations, the respective number of template gaps 920 in each of the rings 910A-910E (e.g., 17, 23, 26, 29, 33) is selected to not have a common divisor, which ensures the single orientation of the visual marker 900.

[0074] In some implementations, the orientation can be used to determine where to start decoding or otherwise interpreting information conveyed (e.g., encoded) by the template gaps 920 present in the locations between the template sub-markers 930 in the visual marker 900. For example, the data in the oriented visual marker 900 to decode can start at the 12 o’clock position and proceed counterclockwise from the innermost ring 910A to the outermost ring 910E to interpret the information represented using the template gaps 920.

[0075] In some implementations, a first plurality of arcs 950 (e.g., a subset) in the rings 910A-910E is color coded to further convey information. In some implementations, the first plurality of arcs 950 is a preset number (e.g., 56) of arcs 950 that are color coded to further convey information using a second technique. In some implementations, the color coding in the second technique uses a minimum number of arcs 950.

[0076] In some implementations, when an instance of the visual marker 900 conveys information in the template gaps 920, a corresponding number of arcs 950 is formed in the rings 910A-910E, and each arc in the first plurality of arcs 950 conveys additional information using the first color or the second color. As Figure 9 illustrated, the arcs 950 of the visual marker 900 are either the first color 951 (e.g., gray) or the second color 952 (e.g., black). In some implementations, the arcs 950 with the first color 951 represent “0,” and the arcs 950 with the second color 952 represent “1.” In some implementations, the first plurality of arcs 950 is decoded in order. For example, as Figure 9 illustrated, the first plurality of arcs 950 (e.g., 56 of 68 arcs 950) is decoded from 12 o’clock on the innermost ring 910A to 5 arcs 950 in the outermost ring 910E, and the innermost ring 910A can be decoded as 1101111001. In some implementations, the length of the arcs 950 does not affect the information conveyed by the visual marker 900 using color.

[0077] In some implementations, the arcs 950 use two colors to encode one bit in each arc of the first plurality of arcs 950. In some implementations, the visual marker 900 uses 4 colors for the arcs 950 such that each arc of the arcs 950 that conveys information conveys 2 bits of information (e.g., 11, 10, 01, 00). In some implementations, more than 4 colors can be used to convey information in the visual marker 900 using the second technique.

[0078] In some implementations, a preset number of the first plurality of arcs 950 is achieved in the visual marker 900 using an indicator or “flip arc” that swaps the arcs 950 and the gaps 940 when the number of arcs 950 is below a threshold. In one example, the threshold number (e.g., minimum) of the first plurality of arcs 950 can be 56, and when the encoded visual marker 900 results in 30 arcs 950, the “flip arc” is enabled and the information conveyed using the template gaps 920 between the template sub-markers 930 (e.g., first technique) is swapped such that the preset number of the first plurality of arcs 950 is available for use in the visual marker 900 with the second technique. In this example, a first encoding of the template gaps 920 uses a “closed” to encode a “1” bit and a “open” to encode a “0” bit in each respective template gap 920, which results in 30 arcs 950. Thus, the “flip arc” is enabled and the data encoded in the template gaps is “flipped” such that a second encoding of the template gaps 920 uses a “closed” to encode a “0” bit and a “open” to encode a “1” bit in each respective template gap 920, which results in 98 arcs 950 (e.g., which exceeds the minimum or preset number of 56 of the first plurality of arcs 950).

[0079] In some implementations, data values (e.g., bits) need to be assigned to each color (e.g., of the arcs 950) to convey information in the visual marker 900 using the second technique. In some implementations, a first arc of the first plurality of arcs 950 that encodes information using colors indicates which of the 2 colors in the visual marker 900 is assigned a data value of “1” and the second color becomes a data value of “0”. In some implementations, any one of the arcs 950 can be used to indicate the color that is assigned a data value of “1”. In some implementations, a preset sequence of the arcs 950 is used to assign data values to the multiple colors used in the arcs 950. In some implementations, the first 8 arcs of the first plurality of arcs 950 indicate data values assigned to 8 colors used in a visual marker such as the visual marker 900 (e.g., 111, 110, 101, 100, 011, 010, 001, 000), respectively.

[0080] In some implementations, the characteristics of the first color 951 and the second color 952 (e.g., multiple colors used in the second technique) are used to assign data values (e.g., highest data value to lowest data value) to the 2 colors in the visual marker 900. For example, the luminance characteristics of the 2 colors can be used to assign data values. As shown, the luminance value of the first color 951 is greater than the luminance value of the second color 952. In some implementations of the first color 951 and the second color 952, a data bit of “0” is assigned to the smallest luminance value or a data bit of “1” is assigned to the largest luminance value. In some implementations, the opacity characteristics of the colors used in the visual marker 900 are used to assign data values. Figure 9

[0081] In some implementations, the relationship between the first color 951 and the second color 952 (e.g., multiple colors used in the second technique) is used to assign data values (e.g., highest data value to lowest data value) to the 2 colors in the visual marker 900. In some implementations, a background color is provided for the visual marker 900. As shown, the background color is a third color 953 (e.g., white). In some implementations, based on the relationship to the background color, data values are assigned to the colors used to convey information in the arc 950 (or first plurality of arcs 950) using the second technique. For example, using a luminance relationship, the first color 951 is closer in luminance to the third color 953 and a data value of “0” is assigned to the first color 951 accordingly (and a data bit of “1” is assigned to the second color 952). In some implementations, other relationships between the colors used in the first plurality of arcs 950 in the visual marker and the background color are used to assign data values to the colors. Figure 9

[0082] Figure 10 is a flow diagram illustrating an example method of decoding a visual marker that conveys information using colors in a plurality of elements that form a corresponding plurality of shapes arranged in increasing size, in accordance with some implementations. In some implementations, corresponding data values assigned to each of the colors used to convey information are decoded using relationships between the colors in the visual marker. In some implementations, the plurality of shapes are arranged in a corresponding plurality of expanding concentric rings. In some implementations, the method 1000 is performed by an electronic device (e.g., the electronic device 120, 200) of FIGS. 1-2. The method 1000 can be performed using an electronic device or by a plurality of devices in communication with one another. In some implementations, the method 800 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1000 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Figures 1-2

[0083] ​​​At block 1010, the method 1000 obtains an image of a physical environment that includes a visual marker, the visual marker including a plurality of elements. In some implementations, the plurality of elements are sequentially arranged in the visual marker. In some implementations, the plurality of elements can be segments or sub-markers that form a plurality of progressively larger markers having respective shapes. In some implementations, the plurality of markers form a plurality of concentric identical symmetric shapes of increasing size (e.g., see block 810). In some implementations, the plurality of elements are variable size arcs in a plurality of concentric identical symmetric rings that form a plurality of markers.

[0084] In some implementations, at block 1010, the visual marker is visible at a surface of an object in the physical environment. In some implementations, at block 1010, an image sensor at the electronic device captures the image of the physical environment that includes the visual marker (e.g., see block 810).

[0085] At block 1020, the method 1000 determines color properties of the visual marker based on the image. In some implementations, the color properties are determined such as, but not limited to, brightness, opacity, etc. of colors in the visual marker (e.g., optionally, background color). In some implementations, the color properties determine a particular color is at a particular location on the visual marker. In some implementations, the color properties are determined using a color is in a particular element (e.g., sequential or ordinal position) of the plurality of elements of the visual marker.

[0086] At block 1030, the method 1000 determines data values for colors exhibited by the plurality of elements, the data values determined based on the determined color properties. In some implementations, a data value for color red is assigned a“0” based on the color red being the brighter of two colors in the visual marker (e.g., brightness, opacity, etc.). In some implementations, a data value for color red is assigned a“0” based on color properties (e.g., brightness, opacity, etc.) of the color red being closer to a background color in the visual marker. In some implementations, a data value for color red is assigned a“1” based on the color red being in a first element of a sequence of the plurality of elements on the visual marker. In some implementations, a data value for color red is assigned a“1” based on the color red being in a particular element of the plurality of elements on the visual marker. In some implementations, data values for a group of 4 colors present in the plurality of elements of the visual marker can be assigned data values 11, 10, 01, 00, respectively, based on the determined color properties.

[0087] At block 1040, the method 1000 decodes data encoded in the colors exhibited by the plurality of elements based on the determined data values of the colors. In some implementations, a sequence of colored elements can be decoded into a sequence of data based on the determined data values of the colors. For example, in a visual marker using two colors, red and blue, in a plurality of elements, a sequence of a red element, a red element, a blue element can be decoded into a sequence of bits 0, 0, 1. In some implementations, clustering such as semantic segmentation can be used to classify the plurality of markers into one of the two color categories of encoded information.

[0088] In some implementations, at block 1040, the method 1000 determines an orientation of the visual marker prior to decoding the plurality of elements (see block 830). In some implementations, at block 1040, the method 1000 further decodes data (e.g., encoded color data) of the visual marker sequentially (e.g., a predetermined order such as by innermost marker to outermost marker and clockwise / counterclockwise) from a starting position of the visual marker in the plurality of elements based on the orientation. In some implementations, at block 1040, the method 1000 decodes the data of the visual marker into binary data such as a string or other payload to initiate a payment, link to a website, link to a location-based or context-based experience, or launch to other web-based experiences. In some implementations, the use of the visual marker can be arbitrary in terms of the user experience after decoding.

[0089] Figure 11 FIG. 11 is a flow diagram illustrating an example method of decoding a visual marker that indicates an orientation and conveys information by modifying gaps in a plurality of markers arranged in a corresponding plurality of shapes that increase in size, in accordance with some implementations. In some implementations, the method 1100 also decodes colors in a plurality of elements that form the plurality of markers arranged in the corresponding plurality of shapes that increase in size. In some implementations, a corresponding data value assigned to each element in the plurality of elements is decoded using a relationship between colors in the visual marker. In some implementations, the method 1100 is performed by an electronic device (e.g., Figures 1-2 The electronic device 120, 200). The method 1000 can be performed using an electronic device or by a plurality of devices in communication with each other. In some implementations, the method 1100 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 1100 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0090] At block 1102, the method 1100 detects a visual marker including a plurality of markers arranged in a corresponding plurality of shapes in an image of a physical environment. In some implementations, each marker of the plurality of markers is formed from a set of template sub-markers arranged according to a respective shape and separated by a template gap. In some implementations, the plurality of markers form a plurality of the same symmetrical at least partially encircling circles, ellipses, rectangles, polygons, or other shapes having different sizes. In some implementations, the plurality of markers are concentric. In some implementations, a first marker corresponds to an inner ring, a second marker corresponds to a second ring encircling the first ring, a third marker corresponds to a third ring encircling the second ring, and so on. In some implementations, the gaps in the plurality of markers can have a uniform size. In some implementations, the visual marker has a unique detectable orientation.

[0091] In some implementations, at block 1102, an image sensor at the electronic device captures an image of the physical environment including the visual marker. In some implementations, the detecting electronic device (e.g., image sensor) detects the visual marker in the image of the physical environment (e.g., see block 810). In some implementations, the visual marker is visible at a surface of an object in the physical environment.

[0092] At block 1104, the method 1100 performs image correction on the detected visual marker in the image of the physical environment. In some implementations, at block 1104, the image can be corrected to account for image capture conditions. In some implementations, the image correction for the visual marker in the image of the physical environment includes color correction, such as local white balancing of colors in the visual marker. In some implementations, the image correction for the visual marker in the image includes correcting for occlusions or spatially varying lighting at the detected visual marker.

[0093] At block 1106, the method 1100 classifies pixels of each marker. In some implementations, classifying pixels of each marker includes segmenting the pixels into a plurality of classes each representing one of the plurality of markers. In some implementations, classifying pixels of each marker includes clustering the pixels of the plurality of markers into one of the corresponding shapes of the plurality of markers. In some implementations, clustering uses a semantic segmentation machine learning model to classify the pixels of the plurality of markers into a plurality of classes each representing one of the corresponding plurality of shapes and at least one other class (e.g., error, outlier, occlusion, etc.). In some implementations, clustering such as semantic segmentation can be used to classify the plurality of markers into one of two color classes of encoded information.

[0094] At block 1108, the method 1100 finds an in-plane orientation of the visual marker from the inter-template gaps in the set of template sub-markers of each of the plurality of markers depicted in the image. In some implementations, the orientation of the visual marker is determined from a first set of the inter-template gaps in the markers of the plurality of markers depicted in the image (e.g., see block 820). In some implementations, determining the orientation includes determining a unique orientation of the visual marker corresponding to the relative positioning of the first set of gaps of the plurality of markers.

[0095] At block 1110, the method 1100 decodes data encoded in at least one of the inter-template gaps in the plurality of markers based on the orientation of the visual marker. In some implementations, the data is encoded using a second set of the inter-template gaps in the plurality of markers. In some implementations, the second set of gaps and the first set of gaps are the same gaps in the plurality of markers. In some implementations, the data is encoded in a second set of gaps in the plurality of markers different from the first set of gaps. In some implementations, at block 1110, the method 1100 further sequentially decodes the data of the visual marker from a starting position of the visual marker in the plurality of markers (e.g., based on version or visual marker type or based on a pre-set order of orientation) (e.g., see block 830). Figure 8 )。

[0096] At block 1112, the method 1100 performs error correction on the data decoded from at least one of the gaps (e.g., 320) of the visual marker. In some implementations, the error correction is based on a plurality of parity bits encoded in the inter-template gaps of the visual marker. In some implementations, the error correction uses a known Reed-Solomon error correction technique.

[0097] At block 1114, the method 1100 classifies the colors of the visual marker. In some implementations, the colors (e.g., arcs 950) are classified based on at least one color characteristic determined for the colors exhibited by the plurality of elements (e.g., segments) forming the plurality of markers. In some implementations, a data value for a color in the plurality of elements is determined based on the determined color characteristic. For example, a data value for the color red is assigned a “0” based on the color red being the lighter of the two colors in the visual marker (e.g., luminance, opacity, etc.). In some implementations, a data value for the color red is assigned a “0” based on the color characteristic (e.g., luminance, opacity, etc.) of the color red being closer to the background color in the visual marker.

[0098] At block 1116, the method 1100 extracts color-coded data in the colors exhibited by the plurality of elements based on the determined data values for the colors. In some implementations, a sequence of colored elements (e.g., the first plurality of arcs 950) can be decoded into a sequence of data based on the determined data values for the colors. For example, in a visual marker that uses both red and blue colors in the plurality of elements, a sequence of a red element, a red element, a blue element can be decoded into a sequence of bits 0, 0, 1. In some implementations, the colors of the plurality of elements encode more than 1 data bit.

[0099] In some implementations, at block 1116, the method 1100 further decodes data encoded in the colors exhibited by the plurality of elements based on the determined data values for the colors. In some implementations, a sequence of colored elements can be decoded into a sequence of data from a starting position of the visual marker (e.g., based on a version or visual marker type or a preset order based on orientation) (see Figure 10 ).

[0100] At block 1118, the method 1100 performs error correction on the color-coded data extracted from the colors exhibited by the plurality of elements of the visual marker. In some implementations, the error correction is based on a plurality of parity bits encoded in the plurality of elements of the visual marker. In some implementations, the error correction uses a known Reed-Solomon error correction technique.

[0101] In some implementations, at block 1120, the method 1100 further decodes the data of the visual marker into binary data such as a string or other payload to initiate a payment, link to a website, link to a location-based or context-based experience, or launch to other web-based experiences. In some implementations, the use of the visual marker can be arbitrary in terms of the user experience after decoding.

[0102] In some implementations, at block 1120, the method 1100 only decodes data encoded in the template gap (e.g., skips blocks 1108-1112). In some implementations, at block 1120, the method 1100 only decodes data encoded in the colors (e.g., skips blocks 1114-1118). In some implementations, portions of the method 1100 are performed in different sequential order or simultaneously. For example, block 1114 can be performed after block 1108, as shown by the dashed arrow. For another example, block 1108 can be performed after block 1110, as shown by the dashed arrow.

[0103] Figure 12This is an illustration of another exemplary visual marker, according to some embodiments, comprising a plurality of marks arranged in corresponding shapes in ascending order of size. In some embodiments, the visual marker 1200 includes a plurality of concentric rings 1210A-1210E, each of which includes a different number of template gaps for conveying information. In some embodiments, the visual marker 1200 includes additional features that can be used in conjunction with, complement, or replace the features or capabilities of the visual marker as described herein according to some embodiments.

[0104] like Figure 12 As shown, the visual marker 1200 includes a first portion 1205 for detection, a second portion 1250 for identifying a set of different colors (e.g., 2, 3, 4, 8, etc.) used in the visual marker 1200, and a third portion 1210 (e.g., rings 1210A-1210E) for encoding data in the visual marker 1200. In some specific embodiments, the first portion 1205 includes a preset (asymmetric) shape for detection. Figure 12 As shown, the first portion 1205 is an outer ring forming an asymmetric boundary (e.g., asymmetric fill, asymmetric shading, gradient, etc.). In some embodiments, the first portion 1205 is an inner region with a predefined asymmetric shape. In some embodiments, the first portion 1205 is an asymmetric shape or logo located at the center of the visual marker 1200 (e.g., central region 1270). In some embodiments, the first portion 1205 is mapped to a color matching a preset value of binary information (e.g., always mapped to the bit value "0").

[0105] In some embodiments, the predefined shape of the first portion 1205 enables the detection, correction, or determination of the orientation of the visual marker 1200 (e.g., captured in an image). In some embodiments, the color of the first portion 1205 is variable (e.g., different for different visual markers), and therefore, the detection of the visual marker 1200 using the first portion 1205 is shape-based and does not use color. In some embodiments, computer vision techniques can be used to perform the detection of the visual marker 1200 in the image. In some embodiments, the visual marker 1200 is corrected based on the image. In some embodiments, the correction distorts the visual marker from the image so that the visual marker appears flat when viewed from a directly overhead orientation.

[0106] like Figure 12 As shown, the second part 1250 is different from and separate from the first part 1205, but includes elements that are part of the third part 1210.

[0107] like Figure 12As shown, the second portion 1250 includes 6 locations where a set of 3 colors (Color 1, Color 2, Color 3) of the visual marker 1200 are repeated in order. In some implementations, the second portion 1250 includes a number of pixels (e.g., 3x3, 4x4, 12x12, etc.) of sufficient size to detect and identify the set of colors in the second portion 1250. In some implementations, the set of 3 colors (Color 1, Color 2, Color 3) of the second portion 1250 are used to encode 2 bits of data at each gap 320 of the visual marker 1200. In some implementations, in the gaps 320 in at least rings 1210B-1210E, Color 1 represents “11”, Color 2 represents “10”, Color 3 represents “01”, and the background color represents “00”.

[0108] In some implementations, the third portion 1210 encodes data of the visual marker 1200 using graphical segments to fill the gaps 320. In some implementations, graphical segments parameterized by size, shape, color, orientation, etc. of the graphical elements are used to encode the gaps 320 of the visual marker 1200. The data (e.g., data portion) of the visual marker 1200 is then decoded based on the graphical segments and the set of colors (e.g., 1250). In some implementations, the second portion 1250 uses a different prescribed shape than the graphical segments used for the third portion 1210 of the visual marker 1200. In some implementations, the second portion 1250 uses known locations based on a particular overall predefined shape of the first portion 1205 or based on a particular overall shape of the visual marker 1200.

[0109] In some implementations, the set of colors (e.g., Color 1-3) of the visual marker 1200 are not predefined (e.g., the set of colors for a given visual marker encoding a first data item can be different than the set of colors for another visual marker encoding a second data item). In various implementations, the colors of the visual marker 1200 can be selected in any manner when the visual marker is designed, created, or modified.

[0110] In some implementations, the set of colors (e.g., colors in the second portion 1250) can be determined based on detectability. In some implementations, the detectability of the data encoding the colors is based on one or more of spacing in a 3D color space, lighting conditions, printing conditions, display conditions, image capture sensors, or aesthetic information.

[0111] In some embodiments, detection region 1260 is used to detect (e.g., in an image) visual markers 1200. In some embodiments, detection region 1260 is a single color (e.g., gray, white). In some embodiments, detection region 1260 uses one or more colors not used elsewhere in visual markers 1200. In some embodiments, detection region 1260 is an outer region having a predefined shape or predefined size (e.g., thickness to diameter) ratio. In some embodiments, detection region 1260 is a white ring at least 2 pixels wide as seen by an image sensor on an electronic device. In some embodiments, the detection of visual markers 1200 in an image (e.g., of a physical environment) can be performed using machine learning (ML) to detect detection region 1260. In some embodiments, first portion 1205 includes or surrounds detection region 1260. In some implementations, the color of the detection area 1260 is consistent (e.g., the same for different visual markers), and therefore, the detection of the visual marker 1200 is based on shape and color.

[0112] like Figure 12 As shown, in some embodiments, the visual marker 1200 includes a central region 1270. In some embodiments, the central region 1270 is used for decoration (e.g., a company logo). In some embodiments, the central region 1270 includes symbols for a specific shape or color for detection, a specific color for color correction (white balance), or a specific shape setting, size setting, or angle setting for the orientation or correction of the visual marker 1200A (e.g., in an image captured in a physical environment).

[0113] In some embodiments, additional portions of the visual marker 1200 may be colored using a single color (e.g., white or gray). In some embodiments, additional portions of the visual marker 1200 are used to perform local white balance of the colors in the visual marker 1200 when detected by an image sensor. In some embodiments, additional portions of the visual marker 1200 are used to detect spatial variations in illumination at the detected visual marker or to correct any detected spatial variations in illumination. For example, when a shadow is detected in an area outside the central region 1270 and the third portion 1210 (e.g., across a portion of the visual marker 1200), the detected shadow in the additional region may be used to correct color variations in the visual marker 1200 (e.g., the first portion 1205, the third portion 1210) caused by the shadow. In some embodiments, the spatial variations in illumination at the detected visual marker are caused by a light source, uneven illumination, objects in the physical environment, etc. In some embodiments, the additional portion is the detection area 1260 or the central region 1270.

[0114] like​ As shown, the visual marker 1200 is generally circular in shape. However, implementations of the visual marker 1200 are not intended to be so limited. In some implementations, other shapes of the visual marker 1200 can be used. In some implementations, the visual marker 1200 is an asymmetric shape, a symmetric shape, an oval, a rectangle, a triangle, a bowtie, etc. In some implementations, the first portion 1205, the second portion 1250, and the third portion 1210 are spatially separated differently in the visual marker 1200.

[0115] In some implementations, a version of the visual marker 1200 can be used to determine the version of the visual marker 1200. In some implementations, the version of the visual marker 1200 changes the number of the set of colors (e.g., the second portion 1250), changes the amount of data (e.g., the number of rings or the number of gaps in the rings in the third portion 1210), the size of the marker, the type of shape, or changes the graphical section used to encode data (e.g., the third portion 1210). In some implementations, the version of the visual marker 1200 is encoded in the inner ring (e.g., 1210A) or another portion (e.g., the center region 1270) of the visual marker.

[0116] In some implementations, detecting the visual marker is a computer vision analysis that classifies an image as containing or not containing the visual marker. In some implementations, the computer vision analysis performs shape detection on the first portion 1205. In some implementations, the computer vision analysis can be performed using ML. ML methods for object detection include machine learning-based methods or deep learning-based methods. In some implementations, machine learning methods first define features from a set of data containing both input and desired output, then use classification techniques to identify objects. In some implementations, deep learning techniques perform end-to-end object detection, for example, using CNNs, without explicitly defining features.

[0117] Various implementations disclosed herein include devices, systems, and methods of providing a visual marker that includes various features described herein (e.g., individually or in combination).

[0118] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatuses, or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

[0119] Unless specifically stated otherwise, it should be appreciated that throughout this specification discussions utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying" or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

[0120] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. A suitable computing device includes a computer system based on a multiprocessor microprocessor that accesses stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more implementations of the subject innovation. The teachings contained herein can be implemented in software, in firmware, or in both software and firmware, either in whole or in part, using any suitable programming, scripting, or other type of language or languages, or combinations thereof.

[0121] Implementations of the methods disclosed herein can be implemented in operations of such computing devices. The order of the blocks presented in the examples above can be varied; for example, blocks can be reordered, combined, or separated into sub-blocks. Certain blocks or processes can be performed in parallel.

[0122] The use of "adapted to" or "configured to," herein means open and inclusive language that is not limited to devices or processes adapted or configured to perform additional tasks or steps. Additionally, the use of "based on" means open and inclusive, as processes, steps, calculations, or other actions that are "based on" one or more recited conditions or values are in practice contingent upon the recited condition(s) or value(s), but can also be based on additional conditions or values that are beyond those that are recited. The headings, lists, and numbering herein are included for ease of explanation only and are not intended to be limiting.

[0123] It will also be appreciated that, although terms such as "first," "second," etc. can be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, a first node could be termed a second node, and, similarly, a second node could be termed a first node, which changes the meanings of the descriptions, only if all occurrences of "first node" are renamed consistently and all occurrences of "second node" are renamed consistently. The first and second nodes are both nodes, but they are not the same node.

[0124] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments of the application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, objects, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0125] As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to detecting" that a stated condition precedent has been met, depending on the context. Similarly, the phrase "if it is determined [that a stated condition precedent has been met]" or "if [a stated condition precedent has been met]" or "when [a stated condition precedent has been met]" can be construed to mean "in response to determining" or "in response to detecting" that the stated condition precedent has been met, depending on the context.

[0126] The foregoing detailed description of the application has been stated herein for the purposes of illustrative and example only, and is not intended to limit the scope of the application in any way. The scope of the application disclosed herein is broader than any particular specific embodiment and includes modifications and variations thereof that are obvious to one of skill in the art. It is intended that the specification and examples be considered as exemplary only, with the true scope of the application being indicated by the appended claims.

Claims

1. A visual marker to convey information, the visual marker comprising: a plurality of markers arranged in a corresponding plurality of annular shapes, wherein each marker is a ring of ring segment sub-markers separated by gaps, the ring segment sub-markers spaced to define positions; wherein a first set of positions have gaps and markers representing values, and a second set of positions provide gaps indicative of an orientation of the visual marker, wherein the gaps at positions in the second set of positions provide a combination of gap positions that is unique to a single orientation of the visual marker.

2. The visual marker of claim 1, wherein at least one gap of at least two of the plurality of markers identifies a detectable orientation of the visual marker.

3. The visual marker of claim 1, wherein a first set of the gaps encode data, and a second, different set of the gaps indicate the orientation.

4. The visual marker of claim 3, wherein at least one gap of the first set of gaps encodes one data bit.

5. The visual marker of claim 1, wherein the visual marker is printed, displayed, or projected on an object.

6. The visual marker of claim 1, wherein the plurality of annular shapes comprises a plurality of concentric symmetric rings of different sizes.

7. The visual marker of claim 1, wherein the distance is the same for all gaps in the plurality of markers.

8. The visual marker of claim 1, wherein the distance between adjacent shapes in the plurality of annular shapes is the same.

9. The visual marker of claim 1, wherein the number of gaps in each of the markers in the plurality of markers has no common divisor.

10. The visual marker of claim 1, further comprising: a plurality of elements forming the plurality of markers, the plurality of elements configured to encode at least 1 data bit based on color properties of colors used in the plurality of elements.

11. The visual marker of claim 1, wherein a property of a plurality of positions in the first set of positions represents at least one information bit.

12. The visual marker of claim 11, wherein the property of the plurality of positions in the first set of positions comprises filled or unfilled.

13. A method for encoding a visual marker, comprising: at an electronic device having a processor: obtaining an image of a physical environment, the physical environment including the visual marker, the visual marker comprising a plurality of markers arranged in a corresponding plurality of annular shapes, wherein each marker is a ring of ring segment sub-markers separated by gaps, the ring segment sub-markers spaced to define positions, wherein a first set of positions have gaps and markers representing values, and a second set of positions provide gaps indicative of an orientation of the visual marker, wherein the gaps at positions in the second set of positions provide a combination of gap positions that is unique to a single orientation of the visual marker; determining the orientation of the visual marker from the second set of positions as gaps in at least two of the markers of the plurality of markers depicted in the image; and decoding data encoded in the first set of positions based on the orientation of the visual marker.

14. The method of claim 13, wherein determining the orientation comprises: determining a unique orientation of relative positioning corresponding to the second set of positions.

15. The method of claim 13, wherein decoding comprises: clustering the pixels of the plurality of markers into one shape of the corresponding plurality of shapes.

16. The method of claim 15, wherein the clustering classifies the pixels of the plurality of markers into a plurality of classes each representing one shape of the corresponding plurality of shapes and at least one other class using a data-driven learning segmentation method.

17. The method of claim 15, wherein the plurality of markers are a plurality of concentric markers, wherein the clustering comprises: randomly selecting a set of pixels from the pixels of the plurality of markers; and using the set of pixels to hypothesize a modeling shape; repeating the steps of randomly selecting a set of pixels and using the set of pixels to hypothesize a modeling shape a prescribed number of times; selecting one of a plurality of sets of pixels based on a number of the pixels of the plurality of markers proximate to the corresponding modeling shape; determining a first shape of the corresponding plurality of shapes based on the selected one of the plurality of sets of pixels; and estimating the remaining shapes of the corresponding plurality of shapes using a pre-set size and distance relationship to the first shape.

18. The method of claim 15, wherein the plurality of markers are a plurality of concentric markers, wherein the clustering comprises: randomly selecting a set of pixels from the pixels of the plurality of markers; and using the set of pixels to hypothesize a modeling shape, wherein the modeling shape is a first shape of the corresponding plurality of shapes; estimating the remaining shapes of the corresponding plurality of shapes using a pre-set size and distance relationship to the first shape; determining a number of the pixels of the plurality of markers proximate to the first shape and the estimated remaining shapes of the corresponding plurality of shapes; repeating the steps of randomly selecting a set of pixels, using the set of pixels to hypothesize a modeling shape, estimating the remaining shapes of the corresponding plurality of shapes, and determining the number of pixels a prescribed number of times; selecting one of a plurality of sets of pixels based on the determined number of proximate pixels of the pixels of the plurality of markers; determining the corresponding plurality of shapes based on the selected one of the plurality of sets of pixels.

19. The method of claim 13, wherein a first color of the plurality of markers of the visual marker and a second color of a background of the visual marker are selected anywhere within a color spectrum.

20. The method of claim 13, further comprising: encoding a version of the visual marker in a first marker of the plurality of markers; and encoding data of the visual marker in the remaining markers of the plurality of markers.

21. The method of any one of claims 13 to 20, further comprising: determining data values of colors exhibited by a plurality of elements forming the plurality of markers, wherein the data values are determined based on color characteristics of the colors; and decoding further data encoded in the colors exhibited by the plurality of elements based on the determined data values of the colors.

22. A system for encoding visual markers, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform the method of any of claims 13-21.

23. A non-transitory computer-readable storage medium storing program instructions computer-executable on a computer to perform the method of any of claims 13-21.

Images