Colored visual indicia for variable uses

By determining an initial color group and generating multiple colors for visual markers based on conditions, the problem of poor detection performance of visual markers in different environments is solved, achieving high detectability under different conditions.

CN118897641APending Publication Date: 2024-11-05APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410945639.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2020-11-03
Filing Date
2020-11-09
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing technologies struggle to maintain the color detectability of visual markers under varying lighting, printing, and image sensor conditions, resulting in poor detection performance of visual markers in different environments.

Method used

By determining an initial color group and generating multiple colors for visual markers based on lighting conditions, printing conditions, and image sensor calibration, detectability is ensured under different conditions, and color selection and encoding are performed using a processor and non-transitory memory.

Benefits of technology

It achieves high detectability of visual markers under different lighting, printing and image sensor conditions, ensuring effective encoding and detection of visual markers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897641A_ABST
    Figure CN118897641A_ABST
Patent Text Reader

Abstract

The present disclosure relates to colored visual indicia for variable uses. Various implementations disclosed herein include devices, systems, and methods of selecting colors for visual indicia including colored indicia encoding data. In some implementations, the color is selected automatically or semi-automatically. In some implementations, the color is selected to remain sufficiently detectable despite changes in lighting or printing / display conditions. In some implementations, a set of colors are obtained that are selectable for use in visual markings. Then, a measure of the distance between the plurality of colors in the set of colors is determined, and a subset of the set of colors for visual marking is selected, the selection being based on the measure of the distance between the colors in the subset of colors. In some implementations, the visual indicia appearance includes graphical elements that encode the data with the subset of colors. In some implementations, input is received using a GUI on a display to determine a plurality of colors based on a source image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Related Applications Citations

[0002] This application is a divisional application of the invention patent application with Chinese national application number 202011238507.2, application date November 9, 2020, and invention name “Colored visual markings for variable uses”. Technical Field

[0003] The present disclosure relates generally to electronic devices, and particularly to systems, methods, and devices for selecting colors for visual indicia. Background Art

[0004] Today, visual markers exist in the form of barcodes, Quick Response (QR) codes, and other proprietary code-based systems. QR codes encode binary data such as a string or other payload to initiate a payment, link to a website, link to a location-based experience or context-based experience, or launch into other web-based experiences. Summary of the invention

[0005] Various embodiments disclosed herein include devices, systems, and methods for determining a plurality of colors (e.g., 2-8) of visual markers that encode data using colored markers. In some embodiments, the visual marker colors are selected to remain detectable despite wide variations in lighting conditions or printing / display conditions. In some embodiments, an initial set of colors is obtained. For example, this initial set of colors may be based on an input image, such as an image of the physical environment in which the visual marker will be deployed. Alternatively, this initial set of colors may be based on the color of the object (e.g., a poster) on which the visual marker will be installed. In another example, this initial set of colors is randomly generated or provided by a visual marker designer. Given the initial color set, in some embodiments, the plurality of colors (e.g., 2-8) of the visual marker are automatically determined (or the user is guided to select) based on the spacing of the plurality of colors in the color space. The determined plurality of colors may be the most invariant or sufficiently invariant to changes in lighting conditions or printing / display conditions (e.g., sufficiently spaced apart by a minimum preset distance or spatial spacing threshold in a 3D color space so that the colors can be distinguished from each other under different conditions). In some implementations, the multiple color selections can also take into account image sensor (eg, camera) characteristics or calibration when determining the multiple colors.

[0006] In some implementations, at an electronic device having a processor, a set of colors that are selectable for use in a visual mark is determined. Then, a measure of distance between multiple colors in the set of colors is determined, and a subset of the set of colors is selected for the visual mark, the selection being based on the measure of distance between colors in the subset of colors. In some implementations, an appearance of the visual mark is generated based on the subset of colors. In some implementations, the visual mark appearance includes a graphical element that encodes data using the subset of colors.

[0007] Various embodiments disclosed herein include devices, systems and methods for determining multiple colors based on a source image of a visual markup that encodes data using color. In some embodiments, a user selects a type of visual markup template and a position and size on a background or source image. In some embodiments, a user previews a colored visual markup. In some embodiments, a colored visual markup is generated, and the colored visual markup includes a colored mark (e.g., the size, shape, color or orientation of a graphic element) that encodes data using a determined color. In some embodiments, a GUI on a display is utilized to receive input from a designer. In some embodiments, a color is semi-automatically or automatically selected to remain fully detectable despite the presence of changes in lighting conditions or print / display conditions. In some embodiments, a colored visual markup is generated, stored, printed or displayed with an optional resolution.

[0008] In some implementations, at an electronic device having a processor, an input selecting a source image is received. In some implementations, a color based on the source image is determined. In some implementations, a visual marker and encoding data using the determined color are generated. In some implementations, a preview including the source image and the colored visual marker is displayed.

[0009] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0011] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0012] Figure 2 is a block diagram of an example controller according to some implementations.

[0013] Figure 3 is a block diagram of an exemplary electronic device according to some implementations.

[0014] Figure 4 is a flow chart illustrating an exemplary method of determining multiple colors of a visual marker for variations in lighting conditions, printing / display conditions, or image sensor calibration according to some implementations.

[0015] Figure 5 is a diagram illustrating the effects of various lighting conditions on multiple colors of a visual indicia according to some implementations.

[0016] Figure 6 is a diagram illustrating the effects of various printing conditions on multiple colors of a visual indicia according to some implementations.

[0017] Figure 7 is a diagram showing the effects of various image sensor calibrations for multiple colors of visual markers according to some specific implementations.

[0018] Figure 8 is a schematic diagram illustrating exemplary clusters resulting from grouping points in a 3D color space resulting from adjusting an initial color group for lighting conditions, printing or display conditions, or image sensor calibration, according to some implementations.

[0019] Fig. 9 is a diagram showing exemplary selections of multiple colors for a visual marker from an initial color set according to some implementations.

[0020] Fig.10 is a schematic diagram illustrating exemplary colored visual indicia according to some implementations.

[0021] Fig.11 is a flow chart illustrating an exemplary method of determining a set of colors for a visual marker that utilizes color to encode data, according to some implementations.

[0022] Fig.12 is a flow chart illustrating an exemplary method of determining multiple colors based on a source image of a visual marker that utilizes color to encode data, according to some implementations.

[0023] Fig.13 is a diagram showing an exemplary user interface for designing visual indicia that encodes data using color, according to some implementations.

[0024] Fig.14 is a schematic diagram illustrating exemplary colored visual indicia according to some implementations.

[0025] Fig.15 is a diagram showing an exemplary user interface including color selection for designing visual indicia that utilizes color to encode data according to some implementations.

[0026] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0027] Many details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings illustrate only some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example implementations described herein. Although Figures 1 to 3 Exemplary implementations are depicted involving handheld electronic devices, but other implementations may involve other types of devices, including but not limited to watches and other wearable electronic devices, mobile devices, laptop computers, desktop computers, gaming devices, head mounted devices (HMDs), home automation devices, and other devices that include or use image capture devices.

[0028] Figure 11 is a block diagram of an exemplary operating environment 100 according to some specific implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary specific implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120, one or both of which may be in a physical environment. The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0029] In some implementations, the controller 110 is configured to create, detect, or use visual markers or manage and coordinate a user's computer generated reality (CGR) environment. In some implementations, the controller 110 includes a suitable combination of software, firmware, or hardware. Figure 2 Controller 110 is described in further detail. In some implementations, controller 110 is a computing device that is located locally or remotely relative to physical environment 105.

[0030] In one example, the controller 110 is a local server located within the physical environment 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the controller 110 is communicatively coupled to the electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0031] In some implementations, the controller 110 and the electronic device 120 are configured to create, detect, or use visual markers or together present a CGR environment to a user.

[0032] In some implementations, the electronic device 120 is configured to create, detect, or use visual markers or present a CGR environment to a user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, or hardware. Figure 3 The electronic device 120 is described in more detail. In some implementations, the functionality of the controller 110 is provided by or combined with the electronic device 120, for example, in the case of an electronic device used as a stand-alone unit.

[0033] According to some specific implementations, when the user appears in the physical environment 105, the electronic device 120 presents a CGR environment to the user. A CGR environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In CGR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one physical law. For example, the CGR system can detect a person's head rotation, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. In some cases (e.g., for accessibility reasons), the adjustment of features of virtual objects in the CGR environment can be performed in response to a representation of physical movement (e.g., a voice command).

[0034] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0035] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0036] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0037] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0038] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0039] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0040] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0041] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0042] Figure 2is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, or similar type interface), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0043] In some implementations, the one or more communication buses 204 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.

[0044] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and a visual marking module 240.

[0045] Operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks.

[0046] In some implementations, the visual marking module 240 is configured to identify, share or manage visual marking information. In some implementations, the visual marking module 240 includes an initial color unit 242, a color variant unit 244, a color selection unit 246 and an encoding / decoding unit 248. The initial color unit 242 is configured to identify or receive an initial color group that can be selected for use in a colored visual marking. The color variant unit 244 is configured to deal with the variation of the group of selectable colors based on lighting conditions, printing conditions or image sensor conditions. The color selection unit 246 is configured to select a subgroup of the group of selectable colors for visual marking, and the selection is based on a measure of the color distance between the subgroup colors. The encoding / decoding unit 248 is configured to encode / decode the colored mark encoding data of the visual marking.

[0047] The visual marker module 240 may be used to create visual markers, for example, for use by CGR applications that are configured to present virtual content (e.g., 3D content) to be used as part of a CGR environment for one or more users. For example, a user may view and otherwise experience a CGR-based user interface that allows the user to select, place, move, and otherwise experience the CGR environment based at least in part on the detection and use of visual markers (e.g., via gestures, voice commands, input device input, etc.).

[0048] Although these modules and units are shown as residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of these modules and units may be located in separate computing devices. Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0049] Figure 3is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external-facing image sensors, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0050] In some implementations, one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0051] In some implementations, one or more displays 312 are configured to present a CGR environment to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) or similar display types. In some implementations, one or more displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, the electronic device 120 includes a single display. As another example, the electronic device 120 includes a display for each eye of the user.

[0052] In some implementations, one or more of the internally facing or externally facing sensor systems include an image capture device or array that captures image data or an audio capture device or array (e.g., a microphone) that captures audio data. In some implementations, the one or more image sensor systems 314 are configured to acquire image data corresponding to at least a portion of a user's face including the user's eyes. For example, the one or more image sensor systems 314 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 314 also include an illumination source that emits light (e.g., to a subject or a portion of a user's face), such as a flash or flash source.

[0053] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or their subsets, including an optional operating system 330 and a visual marking module 340.

[0054] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks.

[0055] In some embodiments, the visual marking module 340 is configured to identify, share or manage visual marking information. In some embodiments, the visual marking module 340 includes an initial color unit 342, a color variant unit 344, a color selection unit 346 and an encoding / decoding unit 348. The initial color unit 342 is configured to identify or receive an initial color group that can be selected for use in the colored visual marking. The color variant unit 344 is configured to deal with the variation of the group of optional colors based on lighting conditions, printing conditions or image sensor conditions. The color selection unit 346 is configured to select a subgroup of the group of optional colors for visual marking, and the selection is based on a measure of the color distance between the subgroup colors. The encoding / decoding unit 348 is configured to encode / decode the colored mark encoding data of the visual marking.

[0056] The visual marker module 340 may be used to create visual markers for use, for example, by CGR applications configured to present virtual content (e.g., 3D content) to be used as part of a CGR environment for one or more users. For example, a user may view and otherwise experience a CGR-based user interface that allows the user to select, place, move, and otherwise experience the CGR environment based at least in part on the detection and use of visual markers (e.g., via gestures, voice commands, input device input, etc.).

[0057] also, Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are allocated among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware selected for a specific specific implementation.

[0058] Various implementations disclosed herein include devices, systems, and methods for determining multiple colors (e.g., 2-8) of a visual marker that encodes data using a colored marker. In some implementations, the multiple colors of the determined visual marker remain detectable to an electronic device despite wide variations in lighting conditions or printing / display conditions (e.g., where the visual marker is deployed).

[0059] In various implementations, a visual marker is a 2D object that encodes binary data such as a string (e.g., a QR code) or other payload for accessing a remote-based experience (e.g., a remote-based experience). In some implementations, the link to the remote-based experience includes a link to initiate payment (e.g., an approved payment endpoint), a link to a website (e.g., a URL), or a link that launches into a web-based experience. In some implementations, the visual marker is used to launch or link only to approved remote-based experiences authorized by the creator of the visual marker. In some implementations, the creator of the visual marker includes an entity that designs the visual marker, an entity that prints (e.g., produces) the visual marker (e.g., a developer), and an entity that manages / hosts the visual marker.

[0060] Figure 44 is a flow chart illustrating an exemplary method 400 for determining multiple colors of a visual marker that responds to variations in lighting conditions, printing / display conditions, or image sensor calibration according to some implementations. In some implementations, the method 400 is performed by a device (e.g., Figures 1 to 3 The method 400 is performed by a controller 110 or an electronic device 120). The method 400 may be performed using an electronic device or by multiple devices communicating with each other. In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0061] At box 410, method 400 obtains an initial color group for use in a visual mark. In some implementations, the initial color group can be any number of colors (e.g., 30, 80, 150). In some implementations, the arbitrary number of colors in the initial color group is based on a computational budget. In some implementations, the initial color group can be based on an initial input image. In some implementations, the initial color group can be any color group. In some implementations, the initial color group is randomly generated. In some implementations, the initial color group is selected by a visual mark designer. In some implementations, the initial color group is selected based on the color at the location in the physical environment. In some implementations, the initial color group is selected based on the input image. In some implementations, the initial color group can be based on the initial input image by extracting the colors in the initial input image and grouping or selecting from the extracted colors to reduce the number of colors in the initial input image until the arbitrary number of the initial color group is reached. For example, this can involve identifying the highest N colors in the image, for example, the 10 colors with the largest number of pixels. In one specific implementation, the 3D color space representing the colors from the original input image may be discretized into the arbitrary number.

[0062] At block 415, once the initial color group is determined, the method 400 determines the effect of the lighting conditions on the initial color group in the visual mark. In some implementations, the initial color group is referred to as palette 0. In some implementations, multiple instances of palette 0 under various lighting conditions are generated as palette I, where I equals 1, 2, ..., N; where N is a positive integer greater than 2. In some implementations, the multiple palettes I are synthetically determined using a chromatic adaptation algorithm. In some implementations, the multiple palettes I are determined using actual images of palette 0 obtained under various real-world lighting conditions.

[0063] Figure 5 is a diagram showing the effects of various lighting conditions on visual markers according to some specific implementations. Figure 5As shown, the appearance of the visual marker 500 using four selected colors to encode data is affected by the lighting conditions (e.g., where the visual marker 500 will be deployed). Figure 5 As shown, the effects of various light sources 500a, 500b, ..., 500g are generated by computer modeling. In some implementations, all possible lighting conditions are known because the possible lighting sources that exist in the world are known (e.g., the sun and its spectrum, light bulbs, LED light sources, incandescent lamps, fluorescent lamps, etc.). In some implementations, a single lighting condition for the deployed visual sign is known because, for example, the visual sign designer knows that the visual sign will be deployed in a museum with a single fixed interior lighting condition.

[0064] At block 420, once the initial color set is determined, the method 400 determines the effect of printing the initial color set. Figure 6 is a diagram showing exemplary printing condition effects on visual indicia according to some specific implementations. Figure 6 As shown, an exemplary RGB color space 610 (illustrated, for example, as a cube) is converted to an exemplary print color space 620. Figure 6 As shown, printing conditions have an impact on the exemplary RGB color space 610. Figure 6 As shown, the printing conditions squeeze various colors of the RGB color space 610 by different amounts in generating a modified printed color space 620 .

[0065] For example, in some implementations, the printer combines four colors of ink (e.g., CMYK) to generate an input printed output. In another example, in some implementations, the printer combines three colors of ink (e.g., RGB) to generate an input printed output.

[0066] In some implementations, the type of printer is known, the model of the printer is known, and even the amount of printed material (e.g., CYMK ink level) is known (e.g., by the user or visual mark developer). In some implementations, the modified printer color space 620 becomes larger or more accurate as more information about the printing conditions is known.

[0067] At block 425, once the initial color set is determined, the method 400 determines the effect of the image sensor or camera on the initial color set of the visual marker. In some implementations at block 425, the method adjusts the initial color set for image sensor calibration or camera calibration. In various implementations, the image sensor or camera may not detect 2 different colors in a real-world physical environment as those same exact 2 colors. In some implementations, the image sensor or camera calibration is responsible for the conversion from real-world colors to colors sensed (e.g., seen) by a specific camera model, a specific camera manufacturer, or a specific type of camera.

[0068] Figure 7 is a diagram showing exemplary camera calibration effects for visual markers according to some specific implementations. Figure 7 As shown, the exemplary RGB color space 710 (eg, illustrated as a cube) is converted to an exemplary modified sensor calibration color space 720 based on camera calibration. Figure 7 As shown, the exemplary camera calibration has an effect on the exemplary RGB color space 710. Figure 7 As shown, the camera calibration modifies various colors of the RGB color space 670 by different amounts when generating the modified sensor calibration color space 720.

[0069] In some implementations, the type of image sensor is known, the model of the image sensor is known, or even one or more possible calibrations for a range of image sensors are known (e.g., known by the visual mark developer). In some implementations, as more information about the image sensor calibration is known or provided (e.g., a more accurate calibration), the modified sensor calibration color space 720 becomes larger or more accurate.

[0070] In some implementations at block 425, method 400 uses information from block 420 about a printer that will be used to print the visual mark to map palette 0 and palette 1 from a first 3D color space (e.g., an RGB 3D color space) to a second 3D color space (e.g., a CMYK 3D color space). In some implementations at block 425, method 400 uses information from block 420 about printer conditions or a printing device that will be used to print the visual mark to map at least a subset of palette 0 and palette 1 from a first 3D color space (e.g., an RGB 3D color space) to a second 3D color space (e.g., a CMYK 3D color space) to account for the effects of lighting and printing conditions on the initial color set.

[0071] In some specific implementations at box 425, method 400 uses information from box 425 about the calibration of the image sensor or camera to be used to detect the visual mark to map colors from an initial color set in the second 3D color space that is modified by lighting and printing conditions to a third 3D color space (e.g., the RGB 3D color space of the image sensor) to account for the effects of the image sensor calibration on the initial color set.

[0072] At block 430, method 400 groups the adjusted initial color components in a third 3D color space (e.g., image sensor RGB 3D color space) into "K" color clusters, where K is a positive integer (e.g., 30, 50, 100, etc.). In some implementations, the grouping in block 430 uses known clustering techniques. In some implementations, the grouping in block 430 uses known nearest neighbor techniques or other known techniques to group the adjusted initial color groups in the third 3D color space in block 430.

[0073] Figure 8 is a schematic diagram showing exemplary clusters resulting from grouping points in a third 3D color space that are derived from adjusting an initial color group for lighting conditions, printing conditions, or image sensor calibration, according to some implementations. Figure 8 As shown, the input image 810 is used to obtain an initial color group 820 including 24 colors to represent the input image 810. Figure 8 As shown, the initial color set 820 produces adjusted colors 840 in the camera RGB 3D color space 830. Figure 8 As shown, adjusted colors 840 are derived from an initial color set 820 modified for printing conditions, lighting conditions, or camera calibration. Figure 8 As shown, exemplary clusters 840a, 840b include subsets of adjusted colors 840 generated from one or more of the initial color groups 820. In some implementations, the “K” clusters of adjusted colors 840 each include a plurality of the initial color groups 820. In some implementations, the number of “K” clusters of adjusted colors 840 is greater than the number of initial color groups 820.

[0074] In blocks 435-450, method 400 selects a plurality of colors (e.g., 2-8) to be used on the visual mark (e.g., from the initial set of colors obtained in block 410). In some implementations, the plurality of colors are selected automatically, semi-automatically, or manually from the initial set of colors. In some implementations, once a first color from the plurality of colors is selected (e.g., automatically, semi-automatically, or manually), the remaining colors from the plurality of colors are automatically selected from the initial set of colors. In some implementations, the remaining colors from the plurality of colors are semi-automatically or manually selected from the initial set of colors. In some implementations, the plurality of colors used for the visual mark are selected to remain detectable and distinguishable from one another despite wide variations in lighting conditions, printing / display conditions, or detection equipment (e.g., image sensors).

[0075] At block 435, the method 400 selects a first color of the plurality of colors of the visual marker from the initial set of colors obtained in block 410. In some implementations, the first plurality of colors are randomly selected from the initial set of colors obtained in block 410. In some implementations, the first plurality of colors are selected from the initial set of colors obtained in block 410 based on an initial input image used to obtain the initial set of colors (e.g., the dominant colors of the initial input image). In some implementations, the first plurality of colors are selected from the initial set of colors obtained in block 410 by a user.

[0076] In some implementations at block 435, the initial color groups are presented or displayed to the user in an ordered sequence. In some implementations, the initial color groups are ordered according to a color spectrum, such as a visible spectrum (e.g., red to blue). In some implementations, the initial color groups are ordered based on which of the K clusters has the largest number of points, such that the subgroup of the initial color group contained in the largest of the K clusters is first in the ordered sequence. In some implementations, the initial color groups are randomly ordered.

[0077] At block 440 , the method 400 determines whether the last of the plurality of colors has been selected (eg, the fourth color of the plurality of 4 colors used in the visual mark is selected). When the determination in block 440 is negative, the method 400 continues to block 445 .

[0078] At block 445, the method 400 removes or eliminates colors in the 3D color space that are close to the most recently selected color in the plurality of colors from the initial color group. In some implementations, the nearest neighboring clusters of the cluster containing the most recently selected color in the plurality of colors are eliminated along with any of the initial color groups contained therein, which causes the nearby colors in the initial color group to be removed or eliminated from some future selections (in block 450). In some implementations, at block 445, clusters within a threshold spatial distance in the third 3D color space from the cluster containing the most recently selected color in the plurality of colors are eliminated.

[0079] At block 450, the method 400 selects a next color in the plurality of colors from the remaining colors in the initial color group. At block 450, the next color in the plurality of colors may be selected from the remaining colors in the initial color group, as described above in block 435. From block 450, the method returns to block 440.

[0080] Fig. 9 is a schematic diagram illustrating an exemplary selection of a plurality of colors from an initial set of colors for use in a coded colored mark in a visual indicia according to some implementations. Fig. 9 As shown, four colors A, B, C, and D are selected from an initial group of 46 colors 920 as a plurality of colors 950 for visual marking.

[0081] like Fig. 9 As shown, A, B, C, D may be automatically selected as the plurality of colors 950 for visual markings based on the size and spatial arrangement of the clusters 940a, 940b, ..., 940n that include the adjusted initial color group 940 in the camera RGB 3D color space 930. In some implementations, after the first color is selected in the initial set of 46 colors 920, the remaining colors in the plurality of colors 950 (e.g., A, B, C, D) are automatically selected based on the total spatial distance between the colors of the plurality of colors 950 (e.g., A, B, C, D) represented by the clusters in the RGB 3D color space 930.

[0082] Alternatively, if Fig. 9 As shown, the initial set of 46 colors 920 can be presented to the user in an ordered sequence. In this implementation, the user selects the first color (A) of the plurality of colors 950 from the sorted initial set of 46 colors 920. Then, in some implementations, the colors that are spatially close to color A based on the 3D color space are removed from the initial set of 46 colors 920, which results in the remaining colors 920a of the initial color set. In one implementation, clusters close to the cluster containing the first color A in the adjusted initial color set 940 in the camera RGB 3D color space 930 are eliminated.

[0083] Then, in this implementation, the 26 remaining colors 920a of the initial set of 46 colors 920 are presented to the user in an ordered sequence. In some implementations, the remaining colors 920a of the initial set of 46 colors 920 are presented such that the remaining colors are listed from best to worst choices in the ordered sequence from first to last. Fig. 9 As shown, the user selects a second color (B) from the sorted remaining colors 920a in the plurality of colors 950. Then, in some implementations, the remaining colors 920a of the initial set of 46 colors 920 that are spatially close to color B based on the 3D color space are removed, resulting in the remaining colors 920b of the initial color set.

[0084] Then, in this implementation, the 19 remaining colors 920b of the initial set of 46 colors 920 are presented to the user in an ordered sequence. In some implementations, the remaining colors 920b of the initial set of 46 colors 920 are presented such that the remaining colors are listed from best to worst choices in the ordered sequence from first to last. Fig. 9 As shown, the user selects a third color (C) from the sorted remaining colors 920b in the plurality of colors 950. Then, in some implementations, the remaining colors 920b of the initial set of 46 colors 920 that are spatially close to color C based on the 3D color space are removed, resulting in the remaining colors 920c of the initial color set.

[0085] Then, in this implementation, the five remaining colors 920c of the initial set of 46 colors 920 are presented to the user in an ordered sequence. In some implementations, the remaining colors 920c of the initial set of 46 colors 920 are presented such that the remaining colors are listed from best to worst choices in the ordered sequence from first to last. Fig. 9 As shown, the user selects the fourth color (D) of the plurality of colors 950 from the sorted remaining colors 920c, and the selection of the plurality of colors 950 is completed. Fig. 9 As shown, the highest color of the sorted initial set of 46 colors 920, remaining colors 920a, 920b, 920c is selected for the plurality of colors 950, however, any color contained in each of the sorted initial set of 46 colors 920, remaining colors 920a, 920b, 920c is selectable at that time.

[0086] Returning to method 400, when the determination in block 440 is affirmative, method 400 continues to block 455. At block 455, method 400 places each of the plurality of colors at a fixed position at the visual marker.

[0087] Fig.10 is a schematic diagram showing exemplary colored visual indicia according to some specific implementations. Fig.10 As shown, the visual mark 1000 includes a first portion 1010 for detection and orientation, a second portion 1014A-D for identifying a color used in the visual mark 1000 (which is a plurality of colors (e.g., 950) in method 400), and a third portion 1020 for encoding data in the visual mark 1000 using color.

[0088] like Fig.10 As shown, the first portion 1010 includes a preset shape for detection and orientation. In some implementations, the first portion 1010 is an inner area having a predefined shape that is consistent for multiple different visual markers (e.g., visual markers that use different colors to encode different data). Fig.10 As shown, the first portion 1010 is an inner ring having a particular overall predefined shape.

[0089] In some implementations, the second portions 1014A-D include known locations or positions in the first portion 1010 or a particular overall shape of the visual marker 1000, each of which is colored in one of a plurality of colors (e.g., 950) used in the visual marker 1000. Fig.10 As shown, the second portion 1014 includes four locations 1014A, 1014B, 1014C, and 1014D in the first portion 1010, each of which is colored with one of the A, B, C, or D colors used in the visual mark 1000. In other specific implementations, the second portion 1014 may include a different number of locations corresponding to each of the colors selected for the visual mark. Fig.10 As shown, the second portion 1014 includes elements that make up the first portion 1010. In other implementations, the first portion and the second portion are distinct and spaced apart.

[0090] In some implementations, the third portion 1020 uses color to encode the data of the visual marker 1000. Fig.10 As shown, the third portion 1020 uses shape and color to encode data of the visual mark 1000. In some implementations, the third portion 1020 is the outer area of ​​the colored shape. Fig.10 , the third portion 1020 includes colored shapes 1025-A1, 1025-A2, ... arranged in a clockwise or counterclockwise order from the starting position. Fig.10 As shown, the third portion 1020 includes a colored shape 1025 that includes a combination of 5 sizes, 2 shapes, 4 colors, and 2 orientations (eg, radially inwardly tapering or radially outwardly tapering).

[0091] In some implementations, the additional portion of the visual marker 1000 can be colored using a single color (e.g., white or gray). In some implementations, the additional portion of the visual marker 1000 can be colored using a single color that is not used in the first portion 1010 or the third portion 1020. In some implementations, the additional portion of the visual marker 1000 is at least one of the following: a central area of ​​the visual marker 1000, an area of ​​the visual marker 1000 between the first portion 1010 and the third portion 1020, or an area (e.g., a ring) outside the third portion 1020. In some implementations, the additional portion of the visual marker 1000 is at least one of the following: an area of ​​the visual marker 1000 adjacent to at least a portion of the first portion 1010 or adjacent to at least a portion of the third portion 1020. In some implementations, the additional portion of the visual marker 1000 is used to perform local white balancing of the colors in the visual marker 1000 when detected by the image sensor. In some implementations, additional portions of the visual marker 1000 are used to detect spatially varying lighting at the detected visual marker or to correct for any detected spatially varying lighting. For example, when shadows are detected in the additional region that is both the central region of the visual marker 1000 and the region outside the third portion 1020 (e.g., across a portion of the visual marker 1000), the detected shadows in the additional region can be used to correct for color variations of the visual marker 1000 (e.g., the first portion 1010, the third portion 1020) caused by the shadows. In some implementations, the spatially varying lighting at the detected visual marker is caused by a light source, uneven lighting, objects in the physical environment, and the like.

[0092] In various implementations, visual marker 1000 is a 2D or 3D object that encodes binary data such as a string (e.g., a QR code) or other payload for accessing a remote-based experience. In some implementations, the link to the remote-based experience includes a link to initiate payment (e.g., an approved payment endpoint), a link to a website (e.g., a URL), or a link to launch into a web-based experience.

[0093] In some implementations, a sensor (e.g., a camera) on the electronic device is used to acquire an image in which the visual marker 1000 is detected in an image of the physical environment. In some implementations, the sensor is a combination of sensors such as an RGB camera, a stereo camera, a depth sensor (e.g., time of flight, structured light), an RGB-D camera, one or more 2D cameras, an IR camera, and the like.

[0094] In some implementations, the electronic device can determine a relative position between the electronic device and the visual marker 1000 detected in the one or more images of the physical environment including the visual marker 1000. In some implementations, the relative position determines a relative orientation of the visual marker 1000 relative to the electronic device. In some implementations, the relative position is determined using computer vision techniques (e.g., VIO or SLAM) or PNP techniques.

[0095] In such Figures 9 and 10 In some implementations shown, at block 455 , the method 400 places colors A, B, C, and D of the plurality of colors 950 at four locations 1014A, 1014B, 1014C, and 1014D of the second portion 1014 of the visual mark 1000 .

[0096] In some implementations, once the plurality of colors are selected, an alternative color selection group (e.g., an alternative plurality of colors) for the visual mark can be presented to the designer, and the electronic device will be able to detect the color selection despite wide variations in lighting conditions or printing / display conditions. In some implementations, the alternative colors in each cluster containing one of the plurality of colors can be used as alternative colors (e.g., A', B', C', D') for the plurality of colors to encode the same data as the initial plurality of colors, which allows the designer to freely choose from an increased number of colors without compromising the lighting or printing invariant decoding of the deployed visual mark. In some implementations, the alternative color selection group (e.g., an alternative plurality of colors) for the visual mark is a combination of colors from the initial color group that are respectively included in each of the K clusters containing the plurality of colors selected for the visual mark. For example, when 100 colors numbered 1-100 are in the initial color group, color 10 (in cluster W), color 30 (in cluster X), color 60 (in cluster Y), and color 90 (in cluster Z) are the plurality of colors. In this example, cluster W also includes colors 12 and 14, cluster X also includes colors 22, 29, and 32, cluster Y includes colors 62, 64, and cluster Z also includes color 2. In this example, an alternative color selection group (e.g., an alternative plurality of colors) for the visual marker is a combination of 4 colors, where 1 color is selected from each of cluster W, cluster X, cluster Y, and cluster Z.

[0097] In some implementations, when obtaining the initial color set (e.g., block 410), the designer of the visual mark may add at least one color that was not initially provided in the initial color set (e.g., in palette 0). Then, when selecting the plurality of colors (e.g., the remainder of method 400), any colors added to the initial color set will also be considered or optimized. In some implementations, when the plurality of colors obtained from the initial color set do not have sufficient spatial separation in the 3D color space, the designer of the visual mark may add a color that was not initially provided in the initial color set. In some implementations, the designer of the visual mark may use color blending techniques or the science of color blending to add at least one color that was not initially provided in the initial color set. In some implementations, the designer of the visual mark may use color blending techniques or the science of color blending to automatically, semi-automatically, or manually add at least one color that was not initially provided in the initial color set. Color blending provides a recipe for generating or selecting a new color that blends naturally with an existing color palette. For example, golden yellow blends with royal blue. In some implementations, color harmony can be used to automatically, semi-automatically, or manually select the plurality of colors or an alternative color selection group out of an initial set of colors.

[0098] In some implementations, before (e.g., automatically or manually) selecting the plurality of colors from the initial color set (e.g., blocks 435-450), the designer of the visual mark can remove shades of white and black from the initial color set. In some implementations, removing shades of white and black from the selection will result in greater diversity in the plurality of colors, since white and black are typically selected based on their respective positions in the 3D color space.

[0099] In some implementations, the actual lighting conditions in which the visual marker will be deployed are known. In such implementations, additional flexibility or accuracy is provided in selecting the multiple colors for use in the colored visual marker. For example, consider that the visual marker will be displayed on a poster illuminated by blue LED lights. In this example, if red, green, and blue are used to encode the data on the visual marker, the data will be viewed by the electronic device as black, black, and blue, and the visual marker will be unreadable. In this example, when the known actual lighting conditions are blue LEDs, the multiple colors should be blue chromaticities or in other words detectable when illuminated by blue LEDs.

[0100] In some implementations, the electronic device that captures the input image on which the visual marker is to be deployed (e.g., for obtaining an initial color set) includes an ambient light sensor. In some implementations, data from the ambient light sensor is included in the information about the lighting conditions used to determine the multiple colors of the colored visual marker.

[0101] Various embodiments disclosed herein include graphical user interfaces (GUIs) for devices, systems, and methods for determining a plurality of colors of visual markers encoding data using colored markers as described herein. In some embodiments, the GUI provides a user with the ability to obtain an initial color group by manual selection, random generation, based on received color input, based on a color input image, etc. In some embodiments, the GUI provides a user with the ability to add colors to the initial color group or remove colors from the initial color group. In some embodiments, the GUI provides a user with the ability to accept default lighting conditions or identify, select, or modify in a set of lighting conditions. In some embodiments, the GUI provides a user with the ability to accept default printing conditions or identify, select, or modify in a set of printing conditions. In some embodiments, the GUI provides a user with the ability to accept default image sensor calibrations or identify, select, or modify in a set of image sensor calibrations. In some embodiments, the GUI provides a user with the ability to prioritize (e.g., weight) between lighting conditions, printing conditions, or image sensor calibrations when generating the plurality of colors used in colored visual markers. In some implementations, the GUI provides the user with the ability to select one or more of the lighting conditions, printing conditions, or image sensor calibrations when generating the multiple colors used in the colored visual mark. In some implementations, the GUI provides the user with the ability to manually, semi-automatically (e.g., recommended colors) or automatically select the multiple colors in the initial color group. In some implementations, the GUI provides the user with the ability to view the detectability intensity or measure of the multiple colors relative to a standard scale or relative to the deployed environment (e.g., digital or physical), including the recommendation of adding additional colors (e.g., or the recommended colors to be added) in order to reach a threshold value of the detectability measure. In some implementations, the GUI provides the user with the ability to be presented with a potential alternative group of multiple colors, generate a potential alternative group of multiple colors, or modify a potential alternative group of multiple colors. In some implementations, the GUI provides the user with the ability to encode data into a visual mark using the multiple colors and parameterized colored shapes.

[0102] Fig.11 1 is a flow chart illustrating an exemplary method 1100 for determining a set of colors for a visual marker that utilizes color to encode data according to some implementations. In some implementations, the set of colors for the visual marker responds to variations in lighting conditions or printing / display conditions. In some implementations, the set of colors for the visual marker is determined by a device (e.g., Figures 1 to 3The method 1100 is performed by a controller 110 or an electronic device 120 of the computer program product. The method 1100 may be performed using an electronic device or by multiple devices communicating with each other. In some implementations, the method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1100 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0103] In some implementations, the method 1100 obtains an initial color group (e.g., 100) based on an input image of a deployed visual marker location, for example, and automatically determines (or guides a user to select) a color group (e.g., 4) for the visual marker based on the relative positions of the plurality of colors in the color space. In some implementations, the determined set of colors may be the most invariant or sufficiently invariant to changes in lighting conditions or printing / display conditions based on exceeding a spatial distance interval threshold in the 3D color space.

[0104] At box 1110, method 1100 determines a set of colors that can be selected for use in the visual tag. In some embodiments, the set of selectable colors is an arbitrary number of colors. In some embodiments, the set of colors totals 30, 40, or 80. In some embodiments, the number of colors in the set is based on a computational budget or time. In some embodiments, the set of colors is based on the physical location at which the visual tag is to be deployed. In some embodiments, the set of colors is based on the colors in the input color image. For example, the input image may include 1 million colors, which can be reduced to 100 colors as a set of colors that still represent the input color image. In some embodiments, when the input color image only includes a chromaticity of blue, the set of colors may include only blue.

[0105] At block 1120, the method 1100 determines a measure of distance between the plurality of colors in the set of colors. In some implementations, a measure of distance between all colors in the set of colors is determined. In some implementations, the measure of distance is based on the spatial position of the plurality of colors in the set of colors in a 3D color space, such as an RGB 3D color space, an XYZ 3D color space, a CYMK 3D color space, etc. In some implementations, the measure of distance is based on the position of the plurality of colors in the set of colors along a color spectrum. For example, the measure of distance may determine that a blue is 10 units away from a first green, and that the blue is 12 units away from a second green.

[0106] At block 1130, method 1100 selects a subset of the set of colors for visual marking, the selection being based on a measure of distance between the colors of the color subset. In some implementations, the color subset is selected so that there is an increased or maximum spacing between the color subsets in the 3D color space. In some implementations, the color subset is selected so that there is a threshold spacing between pairs of the color subsets in the 3D color space. In some implementations, the color subset is selected because the color subset is sufficiently distinguishable or discernible independent of lighting conditions, printing / display conditions, or a device (e.g., an image sensor or a color camera) that detects the visual marking. In some implementations, the color subset is selected because the color subset is the most invariant or sufficiently invariant to changes in lighting conditions, printing / display conditions, or a detection device (e.g., an image sensor or a color camera). In some implementations, black and white are restricted from the color subset. In the example of block 1120, blue and a second green are selected because the second green is more different from blue than the first green.

[0107] At block 1140, method 1100 generates an appearance of the visual mark based on the color subset. In some implementations, the visual mark appearance includes placing each of the color subset at a preset fixed position on the visual mark. In some implementations, the visual mark appearance includes a graphical element that encodes data using the color subset. In some implementations, the graphical element also encodes the data of the visual mark using shape or size.

[0108] For example, in one colored visual mark 10, the colored marks use 2 colors to encode data (e.g., the colored marks have the same shape and size). The series of colored marks in the visual mark is T1, P1, T, T2, P3, P2, P5, T3, P, P4, where T represents a tan colored mark and P represents a pink colored mark. Thus, the data encoded in the visual mark may be 0100111011.

[0109] In some implementations at block 1120, variations of the set of colors are identified based on information about lighting conditions under which the visual indicia is to be deployed, printing device conditions of a printer that will print the visual indicia, display device conditions of a device that will display the deployed visual indicia, or a detection device (e.g., an imaging sensor) that will detect the deployed visual indicia. In some implementations, variations of the set of colors are identified based on information about multiple lighting conditions, multiple printing device conditions, multiple display device conditions, or multiple detection device conditions. In some implementations, the measure of distance takes into account variations of the identified set of colors based on lighting conditions, printing conditions, display conditions, or detection (e.g., imaging) conditions.

[0110] In some implementations, a sensor (e.g., image sensor, camera) on a mobile electronic device with a processor is used to obtain an image of a physical environment including visual markers at a deployed location. In some implementations, the image is a 2D image or a 3D image. In some implementations, a combination of image sensors is used. In some implementations, a color image may be used. Alternatively, in some implementations, a grayscale image may be used. In some implementations, an image of the physical environment is used to generate a CGR environment representing the physical environment. In some implementations, the CGR environment is generated using VIO or SLAM position tracking at a mobile electronic device, etc.

[0111] Fig.12 1 is a flow chart illustrating an exemplary method 1200 for determining a plurality of colors based on a source image of a visual marker that utilizes color to encode data (e.g., size, shape, color, or orientation of a graphical element) according to some implementations. In some implementations, a user selects a type of visual marker template, and a location and size on a background or source image. In some implementations, a user previews the colored visual marker. In some implementations, a user previews the colored visual marker overlaid with a source image. In some implementations, method 1200 is performed by a device (e.g., Figure 1-Figure 3 The method 1200 is performed by a controller 110 or an electronic device 120). The method 1200 may be performed using an electronic device or by multiple devices communicating with each other. In some implementations, the method 1200 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1200 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0112] At block 1210, method 1200 receives input for selecting a source image. In some implementations, the input for selecting a source image is received at a display. In some implementations, the input for selecting a source image is received at a display using a GUI. In some implementations, the source image is selected from a locally stored image. In some implementations, the source image is imported from a remotely stored image.

[0113] At block 1220, method 1200 determines a plurality of colors based on a source image. In some implementations, the plurality of colors are automatically selected from a source image. In some implementations, the plurality of colors are semi-automatically selected based on a source image. In some implementations, a GUI is used to receive input for selecting a first color of the plurality of colors based on a source image. In some implementations, a GUI is used to receive user input for selecting all of the plurality of colors based on a source image. In some implementations, user input into a color picker is used to select the plurality of colors. In some implementations, a GUI is used to receive user input for selecting a first color of the plurality of colors based on a source image, and the remaining colors of the plurality of colors are automatically selected. In some implementations, the plurality of colors is a preset number (e.g., 2-8). In some implementations, the plurality of colors may be determined based on detectability. In some implementations, the detectability of the plurality of colors is based on one or more of spacing in a 3D color space, lighting conditions, printing conditions, display conditions, image capture sensors, or aesthetic information. In some implementations, user input selects the plurality of colors, where one or more of the colors can be automatically modified based on detectability.

[0114] At box 1230, method 1200 generates a colored visual mark and uses the determined multiple colors to encode data. In some implementations, a GUI is used to receive a user input for selecting the type (e.g., triangle, rectangle, hexagon, 2 colors, 3 colors, 4 colors, etc.) of the visual mark template for the colored visual mark. In some implementations, the user input selects the layout or style design options for the visual mark. In some implementations, the visual mark is a 2D code that encodes data using a colored mark (e.g., the size, shape, color, or orientation of the graphic element). In some implementations, a GUI is used to receive a user input for selecting the data to be encoded in the colored visual mark.

[0115] At block 1240, method 1200 displays a preview including the source image and the colored visual marker. In some implementations, the preview may be overlapped relative to the source image. In some implementations, a GUI is used to receive user input for selecting a location or size on the source image or background. In some implementations, a GUI is used to receive user input for storing, copying, previewing, or transmitting the generated colored visual marker. In some implementations, the preview includes a portion or all of the viewable display. In some implementations, the source image is used as all or a portion of the background for the colored visual marker.

[0116] Fig.13 is a diagram showing an exemplary UI for designing visual markers that encode data using color according to some specific implementations. Fig.13 As shown, UI 1300 includes features that allow receiving input from a designer for designing or displaying a designed colored visual mark. In some implementations, UI 1300 can be provided or stored as a system service using an operating system of an electronic device.

[0117] In some implementations, the UI 1300 includes a composition section 1305 that allows receiving input from a designer for entering a new colored visual marker or accessing an already designed and stored colored visual marker. Fig.13 As shown, the selected composition 1305a is being designed, and compositions 1305b and 1305c have been created and stored (e.g., locally). In some implementations, components 1305b and 1305c are stored locally in the designer's document folder. In some implementations, a composite of a visual mark is designed or stored, which may include a visual mark, an input image, additional parameters entered by the designer (such as the multiple colors used to encode data in the visual mark), and one or more of the composite of the input image and the colored visual mark. In some implementations, the composition section 1305 includes a selection 1310a (e.g., +) for starting a new visual mark in the composition section 1305, a selection 1310b (e.g., -) for removing the visual mark from the composition section 1305, and a selection 1310c (e.g., ∞) for copying the selected visual mark in the composition section 1305 as a new visual mark (e.g., with the default parameters of the copied visual mark as a starting point).

[0118] like Fig.13 As shown, the selected composition 1305a includes the visual marker 1400 and the input image 1315 displayed in the viewer 1370. Fig.14 is a schematic diagram illustrating an exemplary visual marker 1400 .

[0119] In some implementations, the parameter portion 1320 allows receiving input from the designer for selecting among various parameters of the colored visual markers. In some implementations, changes made in the parameter portion 1320 are shown in the viewer 1370 (eg, updated in real time).

[0120] In some implementations, the design portion 1322 allows for receiving input from a designer for selecting among visual mark types. In some implementations, the visual mark types correspond to preset amounts of data encoded, such as 32 bytes, 128 bytes, 512 bytes, and the like. In some implementations, the visual mark types correspond to types based on light conditions, such as monochrome, low light, daylight for color, display, fluorescent, and the like. In some implementations, the visual mark types correspond to a specified number of colors (e.g., 2-8 for encoding data in a colored mark). In some implementations, combinations of visual mark types are selectable. As Fig.13 As shown, the visual marker type "Hex-4" can be selected from the drop-down list of available visual marker types.

[0121] like Fig.13 As shown, visual indicia 1400 includes a hexagonal shape, four colors (1435A, 1435B, 1435C, 1435D) for encoding data in the colored indicia, and a separate (eg, aesthetic) background color (1435E).

[0122] In some implementations, the selection image section 1325 allows receiving input from the designer for selecting an image to be used for designing visual marks or compositions. In some implementations, the image that the selection image section 1325 can select is stored locally, stored in a specified location, or can be accessed remotely. In some implementations, the image is selected by the selection image section 1325 using a browse dialog window function. In some implementations, the inferred color button in the selection image section 1325 causes the automatic generation of colors 1435A-1435E based on the input image. In some implementations, the inferred color button is an on-off toggle switch. In some implementations, the reload button in the selection image section 1325 reloads the selected current image from the selection image section 1325 and resets the colors 1435A-1435E to their original settings.

[0123] In some implementations, the coded color section 1330 allows for receiving input from a designer for automatically or semi-automatically selecting a plurality of colors 1435A, 1435B, 1435C, 1435D for encoding data in the colored indicia on the visual mark 1400. In some implementations, the coded color section 1330 includes a background color 1435E that is selected (e.g., manually, automatically) to ensure that the plurality of colors 1435A, 1435B, 1435C, 1435D remain distinguishable from one another, and therefore detectable, under different conditions (e.g., lighting conditions, printing conditions, display conditions, image sensor characteristics, or a combination thereof).

[0124] In some implementations, colors 1435A-1435E are automatically selected based on input image 1315 (e.g., the inferred color button in select image section 1325 is selected). In some implementations, the designer manually changes the selected colors of colors 1435A-1435E by selecting a new color. For example, the designer changes first color 1435A (e.g., using the 3D color space dialog window). In this example, colors 1435B-1435E are automatically modified to ensure their detectability (e.g., see Fig. 9 In some implementations, both the hue of the plurality of colors 1435A-1435E and the luminescence of the plurality of colors 1435A-1435E are evaluated to ensure detectability.

[0125] In some implementations, the designer manually changes the color of one or more of the colors 1435A-1435E by dragging a selected color, such as 1435A, over the input image 1315, and the selected color 1435A changes to match the underlying color at that location (e.g., pixel) of the input image 1315. In this example, when a new color is selected for color 1435A, colors 1435B-1435E are automatically modified to ensure their detectability (e.g., see Fig. 9 1435A-1435E). In some implementations, the designer selects a new color for one or more of the colors 1435A-1435E using other color selection techniques. In some implementations, the colors 1435A-1435E selected by the designer are evaluated (e.g., for aesthetic or detectable properties), and the proposed modified colors presented to the designer in the UI 1300 can be accepted with a single action by the designer.

[0126] In some implementations, the designer switches a color of the visual mark 1400 (e.g., the first color 1435A) with another color of the visual mark 1400 (e.g., the third color 1435C), and the visual mark is recoded accordingly. For example, the designer drags color 1435C over color 1435A until the colors switch in the coded color section 1330. In some implementations, any of the multiple colors in the designed visual mark (e.g., visual mark 1400) can be changed as described herein.

[0127] In some implementations, the position section 1340 allows for receiving input from the designer for positioning the visual marker in the viewer 1370 or over / on the input image. In some implementations, the position of the visual marker 1400 is entered using an X position and a Y position, and the visual marker 1400 is moved to that position over the input image 1315. In some implementations, the input X and Y positions of the visual marker 1400 can be increased or decreased using the increase / decrease section buttons 1342, 1344, respectively. Alternatively, the designer can select the visual marker 1400 (e.g., using a touch screen, a cursor, etc.) and drag the visual marker 1400 around in the viewer 1370 or over the input image 1315 to position the visual marker 1400 relative to the input image 1315. In some implementations, the visual marker can be positioned only so that the visual marker 1400 is fully visible in the viewer 1370 (e.g., not partially off-screen).

[0128] In some implementations, the size section 1345 allows receiving input from the designer for entering a size of the visual marker in the viewer 1370 or over / on the input image. In some implementations, the size of the visual marker 1400 is manually entered in the size section 1345, and the visual marker 1400 changes to that size over the input image 1315. In some implementations, the input size of the visual marker 1400 can be increased or decreased using the increase / decrease section buttons 1346. Alternatively, in some implementations, the designer can select the visual marker 1400 (e.g., using a touch screen, a cursor, etc.) and increase / decrease the size of the visual marker 1400 directly in the viewer 1370 or over the input image 1315. In some implementations, the visual marker can be sized only so that the visual marker 1400 is fully visible in the viewer 1370 (e.g., not partially off-screen).

[0129] In some implementations, the data entry section 1350 allows for receiving input from a designer for entering data to be encoded using colored markers into the visual mark 1400. In some implementations, the data entry section 1350 uses standard text, symbol, or graphic input techniques.

[0130] In some implementations, the save section 1355 receives input for manually saving, periodically saving, or saving changes to the composition currently being designed. In some implementations, the designer manually saves the composition currently being designed by selecting a "save" button in the save section 1355. In some implementations, the designer selects a resolution for saving the visual mark or composition currently being designed using the save section 1355. In some implementations, a "save changes" selection is used to save changes to the composition currently being designed. In some implementations, the "save changes" selection is an on-off toggle switch.

[0131] In some implementations, display composite section 1360 enables receiving input for alternating between displaying visual indicia 1400 or a composite that includes visual indicia 1400 with a selected input image 1315 in viewer 1370 .

[0132] In some implementations, the print section 1380 allows receiving input from the designer for printing a visual mark 1400 or a composite that includes the visual mark 1400 with the selected input image 1315 in the viewer 1370. In some implementations, the designer selects a resolution at which to print the visual mark or composition currently being designed using the print section 1380.

[0133] Fig.14 is a schematic diagram showing exemplary colored visual indicia according to some specific implementations. Fig.14 As shown, the visual marker 1400 includes a first portion 1410 for detection and orientation, a third portion 1420 that encodes data in the visual marker 1400 using a color mark, and a second portion 1445 that includes known locations (e.g., 4) within a specific overall shape of the visual marker 1400, each of which is colored with one of a plurality of colors (e.g., A, B, C, D) used in the encoded color mark of the third portion 1420 of the visual marker 1000.

[0134] like Fig.14 As shown, the first portion 1410 includes a preset shape for detection and orientation. In some implementations, the first portion 1410 is an outer area having a predefined shape that is consistent for multiple different visual markers (e.g., visual markers that use different colors to encode different data). Fig.14 As shown, the first portion 1410 is a thin hexagonal shape with a gap for orientation. In some implementations, additional portions of the visual marker 1400 can also be used for calibration or orientation. Fig.14As shown, the first portion 1410 includes three sections 1410A, 1410B, and 1410C. In some implementations, the three sections 1410A, 1410B, and 1410C are three shades of a single color. In some implementations, the three sections 1410A, 1410B, and 1410C are lighter gray, lighter gray, and gray, wherein gray is darker than light gray, and light gray is darker than lighter gray.

[0135] In some implementations, the second portion 1445 includes a known location or position within the first portion 1410 of the visual marker 1400 or a particular overall shape, each of which is colored in one of the data-encoded colors used in the visual marker 1400. Fig.14 As shown, the second portion 1445 has four positions, including a first position 1445A (e.g., in the first portion 1410), a second position 1445B, a third position 1445C, and a fourth position 1445D. Each of the four positions 1445A-1445D is colored with one of the A, B, C, or D colors used to encode data in the colored mark in the third portion 1420. In some implementations, the second portion 1445 may include a different number of positions corresponding to each of the colors selected for the visual mark. In some implementations, the four positions 1445A-1445D correspond to the plurality of colors, Fig. 9 The plurality of colors 950, or a subset of colors of box 1130 of method 1100.

[0136] In some implementations, the third portion 1420 uses color to encode the data of the visual marker 1400. Fig.14 As shown, the third portion 1420 encodes the data of the visual marker 1400 using at least shape and color. Fig.14 In the embodiment, the third portion 1420 is between the four positions of the first portion 1410 and the second portion 1445. Fig.14 As shown, the third portion 1420 includes a colored mark 1425, which includes a combination of 2 shapes (e.g., square, rectangle) and 4 colors of the second portion 1445. In some implementations, the third portion 1420 includes colored shapes 1425-A1, 1425-A2, ..., 1425A54 arranged in a clockwise or counterclockwise order by quadrant from the starting position. Fig.14 As shown, the third portion 1420 also includes a background location 1445E that uses a different color E than the colors used in the first portion 1410 and the second portion 1445. In some implementations, the color of the background location is selected for detectability of the color of the second portion 1445.

[0137] In some implementations, the visual marker 1400 includes a central region 1430. In some implementations, the central region 1430 is used for decoration (e.g., a company logo). In some implementations, the central region 1430 includes a symbol of a specific shape or color for detection, or a specific shape, size, or angle for orientation or correction of the visual marker 1400 (e.g., in a captured image of the physical environment). In some implementations, the second position 1445B, the third position 1445C, and the fourth position 1445D surround the central region 1430.

[0138] In some implementations, the single color of the three sections 1410A, 1410B, and 1410C is not used in the first portion 1410 or the third portion 1420. In some implementations, the three sections 1410A, 1410B, and 1410C are used to perform local white balancing of the colors in the visual marker 1400 when detected by the image sensor. In some implementations, the three sections 1410A, 1410B, and 1410C are used to detect spatially varying illumination at the detected visual marker 1400 or to correct for any detected spatially varying illumination. In some implementations, the spatially varying illumination at the detected visual marker 1400 is caused by a light source, non-uniform illumination, objects in the physical environment, and the like.

[0139] Fig.15 1 is a schematic diagram illustrating a UI 1300 configured to design another visual mark using color-coded data. In some implementations, the UI 1300 allows receiving input from a designer for determining a plurality of colors based on a source image of a visual mark that uses color-coded data (e.g., size, shape, color, or orientation of a graphical element). Fig.15 As shown, the selected composition 1505a includes parameters for the visual marker 1500 determined using the source image 1515 using the UI 1300. Fig.15 As shown, visual indicia 1500 includes a hexagonal shape, 1 color 1535A, two shapes of outlined or solid rectangles or squares for encoding data in the colored mark, and a separate background color 1535B.

[0140] Fig.15 15 is a schematic diagram showing an exemplary dialog window 1540 of the coded color section 1330 that allows manual selection of a color (e.g., 1535A) of the plurality of colors in the colored visual mark being designed. Fig.15 As shown, the dialog window 1540 allows manipulation of the red slider, the blue slider, and the green slider to determine the newly selected color for the color 1535A of the visual marker 1500.

[0141] In some implementations, a set of colors that are selectable for use in a visual mark and a measure of distance between a plurality of colors in the set of colors are determined. In some implementations, a subset of the set of colors is selected for use in the visual mark, the selection being based on the measure of distance between colors in the subset of colors. In some implementations, an appearance of the visual mark is generated based on the subset of colors, wherein the visual mark appearance includes a graphical element that encodes data using the subset of colors.

[0142] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0143] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0144] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0145] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0146] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0147] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0148] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, objects or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components or their groupings.

[0149] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0150] The foregoing specific embodiments and the content of the invention of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementation, but by the full breadth allowed by the patent law. It should be understood that the specific implementation shown and described herein is only an illustration of the principles of the present invention, and those skilled in the art can implement various modifications without departing from the scope and essence of the present invention.

Claims

1. A method, the method include: At an electronic device having a processor: receiving an input for selecting a source image; determining the color based on the source image and based on a distance between colors exceeding a spatial distance threshold in a 3D color space; as well as An appearance of a visual indicia is generated based on the determined color, the appearance of the visual indicia including a graphical element that encodes data using the determined color.

2. The method according to claim 1, wherein determining the color further comprises: include: Variations of the color for lighting conditions, printing device conditions, display device conditions, or based on camera calibration are identified.

3. The method according to claim 1, wherein determining the color further comprises: include: Variations of the color for multiple lighting conditions, multiple printing device conditions, or multiple display device conditions are identified. The method of claim 3 , wherein identifying variations of the color comprises identifying nearest neighbors of corresponding ones of the colors in the 3D color space.

5. The method according to claim 3, further comprising: include: Respective ones of the variations of the colors are clustered with corresponding ones of the colors to form one or more color clusters.

6. The method according to claim 5, further comprising: include: An alternative color for the visual indicia is determined using a color in each of the one or more color clusters.

7. The method of claim 1 , wherein the color is determined based on the distance between the colors exceeding the spatial distance threshold in the 3D color space. include: Determine the colors that have the largest gaps between them.

8. The method of claim 1, wherein determining the color further comprises: include: The color subset is selected based on determining that the colors of the color subset are distinct enough to be detected by a device that senses the printed or displayed visual indicia under various lighting conditions for the visual indicia.

9. The method of claim 1, wherein determining the color further comprises: include: displaying colors associated with the source image in an ordered sequence; receiving input selecting a first color from among said colors in said ordered sequence; as well as A second color is removed from the colors in the ordered sequence based on a measure of distance from the first color.

10. The method according to claim 9, further comprising: include: receiving input selecting a third color from among said colors in said ordered sequence; as well as A fourth color is removed from the colors in the ordered sequence based on a measure of distance from the third color. The method of claim 1 , wherein generating the appearance of the visual indicia comprises displaying the visual indicia.

12. A system, include: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the system to: receiving an input for selecting a source image; determining the color based on the source image and based on a distance between colors exceeding a spatial distance threshold in a 3D color space; as well as An appearance of a visual indicia is generated based on the determined color, the appearance of the visual indicia including a graphical element that encodes data using the determined color.

13. The system of claim 12, wherein the instructions, when executed, cause the system to: determine the color by identifying variations of the color for a plurality of lighting conditions, a plurality of printing device conditions, or a plurality of display device conditions.

14. The system of claim 13, wherein the instructions, when executed, cause the system to: identify nearest neighbors of corresponding ones of the colors in the 3D color space.

15. The system of claim 13, wherein the instructions, when executed, cause the system to: Respective ones of the variations of the colors are clustered with corresponding ones of the colors to form one or more color clusters.

16. The system of claim 15, wherein the instructions, when executed, cause the system to: An alternative color for the visual indicia is determined using a color in each of the one or more color clusters.

17. The system of claim 12, wherein the instructions, when executed, cause the system to: determine the colors based on the distance between the colors exceeding the spatial distance threshold in the 3D color space by determining the colors having the largest separation therebetween.

18. A system according to claim 12, wherein the instructions, when executed, cause the system to: determine the color by selecting the color subset based on determining that the colors of the color subset are distinct enough to be detected by a device that senses the printed or displayed visual mark under various lighting conditions of the visual mark.

19. The system of claim 12, wherein the instructions, when executed, cause the system to: displaying colors associated with the source image in an ordered sequence; receiving input selecting a first color from among the colors in the ordered sequence; and A second color is removed from the colors in the ordered sequence based on a measure of distance from the first color.

20. A non-transitory computer-readable storage medium storing program instructions executable on a computer to perform operations including: On an electronic device having a processor: receiving an input for selecting a source image; determining the color based on the source image and based on a distance between colors exceeding a spatial distance threshold in a 3D color space; and An appearance of a visual indicia is generated based on the determined color, the appearance of the visual indicia including a graphical element that encodes data using the determined color.

21. A method, include: At an electronic device having a processor: determining, based on distances between colors in the set of colors exceeding a spatial distance threshold in a 3D color space, the set of colors to be selected for use in a visual marker; and An appearance of the visual indicia is generated based on the set of colors, wherein the visual indicia appearance includes a graphical element that encodes data using the subset of colors.

22. The method according to claim 21, further comprising: include: Variations of the set of colors for lighting conditions, printing device conditions, display device conditions, or based on camera calibration are identified.

23. The method according to claim 21, further comprising: include: Variations of the set of colors for a plurality of lighting conditions, a plurality of printing device conditions, or a plurality of display device conditions are identified.

24. The method of claim 23, wherein identifying variations of the set of colors comprises identifying nearest neighbors of corresponding colors in the set of colors in the 3D color space.

25. The method according to claim 23, further comprising: include: Corresponding ones of the variations of the set of colors are clustered with corresponding colors of the set of colors to form one or more color clusters.

26. The method according to claim 25, further comprising: include: An alternative color for the visual indicia is determined using a color in each of the one or more color clusters.

27. The method of claim 21, wherein determining the set of colors include: Determine the colors that have the largest gaps between them.

28. The method of claim 21, wherein determining the set of colors further comprises: include: The color subset is selected based on determining that the colors of the color subset are distinct enough to be detected by a device that senses the printed or displayed visual indicia under various lighting conditions for the visual indicia.

29. The method of claim 21, wherein generating the appearance of the visual indicia comprises displaying the visual indicia.

30. A system, include: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the system to perform a method according to any one of claims 21-29.

31. A non-transitory computer-readable storage medium storing program instructions executable on a computer to perform the method according to any one of claims 21-29.

32. A method, include: At an electronic device having a processor: generating a first set of colors that can be selected for use in the visual mark; selecting a first color from the first set of colors; generating a second set of colors by eliminating the color subset from the first set of colors based on the color subset being within a spatial distance threshold from the first color in the 3D color space; selecting a second color from the second set of colors; An appearance of the visual mark is generated based on the first color and the second color, wherein the appearance of the visual mark includes a graphical element that encodes data using the first color and the second color.

33. The method according to claim 32, further comprising: include: switching the second color to a third color; as well as The data is re-encoded using the first color and the third color.

34. The method according to claim 32, further comprising: include: Variations of the first color and the second color for lighting conditions, printing device conditions, display device conditions, or based on camera calibration are identified.

35. The method according to claim 32, further comprising: include: Variations of the first color and the second color are identified for a plurality of lighting conditions, a plurality of printing device conditions, or a plurality of display device conditions.

36. The method of claim 35, wherein identifying variations of the first color and the second color comprises identifying nearest neighbors of the first color and the second color in the 3D color space.

37. The method according to claim 35, further comprising: include: Corresponding ones of the variations of the first color and the second color are clustered with corresponding ones of the first color and the second color to form one or more color clusters.

38. The method according to claim 37, further comprising: include: An alternative color for the visual indicia is determined using a color in each of the one or more color clusters.

39. The method of claim 32, wherein generating the appearance of the visual indicia comprises displaying the visual indicia.

40. A system, include: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the system to perform a method according to any one of claims 32-39.

41. A non-transitory computer-readable storage medium storing program instructions executable on a computer to perform the method according to any one of claims 32-39.

42. A method, include: At an electronic device having a processor: detecting the visual marker based on identifying a first portion of the visual marker; determining a set of colors for encoding data of the visual indicia, wherein the determining is based on identifying a second portion of the visual indicia; as well as Elements in a third portion of the visual indicia are decoded using the determined set of colors.

43. The method according to claim 42, further comprising: include: determining a set of shapes for encoding data from the visual marker based on identifying the second portion of the visual marker; Wherein decoding the element in the third portion of the visual indicia further comprises using the determined set of shapes.

44. A system, include: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the system to perform a method according to any one of claims 42-43.

45. A non-transitory computer-readable storage medium storing program instructions executable on a computer to perform the method according to any one of claims 42-43.

46. ​​A method, include: At an electronic device having a processor: Generate user interfaces for designing visual markers that utilize color-coded data; receiving, via the user interface, input selecting a first color from a first set of colors; generating a second set of colors by eliminating the color subset from the first set of colors based on the color subset being within a spatial distance threshold from the first color in the 3D color space; receiving, via the user interface, input selecting a second color from the second set of colors; receiving, via the user interface, the data to be encoded; as well as An appearance of the visual mark is generated based on the first color and the second color, wherein the appearance of the visual mark includes a graphical element that encodes the data using the first color and the second color.

47. The method according to claim 46, further comprising: include: receiving, via the user interface, an input for switching the second color to a third color; as well as The data is re-encoded using the first color and the third color.

48. The method according to claim 46, further comprising: include: Variations of the first color and the second color for lighting conditions, printing device conditions, display device conditions, or based on camera calibration are identified.

49. The method according to claim 46, further comprising: include: Variations of the first color and the second color are identified for a plurality of lighting conditions, a plurality of printing device conditions, or a plurality of display device conditions.

50. The method of claim 49, wherein identifying variations of the first color and the second color comprises identifying nearest neighbors of the first color and the second color in the 3D color space.

51. The method according to claim 49, further comprising: include: Corresponding ones of the variations of the first color and the second color are clustered with corresponding ones of the first color and the second color to form one or more color clusters.

52. The method according to claim 51, further comprising: include: An alternative color for the visual indicia is determined using a color from each of the one or more color clusters.

53. The method of claim 46, wherein generating the appearance of the visual indicia comprises displaying the visual indicia.

54. A system, include: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the system to perform a method according to any one of claims 46-53.

55. A non-transitory computer-readable storage medium storing program instructions executable on a computer to perform the method according to any one of claims 46-53.