Methods, systems, and apparatus for measuring surface color

Abstract

A method (300) and a system (100) are provided for determining the surface color of a target surface (102) in an environment (104) with an ambient light source (106). Multiple images (105) of the target surface (102) are acquired while it is illuminated with a light source (118) of variable intensity and constant color and an ambient light source (106) of constant intensity and constant color, wherein the intensity of the light source (118) on the target surface (102) varies with known quantities between the acquisition of the images (105). A color feature tensor independent of the ambient light source (106) is extracted from the image data, and the surface color of the target surface (102) is inferred using the color feature tensor.

Description

[0001] Cross-application

[0002] This application claims the benefit of U.S. Nonprovisional Application No. 16 / 953,029, filed November 19, 2020, entitled “Method, system, and device for color measurement of a surface,” the contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to color measurement, and more particularly to a method and system for measuring surface color using a color measuring device. Background Technology

[0004] The purpose of color measurement is to provide a reliable estimate of the true color value of a given surface in a format defined by a specific color space. However, current color measurement methods typically require specialized equipment, such as colorimeters, which are not readily available in everyday life.

[0005] Despite advancements in image acquisition technology and its ease of use, the sensitivity to ambient lighting conditions still prevents mobile phones from being a suitable choice for color measurement. Specifically, color information obtained directly from digital images captured by a mobile phone cannot provide reliable results because the colors in a photograph can be strongly influenced by ambient lighting conditions. Currently available color correction operations on mobile phones, including white balance and human visual observation, offer no meaningful assistance. Therefore, depending on the ambient lighting conditions, the color of a target surface shown in a digital image captured by a mobile phone may differ significantly from the true color of the target surface.

[0006] Attempts have been made to use mobile phones as color measurement devices. However, this requires additional equipment. Specifically, since controlling ambient lighting is crucial for current color measurement methods, color measurement using mobile phones typically requires covering at least a portion of the target surface with, for example, an opaque cup-shaped object to create an absolute dark cavity on the surface, within which color measurements are performed. This absolute dark cavity eliminates ambient lighting, allowing control over illumination on the target surface. However, creating such an absolute dark cavity on the target surface is not only inconvenient but also costly, as it may require additional equipment. Furthermore, irregular shapes on the target surface may be difficult to cover, making accurate color measurements impossible. Summary of the Invention

[0007] This invention provides a color measuring device that can be used as a colorimeter to accurately measure the color of a surface. The color measuring device and the method performed by the color measuring device of this invention eliminate the disadvantages of existing methods that use a mobile phone as a colorimeter to measure surface color.

[0008] According to one aspect, the present invention provides a method for measuring the color of a target surface. The method includes acquiring multiple images of the target surface while illuminating it with a light source of variable intensity and constant color and an ambient light source of constant intensity and constant color, wherein the light source intensity on the target surface varies with known quantities between the acquisition of the images. The method further includes determining a color feature tensor corresponding to the product of the color of the light source and the surface color of the target surface based on image data from the multiple images, and inferring the surface color of the target surface based on the color feature tensor.

[0009] Based on the above, the intensity of light illuminating the target surface can be changed by altering the distance between the light source and the target surface.

[0010] According to any of the above, inferring the surface color of a target surface from a color feature tensor may include inputting the color feature tensor into a training model for processing the color feature tensor, and then inferring the surface color of the target surface from the processed color feature tensor.

[0011] According to any of the above, the trained model is a linear model. The linear model is trained as follows: the weights of the weight matrix are initialized; a batch of training samples is received, each training sample in the training dataset comprising an image of the surface and a ground truth surface color tensor representing the true color of the surface. For each corresponding sample in this batch of training samples, the image of the corresponding training sample is processed to generate a color feature tensor corresponding to the product of the training color of the light source and the ground truth surface color; the surface color tensor is computed as the product of the color feature tensor and the weight matrix, and an error value is determined as the difference between the surface color tensor and the ground truth surface color tensor; finally, the weights of the weight matrix are updated based on the error value. The linear model is then trained by receiving more batches of training samples until the weights of the weight matrix are optimized.

[0012] According to any of the above aspects, the trained model approximates a neural network. The neural network can be trained as follows: Initialize the weights of the neural network; receive a batch of training samples from a training dataset, each training sample including an image of the surface and the ground truth surface color of the surface; continue training the neural network by: for each corresponding training sample, processing the image of the training sample to generate a color feature tensor corresponding to the product of the light emitted by the light source and the ground truth surface color of the surface; forward propagating the generated color feature tensor for the corresponding training sample through the neural network to infer the surface color of the training sample; calculating the error value as the difference between the inferred surface color and the ground truth surface color; and performing backpropagation to update the weights of the neural network based on the error value.

[0013] According to any of the foregoing aspects, the acquisition may include detecting the distance between the color measuring device and the target surface at a first time using a time-of-flight sensor, and detecting the distance between the color measuring device and the target surface at a second time using a time-of-flight sensor. When the distance between the color measuring device and the target surface changes, a light source is controlled to emit light with a constant color and constant intensity, thereby illuminating the target surface and acquiring digital frames of the target surface.

[0014] According to any of the above, the intensity of light illuminating the target surface can be changed by altering the amount of electricity supplied to the light source at a constant distance from the target surface.

[0015] According to any of the above aspects, the inference may further include converting the inferred surface color of a surface in the first color space into a surface color in the second color space.

[0016] According to any of the above, the training model can be trained by dividing the color space into multiple color subspaces, initializing the weights of the subspace separation model and the weights of the color subspace model for each of the multiple color subspaces, and receiving training samples including an image of the surface and the ground truth surface color of the surface. Alternatively, the training model can be trained by processing the image of the training samples to generate a color feature tensor corresponding to the product of the light emitted by the light source and the ground truth surface color, using the received color feature tensor and calculating the subspace color tensor through each color subspace model, generating a subspace weight tensor through the subspace separation model, and inferring the surface color by applying the subspace weight tensor to the subspace color tensor. Alternatively, the training model can be trained by determining the error value as the difference between the inferred surface color and the ground truth surface color, and performing backpropagation to update the weights of each color subspace model and the weights of the subspace separation model.

[0017] Based on any of the above, color spaces can be divided manually.

[0018] According to any of the above, the training of multiple color subspace models and subspace separation models is carried out simultaneously.

[0019] According to any of the foregoing, the determination may include generating multiple linear equations based on image data of each of a plurality of images, wherein the image data in the images is the sum of the product of the color of the light source and the surface color of the target surface and the product of the ambient light source and the surface color of the target surface, and determining a color feature tensor using linear regression and through multiple linear equations.

[0020] In another aspect, the present invention provides a color measurement device for measuring the surface color of a target surface illuminated by a constant-intensity, constant-color ambient light source. The color measurement device includes an image acquisition device for acquiring multiple images of the target surface and a light source for illuminating the target surface with light of a constant color. The intensity of the light emitted from the light source varies depending on a known quantity between the acquisition of consecutive images and the color measurement system. The color measurement system is used to determine a color feature tensor corresponding to the product of the color of the light emitted by the constant light source and the surface color of the target surface, based on image data included in the multiple images, and to infer the surface color of the target surface based on the color feature tensor.

[0021] In another aspect, the present invention provides a computer-readable medium having instructions tangibly stored thereon. When executed by a processing unit, the instructions cause the processing unit to: acquire multiple digital images of a target surface using a camera of a color measuring device, since the target surface is illuminated by constant-color light emitted by a controlled light source of the color measuring device, which is positioned relative to the camera, and by an ambient light source of constant intensity and constant color, wherein the intensity of the light illuminating the target surface varies with known quantities between the acquisition of the images; determine a color feature tensor corresponding to the product of the color of the light emitted by the constant light source and the surface color of the target surface, based on image data included in the multiple images; and infer the surface color of the target surface based on the color feature tensor.

[0022] At least some of the above aspects can advantageously enable robust and reliable accurate color measurement under the influence of ambient lighting conditions, thereby enabling devices such as mobile phones to be used as reliable color measurement devices. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating a color measuring device for determining the color of a target surface in an environment, according to an exemplary embodiment of the present invention.

[0024] Figure 2 It shows Figure 1A simplified block diagram of a color measurement device.

[0025] Figure 3 An exemplary embodiment of the present invention is shown, by Figure 2 A flowchart of the color measurement method performed by the color measurement system 120;

[0026] Figure 4 A flowchart illustrating a method for inferring the color of a target surface using a trained linear model is shown.

[0027] Figure 5 A flowchart is shown for a method to infer values ​​of a target surface using multiple color subspace models.

[0028] Similar reference numerals can be used to denote similar components in different accompanying drawings. Detailed Implementation

[0029] This invention is illustrated with reference to the accompanying drawings, in which embodiments are shown. However, many different embodiments may be used, and therefore the description should not be construed as limiting to the embodiments set forth herein. Rather, these embodiments are provided to make the invention thorough and complete. Throughout the specification, similar numerals refer to similar elements. The separate blocks or separations of functional elements or modules of the illustrated systems and devices do not necessarily require physical separation of these functions or modules, as communication between these elements can occur without any such physical separation via message passing, function calls, shared memory spaces, etc. Therefore, although functions or modules are shown separately herein for ease of explanation, these functions or modules do not need to be implemented in physically or logically separate platforms. Different devices may have different designs such that while some devices implement some functions in fixed-function hardware, others may implement those functions in a programmable processor with code available from a machine-readable medium.

[0030] Figure 1 An example of a color measuring device 100 according to the present invention is shown, the color measuring device 100 being used to determine the color of a target surface 102 within an environment 104, the environment 104 including an ambient light source 106.

[0031] The target surface 102 can be a flat or irregularly shaped surface, and its reflection depends on the specific bandwidth of the visible light spectrum of the surface color or the true color of the surface to be measured.

[0032] The environment 104 surrounding the target surface 102 includes one or more light sources, collectively referred to as ambient light source 106, for illuminating the target surface 102. As a non-limiting example, ambient light source 106 may include any combination of natural light and artificial lighting such as fluorescent light, cold light, and gas discharge power sources. Ambient light source 106 has a constant intensity and a constant color, such that the illumination of the target surface 102 by ambient light source 106 remains substantially constant during the color determination process, as detailed below.

[0033] Color measuring device 100 is used to infer (i.e., predict) the surface color of target surface 102 by processing image 105 (also referred to as a frame), said image 105 comprising pixel blocks of pixel values ​​107 acquired by a camera on target surface 102. Color measuring device 100 may be as follows: Figure 1 The smartphone shown is a smartphone or any other computing device that includes a camera, or can connect to and communicate with an external camera, including personal laptops or desktop computers, or other mobile computing devices such as tablets, personal digital assistants (PDAs), and digital single-lens reflex (DSLR) cameras.

[0034] See Figure 2 A simplified block diagram of a color measuring device 100 is shown. While exemplary embodiments of the color measuring device 100 are shown and discussed below, other embodiments may be used to implement the examples disclosed herein, which may include components different from those shown. Figure 2 A single instance of each component of the color measuring device 100 is shown, but each component shown may have multiple instances.

[0035] Color measurement device 100 includes one or more processors, collectively referred to as processor 108, such as a central processing unit, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), dedicated logic circuit, tensor processing unit, neural processing unit, dedicated artificial intelligence processing unit, or a combination thereof. Processor 108 controls the overall operation of color measurement device 106. Processor 108 is coupled to multiple components via a communication bus (not shown), which provides a communication path between the components and processor 108. Processor 108 is used to control various component devices, such as one or more non-transient memory 110, battery 112, image acquisition device in the form of camera 114, time-of-flight (ToF) sensor 116, etc. Processor 108 also supports the execution of subroutines in color measurement device 100 for manipulating data. For example, processor 108 can be used to manipulate image data of image 105 captured by camera 114. These operations may include comparison, cropping, compression, color adjustment, and brightness adjustment, etc.

[0036] A bus 117 may be present for providing communication between components of the color measurement device 100, including a processor 108, a battery interface 113, a camera 114, one or more optional I / O interfaces 115, one or more network interfaces 119, and one or more memories 110. Bus 117 can be any suitable bus architecture, including, for example, a memory bus, a peripheral bus, or a video bus.

[0037] The color measurement device 100 may include one or more optional network interfaces 109 for wired or wireless communication with a network (e.g., intranet, internet, P2P network, WAN and / or LAN) or other nodes. The network interface 109 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intranet and / or extranet communication.

[0038] Color measurement device 100 may also include one or more optional input / output (I / O) interfaces 115, which may allow connection to one or more optional input devices 115a and / or optional output devices 115b. In the example shown, input devices 115a (e.g., lighting devices, mice, microphones, touchscreens, and / or keyboards) and output devices 208 (e.g., displays, speakers, and / or printers) are shown as optional and built into the color measurement device. In other examples, one or more input devices 115a and / or output devices 115b may be externally connected to color measurement device 100. In other examples, there may be no input devices 115a and output devices 115b, in which case I / O interface 115 may not be required.

[0039] One or more non-transitory memories 110 may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The non-transitory memories 110 may store instructions executed by the processor 108, such as those described in this invention. The memories 110 may include other software instructions, such as those for implementing an operating system and other applications / functions.

[0040] In one embodiment, memory 110 stores machine-readable software instructions executable by processor 108 for implementing color measurement system 120 to process images acquired by camera 114 to isolate pixel blocks of pixel values, thereby determining the color tensor C of the pixel blocks, and inferring (i.e., predicting) the surface color of target surface 102 based on the color tensor C, as detailed below. Color measurement system 120 may include image processing module 122 and color processing module 124. As used herein, "module" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry. Hardware processing circuitry may include a microprocessor, the core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a system-on-a-chip (SoC) or any combination of other hardware processing circuitry. In some alternative embodiments, the color measurement system 120, including image processing module 122 and color processing module 124, can be implemented as a single hardware device. For example, the hardware device is designed to process images acquired by camera 114 to isolate pixel blocks of pixel values, thereby determining the color tensor C of the pixel blocks, and inferring (i.e., predicting) the surface color of the target surface 102 based on the color tensor C, as detailed below. The single hardware device includes electronic circuitry that performs the functions of image processing module 122 and color processing module 124. In other exemplary embodiments, system 120 can be implemented as multiple hardware devices (e.g., multiple ASICs, FPGAs, and / or SoCs). Each hardware device includes electronic circuitry that performs the functions of one of image processing module 122 and color processing module 124, details of which will be detailed below. It should be understood that image processing module 122 and color processing module 124 are not necessarily separate units of system 120, and the illustration of image processing module 122 and color processing module 124 as separate blocks within system 120 may merely be a conceptual representation of the overall operation of system 120.

[0041] In some embodiments, the image processing module 122 is used to process the image 105 acquired by the camera 114, which may include cropping pixel blocks of image pixels 107 from the acquired image 105 and determining the color tensor C of the pixel blocks 107. The color processing module 124 is used to predict or infer the color of the target surface using the color tensor C from the image processing module 122 through a trained model.

[0042] In some embodiments, the color processing module 124 can be a machine learning-based software module used to implement a training model that uses acquired image data to predict the surface color of a target surface, as detailed below. In this invention, the surface color of the target surface refers to the color of light reflected from the target surface under a white light source. In other words, the surface color refers to the true color of the surface and is not affected by any color difference from the ambient light source.

[0043] In some examples, the color measuring device 100 may also include one or more electronic storage units (not shown), such as solid-state drives, hard disk drives, disk drives, and / or optical disk drives. In some examples, one or more datasets and / or modules may be provided by external memory (e.g., an external drive that is wired or wirelessly connected to the color measuring device 100), or may be provided by transient or non-transient computer-readable media. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage. For example, components of the color measuring device 100 may communicate with each other via a bus.

[0044] Battery 112 serves as a power source and can be either rechargeable or non-rechargeable. Battery 112 powers at least some components of the color measuring device 100. Battery interface 113 provides mechanical and electrical connection to battery 112. Battery interface 113 may be coupled to a regulator (not shown) that provides power V+ to the circuitry of the color measuring device 100. In some embodiments, battery 112 is a high-capacity, non-rechargeable, sealed battery expected to have a relatively long lifespan, such as 5 to 7 years. In some embodiments, color measuring device 100 may also include a power interface (not shown), such as a power port, for connection to an external power source (not shown), such as an alternating current (AC) power adapter. Color measuring device 100 may utilize an external power source instead of battery 112. If battery 112 is a rechargeable battery, the external power source may be used to charge battery 112.

[0045] The color measurement device 100 is capable of detecting certain aspects of the environment 104. In some embodiments, the color measurement device 100 includes one or more time-of-flight (ToF) sensors 116 for measuring the distance from the device 100 to the surface directly in front of the ToF sensor 116 by measuring the travel time of an infrared light signal emitted by the ToF sensor 116 as it deflects back from the surface of an object. The color measurement device 100 may also include other types of sensors, including light sensors, temperature sensors, pressure sensors, humidity sensors, gyroscopes, accelerometers, and door contact switches (not shown).

[0046] Digital camera 114 is capable of capturing light reflected from an object in the form of digital images (commonly referred to as frames), each frame including multiple pixel values ​​indicating color values. In some embodiments, where camera 114 is built into a color measurement device, digital camera 114 can be operatively coupled to processor 108 via bus 117, and frames can be transmitted via bus 117. In other embodiments, camera 114 can be externally connected to color measurement device 100. In these embodiments, camera 114 can be operatively coupled to processor 108 via I / O interface 115, wherein frames captured by digital camera 114 are received by I / O interface 115 and provided to processor 108 for processing via bus 117. Digital camera 114 can use a combination of a photoelectric sensor based on a charge-coupled device (CCD) or a metal-oxide-semiconductor (MOS) based device with a Bayer color filter array (CFA) to acquire color image data. Generally, cameras integrated into small consumer products (such as mobile devices or handheld devices) typically use CMOS photoelectric sensors, which are generally cheaper and consume less power in battery-powered devices compared to CCDs, whose cost may be a concern. CCD photoelectric sensors, on the other hand, are typically used in high-end broadcast-quality cameras. Photoelectric sensors are capable of detecting the number of visible photons arriving at each sensor and generating a corresponding charge indicating the number of visible photons arriving at each sensor. The information acquired by each photoelectric sensor is defined as a pixel in the output. In some other embodiments, camera 114 is externally connected to color measurement device 100, and the image 105 acquired by camera 114 can be coupled to processor 108 via a network.

[0047] Typically, Bayer CFAs are placed in front of photosensitive sensors to provide color sensitivity because photosensitive sensors are designed to be sensitive to light, not color. Specifically, Bayer CFAs employ a so-called "Bayer array," a checkerboard arrangement of red, green, and blue filters on a square grid of photosensitive sensors, enabling digital cameras to capture color information. Bayer CFAs typically have one color filter element in front of each photosensitive sensor. Each filter element acts as a bandpass filter for the incident light, passing through different bands of the visible light spectrum. Typically, half of the color filters are green, with the remainder evenly distributed between red and blue to mimic the greater resolving power of human vision for green light. Each Bayer array includes a 2×2 color filter block that repeats throughout the CFA. The filter blocks vary. Examples of filter blocks can include blue-green-green-red (BGGR), RGBG, GRGB, or RGGB. The output of a Bayer filter camera is a Bayer array image containing RGB signals. Without any processing of the signal, the Bayer array image is called the RAW format.

[0048] Because each photodetector is filtered to record only the brightness value of one of the three colors, the pixel value of each photodetector cannot individually and completely specify each of the red, green, and blue values. To obtain a full-color image, various deBayer (or commonly referred to as demosaic) algorithms can be applied by the raw converter (not shown) to insert a complete set of red, green, and blue values ​​for each pixel. These algorithms use pixels surrounding the corresponding color to estimate the value of a particular pixel. Many other demosaic algorithms can also be applied, including nearest-neighbor interpolation demosaic algorithms that can replicate neighboring pixels of the same color channel. Another example of a demosaic algorithm is bilinear interpolation, where the red value of a non-red value is calculated as the average of two or four neighboring red values; similar calculations can be performed for blue and green. Other demosaic algorithms include Variable Number of Gradients (VNG), Pixel Grouping (PPG), Adaptive Homogeneity-Directed (AHD), and Aliasing Minimization and Zipper Elimination (AMaZE), among others. Since color information is represented using the three primary colors of red, green, and blue, sensor data is referred to as being in the RGB color space, which defines the mathematical relationship between the wavelength distribution in the electromagnetic visible spectrum and the physiologically perceived colors in human color vision.

[0049] The image acquisition device also includes a constant color light source 118 (hereinafter referred to as light source 118). For the camera 114, light source 118 can be a flash unit. Light source 118 can be directly integrated into the camera 114, such as... Figure 2 As shown. However, it should be understood that some cameras support separate mounting of the light source via a standardized "accessory mount" bracket (not shown). In other setups, the flash unit may be a large stand-alone unit or studio flash connected to the color measurement device 100. While the color of the light source 118 is constant, the processor 108 can vary the intensity of the light source 118, as detailed below.

[0050] Figure 3 This is a flowchart of a method 300 for performing color measurement of a target surface using a color measuring device, executed by the color measuring system 120 of the present invention, according to an exemplary embodiment of the present invention. Method 300 can be implemented in software executed by processor 108. Method 300 may include more or fewer actions than shown and described, and may be performed in different orders. When executed by the processor 108 of the color measuring device 100, computer-readable code or instructions of the software implementing method 300 may be stored in memory 110 or a computer-readable medium.

[0051] In step 310, the color measurement device 100 is used to acquire multiple images 105 of the target surface 102 using the camera 114, since the target surface 102 is illuminated by light emitted from the light source 118 and ambient light, wherein each frame includes a pixel block with pixel value 107. In some embodiments, the acquired digital image of the target surface 102 is the output of the camera 114 in the Red-Green-Blue (RGB) color space, wherein each color pixel value includes red, green, and blue components.

[0052] During the acquisition of each of a plurality of images (i.e., frames) by camera 114, color measurement device 100 illuminates target surface 102 with light emitted by light source 118. The light emitted by light source 118 may have a constant color and a constant intensity. Between the acquisition of each consecutive image 105, the intensity of the light illuminating target surface 102 varies due to a known quantity. The intensity of the light illuminating target surface 102 may be changed, and the change in light intensity may be represented by an intensity factor α. In some embodiments, the intensity of the light illuminating target surface 102 may be changed by changing the amount of electricity supplied to light source 118 or by changing the distance between light source 118 and target surface 102.

[0053] In some embodiments, the size of the environment 104 may be limited, thereby restricting the movement of the color measuring device 100. For example, the target surface 102 may be located in a confined space where the movement of the color measuring device 100, as detailed below, would be restricted, making it impossible to change the light intensity. In such embodiments, the intensity of light emitted from the light source 118 can be changed by altering the power supply to the light source 118. For example, the light source 118 may be a light source capable of emitting light of varying intensities and constant colors, such as a light-emitting diode (LED) (not shown), wherein the intensity of light emitted by the light source 118 is changed by adjusting the power supplied to the light source 118 using known means. The amount of light emitted by the light source 118 is determined by the power supply voltage or current supplied to the light source 118, which varies depending on a known quantity supplied by the processor 108 via the battery interface 113. It should be understood that, in terms of accuracy, changes in light intensity achieved by changing the voltage or current via the processor 108 are superior to changes in light intensity achieved by manually altering the distance between the color measuring device 100 and the target surface. Therefore, the light intensity variation achieved by changing the power supply to the light source 118 can provide a more precise intensity variation of the light emitted by the light source 118, thereby enabling more accurate color prediction.

[0054] Based on the specific characteristics of the light source 118, the image processing module 122 can determine the amount of change in the light intensity emitted by the light source 118 corresponding to a change in power supply. The camera 114 is used to acquire multiple images 105 of the target surface 102, wherein the intensity of the light emitted by the light source 118 varies between the acquisition of each consecutive image 105. Each acquired image 105 can be associated with a power value acquired in data storage. In such embodiments, the distance and angle between the color measuring device 100 and the target surface 102 remain approximately the same, while the intensity of the light emitted by the light source 118 varies. In some embodiments, a graphical user interface (GUI) display can be used to instruct the user to keep the color measuring device 100 stationary during the acquisition of digital images (i.e., frames) by the camera 114. In other embodiments, a bracket or mounting device can be used to fix the color measuring device 100 stationary relative to the target surface 102.

[0055] In other embodiments, the intensity of light emitted from the light source 118 can also be changed by altering the distance between the color measuring device 100 and the target surface 102. Starting from an initial distance from the target surface 102, the color measuring device 100 can be continuously moved closer to or further away from the target surface 102 at a relatively constant angle. The user can also manually move the color measuring device 100. In some embodiments, a graphical user interface (GUI) can be used to instruct the user to continuously move the color measuring device 100 closer to or further away from the target surface 102 while maintaining a constant angle between the color measuring device 100 and the target surface 102. It should be understood that the color measuring device 100 can also be moved by an automated device, such as by fixing it to a mechanized mounting. Other methods, such as voice commands, can also be used to change the distance and angle of the color measuring device 100. A ToF sensor 116 can be used to detect the travel time of the infrared signal emitted from the color measuring device 100 and reflected back from the target surface 102, thereby determining the change in distance between the color measuring device 100 and the target surface 102. Other suitable distance measurement methods can also be implemented, including binocular ranging if the color measuring device 100 has two or more cameras on the side.

[0056] As the color measuring device 100 moves relative to the target surface 102, the light source 118 emits light of constant color and constant intensity to illuminate the target surface 102 at different distances. The light source 118 can be used to emit light illuminating the target surface 102 at fixed time intervals or at different distances from the target surface 102. Whenever the target surface 102 is illuminated by light emitted from the light source 118, the camera 114 can be used to acquire one or more images 105 of the target surface 102.

[0057] In step 320, the image processing module 122 determines the color feature tensor of image 105 and its corresponding intensity variation factor α by removing the component of light reflected from the target surface 102 by ambient light source 106 from the color tensor extracted from image 105. The color feature tensor can be expressed as the product of light reflected from the target source by light source 118 of color measurement device 100 and the surface color of target surface 102.

[0058] In some embodiments, image 105 is received by image processing module 122, which then determines a color tensor C representing the pixel values ​​in each image 105, indicating the light reflected from the target surface 102 by all light sources (106 and 118). For example, color tensor C could be the average pixel value of all pixel values ​​in image 105. Other suitable methods for determining color tensor C can be used, such as a weighted average of pixel values ​​in each image 105, modal values, medians, or a histogram of the RGB values ​​of image 105.

[0059] In other embodiments, it is not necessary to process all pixels of image 105; the color tensor C can be derived from pixel blocks of pixel values ​​in image 105. This improves processing efficiency while reducing computational resources used. For example, image processing module 122 can separate or crop pixel blocks 107 representing the pixel values ​​of the target surface color from the acquired image 105 and determine the color tensor C as described above based on the pixel values ​​contained in pixel blocks 107. In some embodiments, pixel block 107 may be the center pixel block of image 105. It should be understood that pixel block 107 can be obtained from any part of image 105 such that the pixel values ​​of pixel block 107 substantially and uniformly represent the color of target surface 102. Pixel block 107 may be selected by the user, or optionally, the pixel block may be a fixed portion of image 105 (e.g., the center pixel block). In some embodiments, the size of each pixel block 107 is 120 × 120 pixels. It should be understood that camera 114 can acquire image 105 at any suitable resolution, and therefore, the size of pixel block 107 can be adjusted accordingly. Furthermore, the pixel block size can be adjusted based on factors including processing time, available computing resources, and the size of the target surface 102. The image processing module 122 can then determine the color tensor C representing the pixel values ​​in the pixel block 107. In some embodiments, the color tensor C can be determined in a color space consistent with the color space of the acquired image 105.

[0060] The color tensor C can be represented as the product of the light from all light sources illuminating the target surface 102 and the surface color of the target surface 102, as shown by the following mathematical equation (1):

[0061] Equation (1) is C = L × R.

[0062] Where L is the light reflected from the target surface 102, and R is the surface color of the target surface 102.

[0063] During image acquisition 105, light emitted by the light source 118 of the color measurement device 100 and the ambient light source 106 illuminates the target surface 102. Therefore, according to the above equation (1), the light L reflected from the target surface 102 has two components, namely, light emitted from the light source 118 or L.control The emitted light, and the light from the ambient light source 106 or L env The emitted light. Therefore, equation (1) can be extended to equation (2), as follows:

[0064] C = L × R

[0065] C=(L control +L env )×R

[0066] C = L control ×R+L env ×R equation (2)

[0067] As can be seen from equation (2), since the ambient light source 106 has a constant intensity and a constant color, term L appears in all frames 105 captured by camera 114. env ×R remains constant. Therefore, any change in the intensity of the light emitted by light source 118 (as shown by α) can be proportionally reflected in the color tensor C. In a non-limiting example where the intensity of the light emitted by light source 118 decays quadratically as a function of the distance traveled by the light emitted by light source 118, the change in light intensity α can be expressed as... Where d is the distance between the light source 118 and the target surface 102. For example, the image processing module 122 can obtain the value of d through the ToF sensor 116 and process the change in light intensity. This applies to the intensity of light emitted from light source 118. In an embodiment where the light intensity is changed by altering the power supply (P) of light source 118, the light intensity change α can be a function of the power supply value f(P), where the function f(.) is unique for each light source 118 depending on its physical characteristics.

[0068] Therefore, equation (2) can be extended to equation (3), as shown below:

[0069] C=αL control_source ×R+L env ×R equation (3)

[0070] Among them, from the target surface 102L control The reflected light is further represented as the light from the light source 118L control_source The product of the emitted light intensity and the change in light intensity α. It should be understood that the light emitted by light source 118 may vary depending on the model or manufacturer of light source 118, and L... control_source Each model and / or manufacturer of the 118 light source is unique.

[0071] According to equation (3), the following linear equations (4) can be obtained from multiple images 105 acquired with different light intensity variations α:

[0072] Equation (4)

[0073] The color tensor C and light intensity change α from the linear equation (4) are known values ​​as described above, and are derived from the ambient light source 106L. env The components of light emitted by ×R are the same for all linear equations (4). Therefore, each of equations (4) adopts the linear equation form y = ax + b, where y is the color tensor C, x is the light intensity change α, and a is the color feature tensor. b is used as an ambient light source 106L env The result of ×R is the component of light reflected from the target surface. Color feature tensor. The color feature tensor represents the light reflected from the target surface 102, which has a surface color of R, as a result of illumination from light emitted from light source 118. The color feature tensor can be determined using any suitable mathematical modeling technique. and L env ×R. In some embodiments, linear regression is used in conjunction with the following equation (5) to determine the color feature tensor. and L env ×R:

[0074]

[0075] In step 330, the color processing module 124 receives the color feature tensor determined by the image processing module 122. The trained model is then used to predict the value of the target surface color R.

[0076] In some embodiments, the present invention utilizes a trained linear model to predict the color R of the target surface 102. The trained linear model may be specific to a particular brand or model of the light source 118. The linear model can be trained using any suitable machine learning algorithm to learn the model's parameters (e.g., weights and biases).

[0077] Figure 4 A flowchart of a method 400 for training a linear model for predicting the color R of a target surface is shown according to an embodiment of the present invention. Method 400 may utilize a supervised learning algorithm to learn the weights of the linear model during training. As will be understood from the details below, the linearity of the model stems from the absence of a non-linear activation function.

[0078] In step 410, each weight of the linear model can be initialized with an initial value, which can be a random value, zero, or any other suitable initial value.

[0079] In step 420, the linear model receives samples from the training dataset, where each sample includes a data pair, the data pair including a training color feature tensor, the training color feature tensor being the color of the light emitted by the light source 118 and the target surface color (i.e., L). control_source The training color feature tensor is the product of (×R) and the corresponding label identifying the target surface color value R. The training color feature tensor can be determined based on a target surface under various ambient lighting conditions, multiple target surfaces under the same ambient lighting conditions, or multiple target surfaces under various lighting conditions.

[0080] In step 430, the model calculates the dot product of the weight matrix of the linear model and the training color feature tensor to obtain the predicted target surface color.

[0081] In step 440, the predicted target surface color is compared with the R value identified by the label corresponding to the training color feature tensor from the same training samples to determine the error value by applying a loss function. The loss function varies depending on the target of the linear model. As a non-limiting example, if the goal of the linear model is to predict a target surface color that is as visually close as possible to the target surface color R, the loss function can be calculated based on the training dataset in the CIELab color space, using a Euclidean metric between the predicted target surface color and the target surface color R. The CIELab color space, defined by the International Commission on Illumination (CIE), represents colors using three values: L* represents brightness from black (0) to white (100), a* represents from green (-) to red (+), and b* represents from blue (-) to yellow (+). The CIELab color space is designed such that the same number of numerical changes in the CIELab color space values ​​correspond to approximately the same number of visually perceived changes. For example, the predicted target surface color in the RGB color space of camera 114 can be converted to a CIELab color space value using a known linear transformation (e.g., by converting RGB values ​​to values ​​in the XYZ color space and then converting the XYZ values ​​to CIELab color space values). For vectors a and b of dimension n, the Euclidean metric between these vectors can be determined as follows:

[0082]

[0083] Where (a1, a2, a3...a... n ) are the CIELab color space values ​​of the target surface color R from the training dataset, while (b1, b2, b3...b n ) are the CIELab color space values ​​of the target surface color predicted by the linear model, and (a1, a2, a3...a...n (b1, b2, b3...b) n The Euclidean metric between these values ​​is used to indicate the difference in color values.

[0084] Alternatively, if the goal of the linear model is to predict the target surface color as closely as possible to the color space values ​​of the target surface color R from the training dataset, then the loss function can be the mean absolute difference between the predicted target surface color and the target surface color R from the training dataset. For example, the mean absolute difference between vectors a and b of dimension n can be calculated as follows:

[0085]

[0086] In step 450, the error value (i.e., the loss) calculated in step 440 can be backpropagated to update the values ​​of the weights in the weight matrix using the gradient descent algorithm, thereby minimizing the loss. Steps 420 to 450 can be iteratively repeated until the error value (i.e., the loss) is below a threshold, such that the predicted target surface color is considered sufficiently close to the target surface value R, and the weights of the linear model have been optimized.

[0087] Once the linear model is trained (i.e., its weights and biases are learned), the trained linear model can be deployed to the color measurement device 100 in any form, including as a standalone program or as a module, component, subroutine, or other unit stored in memory 110, suitable for execution by processor 108. The trained model can also be deployed to a computing system (in other words, stored in the computing's memory) for execution by one or more processors 108 of the computing system. For example, the computing system may include one or more physical machines, such as servers, or one or more virtual machines provided by a private or public cloud computing platform. The computing system can be a distributed computing system, comprising multiple computers distributed across multiple sites and interconnected via a communication network, and the trained linear model can be deployed and run on the computers located at these multiple sites.

[0088] It may be necessary to convert the predicted target surface color in the first color space (e.g., the RGB color space of the acquired image 105) to a second color space that may be more suitable for another purpose. For example, the sRGB color space may be more suitable for display on an electronic display unit, and the CIELab color space can provide a target surface color representation independent of the display device. Therefore, optionally, in step 460, the predicted target surface color R from step 450 can be converted to the second color space using a conversion function known in the art.

[0089] In some other embodiments, the color processing module 124 uses a trained neural network to predict the color of the target surface. Figure 5A flowchart of a method 500 for training a neural network to approximate a model is shown. According to the present invention, the neural network is used to predict the color R of a target surface. In method 500, a color processing module 124 may include a neural network trained using a training dataset comprising a color feature tensor L representing light emitted from a light source 118 and reflected from a target surface 102. control_source ×R. As a non-limiting example, the neural network may be a multi-layer perceptron (MLP) network, which may include an input layer, one or more hidden layers, and an output layer. For example, the neural network may be trained to meet the accuracy requirements for inferring the target surface color R. One or more hidden layers of the neural network may enable method 500 to convert the inferred target surface color R to any desired color space without further conversion steps.

[0090] During the training of the neural network, frames are provided as input, or alternatively, the color of the light emitted by the light source 118 and the target surface color (i.e., L) from the training dataset are provided. control_source The training color feature tensor is the product of (×R), and the corresponding label identifies the target surface color value R. A nonlinear activation function can be applied to the intermediate predicted target surface color to derive the predicted target surface color in the desired color space. Therefore, the need for further color space transformation is eliminated. In some embodiments, the activation function is implemented using a rectified linear unit (ReLU); however, other activation functions may be used as appropriate.

[0091] Figure 5 A flowchart of a method 500 for predicting the color R of a target surface is shown according to another exemplary embodiment of the present invention, the method being implemented in step 320. Specifically, method 500 involves partitioning a color space to represent the target surface color R as a plurality of color subspaces, and training a model (whether a linear network or a neural network) for each color subspace weight value optimized for the color subspaces, thereby generating a plurality of models to be deployed to predict the target surface color R.

[0092] Specifically, in step 510, training samples from the training dataset are received as input to a neural network used to generate a weight tensor containing subspace weight values ​​corresponding to each color subspace. These weight values ​​indicate the weights assigned to the output of each color subspace when determining the final target surface color prediction. The training samples include training data pairs, each containing a training color feature tensor, which is the color of the light emitted by the light source 118 and the target surface color (i.e., L). control_sourceThe product of (×R) and the corresponding label identifying the target surface color value R. The trained color feature tensor indicates the light reflected from the target surface as a result of illumination from light emitted from light source 118.

[0093] In step 520, the color space is divided into multiple color subspaces. In some embodiments, the color space can be divided into multiple color subspaces according to manually defined boundaries. In other embodiments, the color subspace division boundaries can be learned using machine learning algorithms.

[0094] In step 530, each subspace model receives the training color feature tensor of the training samples. As input, each color subspace model can infer the target surface color R;

[0095] In step 540, based on the weight tensor, each subspace model uses the final target surface color inference as a weighted value for the target surface color inference;

[0096] In step 550, the finally inferred target surface color is compared with the R value identified in the labels of the training samples to determine the error value (i.e., the loss). The error value (i.e., the loss) is calculated using backpropagation with the gradient ascent algorithm to adjust the weight values ​​of the model matrix for each color subspace, as well as the weights in the neural network that generates the weight tensor, thereby minimizing the loss. Steps 510 to 550 can be repeated iteratively until the error value (i.e., the loss) is below a threshold, such that the final target surface color inference is considered sufficiently close to the target surface color R.

[0097] In some embodiments, the color subspace model and the weight tensor neural network can be trained simultaneously or sequentially.

[0098] The steps and / or operations in the flowcharts and accompanying drawings described herein are for illustrative purposes only. These steps and / or operations can be varied in many ways without departing from the teachings of this invention. For example, steps may be performed in a different order, or steps may be added, deleted, or modified.

[0099] While the invention has been described at least in part according to the method, those skilled in the art will understand that the invention also relates to various components for performing at least some aspects and features of the described method, whether by hardware components, software, or any combination of both, or in any other manner. Furthermore, the invention relates to pre-recorded storage devices or other similar machine-readable media, including program instructions stored thereon for performing the methods described herein.

[0100] The invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. The invention is intended to cover and encompass all suitable technical modifications. Therefore, the scope of the invention is described by the appended claims rather than by the foregoing description. The scope of the claims should not be limited to the embodiments set forth in the examples, but should be given the broadest interpretation consistent with the entire description.

Claims

1. A method for measuring the color of a target surface using a color measuring device, characterized in that, include: Multiple digital images of the target surface are acquired using a camera of the color measuring device, the target surface being illuminated by constant color light emitted by a controlled light source of the color measuring device, which is positioned relative to the camera, and by ambient light of constant intensity and constant color, wherein the intensity of the light illuminating the target surface varies by a known amount between the acquisition of the images; Multiple linear equations are generated based on the image data included in each of the multiple digital images, wherein the image data is the sum of the product of the color of the controlled light source and the surface color of the target surface, and the product of the ambient light and the surface color of the target surface; a color feature tensor corresponding to the product of the color of the light emitted by the controlled light source and the surface color of the target surface is determined based on the multiple linear equations; The surface color of the target surface is inferred based on the color feature tensor.

2. The method according to claim 1, characterized in that, It also includes changing the intensity of light illuminating the target surface by altering the distance between the controlled light source and the target surface.

3. The method according to claim 1 or 2, characterized in that, Inferring the surface color of the target surface based on the color feature tensor includes inputting the color feature tensor into a trained model that processes the color feature tensor, and inferring the surface color of the target surface based on the processed color feature tensor.

4. The method according to claim 3, characterized in that, The trained model is a linear model, which is trained in the following way: Initialize the weights of the weight matrix; Receive a batch of training samples in the training dataset, each training sample in the training dataset including an image of the surface and a ground truth surface color tensor representing the true color of the surface; For each corresponding sample in this batch of training samples: The images of the corresponding training samples are processed to generate a color feature tensor corresponding to the product of the training color of the controlled light source and the ground truth surface color; The product of the color feature tensor and the weight matrix is ​​used as the surface color tensor for calculation; The error value is defined as the difference between the surface color tensor and the ground true surface color tensor; The weights of the weight matrix are updated based on the error value; Receive more batches of training samples until the weights of the weight matrix are optimized.

5. The method according to claim 3, characterized in that, The trained model is approximated by a neural network, which is trained in the following manner: Initialize the weights of the neural network; Receive a batch of training samples from a training dataset, each training sample in the training dataset including an image of a surface and the ground truth surface color of the surface; For each corresponding training sample: The images of the training samples are processed to generate a color feature tensor corresponding to the product of the light emitted by the controlled light source and the ground truth surface color of the surface. The surface color of the training samples is inferred by using the color feature tensor generated for the corresponding training samples through forward propagation of the neural network. The difference between the inferred surface color and the true surface color is used as the error value for calculation. Perform backpropagation to update the weights of the neural network based on the error value.

6. The method according to any one of claims 1 to 2, 4 to 5, characterized in that, The data collection includes: The distance between the color measuring device and the target surface is detected in the first instant using a time-of-flight sensor; The time-of-flight sensor is used to detect the distance between the color measuring device and the target surface at a second time. When the distance between the color measuring device and the target surface changes, the controlled light source is controlled to emit light of constant color and constant intensity to illuminate the target surface, and digital frames of the target surface are acquired.

7. The method according to any one of claims 1 to 2, 4 to 5, characterized in that, The intensity of light illuminating the target surface is changed by altering the amount of electricity supplied to the controlled light source at a constant distance from the target surface.

8. The method according to any one of claims 1 to 2, 4 to 5, characterized in that, The inference also includes converting the inferred surface color of the surface in the first color space to the surface color in the second color space.

9. The method according to claim 3, characterized in that, The trained model is trained in the following way: Divide the color space into multiple color subspaces; Initialize the weights of the subspace separation model and the weights of the color subspace model for each of the plurality of color subspaces; Receive training samples including images of the surface and ground truth surface colors of the surface; The images of the training samples are processed to generate a color feature tensor corresponding to the product of the light emitted by the controlled light source and the ground truth surface color; Using the received color feature tensor, compute the subspace color tensor through each color subspace model; The subspace weight tensor is generated using the subspace separation model. Surface color is inferred by applying the subspace weight tensor to the subspace color tensor; The error value is defined as the difference between the inferred surface color and the true surface color of the ground. Perform backpropagation to update the weights of each color subspace model and the weights of the subspace separation model in the plurality of color subspace models.

10. The method according to claim 9, characterized in that, The color space is defined manually.

11. The method according to claim 9, characterized in that, The multiple color subspace models and subspace separation models are trained simultaneously.

12. A color measuring device for measuring the color of a target surface irradiated with ambient light of constant intensity and constant color, characterized in that, include: An image acquisition device for acquiring multiple images of the target surface; A light source for illuminating a target surface with constant color light, wherein the intensity of the light illuminating the target surface varies by a known amount between the acquisition of consecutive images; A color measurement system, for Multiple linear equations are generated based on the image data included in each of the multiple images, wherein the image data is the sum of the product of the color of the light source and the surface color of the target surface, and the product of the ambient light and the surface color of the target surface; a color feature tensor corresponding to the product of the color of the light source and the surface color of the target surface is determined based on the multiple linear equations; The surface color of the target surface is inferred based on the color feature tensor.

13. The color measuring device according to claim 12, characterized in that, The intensity of the light illuminating the target surface is changed by altering the distance between the light source and the target surface.

14. The color measuring device according to claim 12 or 13, characterized in that, The image acquisition device is used to acquire multiple images of the target surface in the following manner: The distance between the color measuring device and the target surface is detected in the first instant using a time-of-flight sensor; The time-of-flight sensor is used to detect the distance between the color measuring device and the target surface at a second time. When the distance between the color measuring device and the target surface changes, the light source is controlled to emit light of constant color and constant intensity to illuminate the target surface, and digital frames of the target surface are acquired.

15. The color measuring device according to claim 12 or 13, characterized in that, The surface color of the target surface is inferred by inputting the color feature tensor into a trained model that processes the color feature tensor, and inferring the surface color of the target surface based on the processed color feature tensor.

16. The color measuring device according to claim 12 or 13, characterized in that, The intensity of light illuminating the target surface is changed by altering the amount of electricity supplied to the light source at a constant distance from the target surface.

17. The color measuring device according to claim 12 or 13, characterized in that, The color measurement system is also used to convert the inferred surface color of a surface in a first color space into a surface color in a second color space.

18. A computer-readable medium having instructions tangibly stored thereon, characterized in that, When the instruction is executed by the processing unit, it causes the processing unit to perform the method of any one of claims 1 to 11.

19. A computer program product comprising instructions, characterized in that, When the instruction is executed by the processing unit, it causes the processing unit to perform the method of any one of claims 1 to 11.

Images