Information processing device and information processing method

Abstract

To provide a mechanism capable of more easily reproducing on a display side a color of an object on an imaging side.SOLUTION: An information processing device includes a control unit. The control unit acquires characteristic data relating to spectral sensitivity characteristics of an imaging device, imaging side spectral data relating to spectral distribution characteristics of a light source located in an imaging environment in which the imaging device performs imaging, and color spectral data relating to spectral reflectivity characteristics of a prescribed color. The control unit calculates, by using the characteristic data, the imaging side spectral data, and the color spectral data, an RGB value outputted from the imaging device when imaging of the prescribed color is performed by using the imaging device. The control unit acquires display side spectral data relating to spectral distribution characteristics of a light source located in a display environment in which imaging data imaged by the imaging device is displayed on a display device. The control unit calculates, by using the display side spectral data and the color spectral data, an XYZ value when the prescribed color is displayed on the display device. The control unit calculates conversion information for converting the RGB value into the XYZ value.SELECTED DRAWING: Figure 1B

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] When image data obtained by photographing a subject with a camera is displayed on a monitor, a display method is known in which the image data is displayed so that the color actually perceived by an observer visually of the subject and the color displayed on the monitor are the same color (see, for example, Patent Document 1).

[0003] In this display method, a camera captures multiple images displayed on a reference monitor, and the correspondence between the RGB values of the images captured by the camera and the XYZ values of the images is maintained. In this display method, the RGB values obtained by capturing an image of a subject using the reference monitor as a light source are converted into XYZ values based on the correspondence maintained, and the images are displayed on the reference monitor, so that the colors of the subject captured on the reference monitor are reproduced on the reference monitor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-177484 Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned display method, it was necessary to display multiple images on a reference monitor and take pictures with a camera. In addition, it was necessary to use the reference monitor as a light source to take pictures of the subject, which also limited the conditions for taking pictures of the subject. As such, the conventional display method had a problem in terms of more easily reproducing the color of the subject on the shooting side on the monitor (display side).

[0006] Therefore, the present disclosure provides a mechanism that can more easily reproduce the color of a subject captured on the image capture side on the display side.

[0007] It should be noted that the above problem or object is merely one of multiple problems or objects that can be solved or achieved by multiple embodiments disclosed in this specification. [Means for solving the problem]

[0008] According to the present disclosure, there is provided an information processing device. The information processing device includes a control unit. The control unit acquires characteristic data related to the spectral sensitivity characteristics of an imaging device. The control unit acquires capture-side spectral data related to the spectral distribution characteristics of a light source in an imaging environment in which the imaging device captures an image. The control unit acquires color spectral data related to the spectral reflectance characteristics of a predetermined color. The control unit uses the characteristic data, the capture-side spectral data, and the color spectral data to calculate RGB values to be output by the imaging device when the predetermined color is captured using the imaging device. The control unit acquires display-side spectral data related to the spectral distribution characteristics of a light source in a display environment in which imaging data captured by the imaging device is displayed on a display device. The control unit uses the display-side spectral data and the color spectral data to calculate XYZ values when the predetermined color is displayed on the display device. The control unit calculates conversion information for converting the RGB values into the XYZ values. [Brief explanation of the drawings]

[0009] [Figure 1A] 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. [Figure 1B] FIG. 1 is a diagram for explaining an overview of information processing according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an example configuration of an imaging device according to an embodiment of the present disclosure. [Figure 3] 1 is a block diagram illustrating an example configuration of a conversion device and a display device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of a spectral data acquisition unit according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating an example of conversion information according to an embodiment of the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating an example of conversion information calculated by a conversion information calculation unit according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram for explaining an example of a conversion process performed by a conversion device according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating a flow of an example of information processing executed by an information processing device according to an embodiment of the present disclosure. [Figure 10] 10 is a flowchart illustrating an example of a flow of a conversion process executed by a conversion device according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is a block diagram illustrating a configuration example of an information processing device according to a second modified example of the embodiment of the present disclosure. [Figure 12] FIG. 10 is a block diagram illustrating a configuration example of an information processing device according to a fourth modified example of the embodiment of the present disclosure. [Figure 13] FIG. 10 is a block diagram showing a configuration example of an error calculation unit according to a fourth modified example of the embodiment of the present disclosure. [Figure 14] FIG. 10 is a diagram illustrating an example of display information according to a fourth modified example of the embodiment of the present disclosure. [Figure 15] FIG. 10 is a diagram showing another example of display information according to the fourth modified example of the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] One or more embodiments (including examples and modifications) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects.

[0012] <<1. Overview of the Information Processing System>> <1.1. Example of an outline of an information processing system> 1A is a diagram illustrating an overview of an information processing system 10 according to an embodiment of the present disclosure. The information processing system 10 includes an information processing device 100, a conversion device 200, an imaging device 300, and a display device 500.

[0013] The imaging device 300 is a camera that uses the imaging-side illumination device 600A as an imaging-side light source to capture an image of the subject 400. The imaging device 300 may capture a moving image of the subject 400 or a still image.

[0014] The display device 500 is a device that displays to an observer (not shown) the captured image captured by the imaging device 300. The display device 500 uses the display-side illumination device 600B as a display-side light source to display the captured image to the observer.

[0015] In this way, when an image captured by the imaging device 300 is displayed on the display device 500, it may be desirable to estimate the color of the subject that the observer would see if the display-side illumination device 600B were used as a light source, and reproduce it on the display device 500.

[0016] For example, suppose an image of a patient taken at Hospital A is displayed on a monitor (display device 500) at Hospital B and a doctor (observer) examines the patient. In this case, by reproducing the facial color of the patient displayed on the monitor in the same color as when the doctor observes the patient at Hospital B, the doctor can examine the patient at Hospital A in the same way as if the patient were at Hospital B.

[0017] As described above, one method for reproducing colors under the display-side illumination device 600B from an image captured by the imaging device 300 is to acquire the color coordinates of the captured image as absolute color coordinates XYZ values. More specifically, in this method, the information processing system 10 acquires the color coordinates of the subject 400 and the capturing-side illumination device 600A as XYZ values. This allows the information processing system 10 to maintain absolute color coordinates from the time of imaging by the imaging device 300 to the time of display by the display device 500.

[0018] In this case, however, the imaging device 300 must be a device that generates an image with XYZ values. That is, the imaging device 300 must be a device called an XYZ camera that mimics the spectral characteristics of the human eye. However, XYZ cameras are expensive and not common.

[0019] Therefore, in the information processing system 10 according to the proposed technique of the present disclosure, an image of RGB values (hereinafter also referred to as an RGB image) is captured using an imaging device 300 that captures an RGB image (a so-called RGB camera). In the information processing system 10, the conversion device 200 performs conversion processing to convert the RGB values of the RGB image into XYZ values, thereby acquiring the captured image of XYZ values.

[0020] The information processing device 100 generates conversion information used in the conversion process by the conversion device 200. For example, when an RGB image captured by the imaging device 300 is displayed on the display device 500, the information processing device 100 generates conversion information for displaying the same colors as those visually recognized when a viewer directly observes the subject 400 on the display device 500.

[0021] In this way, the information processing system 10 of FIG. 1A is a system that estimates the color of the subject 400 that would be viewed if the display-side illumination device 600B were used as the light source, from an image captured by the imaging device 300 using the capturing-side illumination device 600A as the light source, and reproduces the color on the display device 500.

[0022] <1.2. Issues with conventional technology> Here, a system disclosed in Japanese Patent Application Laid-Open No. 2001-8220 is known as a color reproduction system that estimates and reproduces colors under a specified light source based on an RGB image. This color reproduction system estimates the color of a subject under a specified illumination light using statistical data on the subject's spectral reflectance. In this case, the color reproduction system switches the statistical data according to the subject's image capture signal to estimate the subject's color with high accuracy.

[0023] However, because the above system estimates the color of the subject using statistical data based on a specified illumination spectrum, there is a risk that the accuracy of the color estimation of the subject may deteriorate if, for example, the illumination light on the photographing side where the subject is photographed or the illumination light on the observation side where the monitor is installed changes.

[0024] Therefore, there is a demand for a system that allows for more flexible changes after installation, such as the ability to change the illumination light later.

[0025] Furthermore, the above system performs color estimation using statistical data on the subject's spectral reflectance, which is stored in 1-nm increments within the visible range of 380 nm to 780 nm. In this case, the system stores 400 pieces of data per color, which requires a significant amount of computation to calculate the statistical data. This may require the system to be built using special hardware for the computation. Alternatively, the system may only be able to perform color estimation for still images, not moving images.

[0026] Therefore, it is desirable to build a system that can further reduce the amount of calculation and more easily reproduce the color of the subject captured on the monitor.

[0027] Therefore, the proposed technology of the present disclosure provides a mechanism that allows the color of a subject on the shooting side to be more easily reproduced on a monitor (display device) in a color reproduction system technology that reproduces the color of a subject on the shooting side on a monitor (display device) on the display side.

[0028] More specifically, the information processing device 100 of the information processing system 10 calculates conversion information for converting a captured image (RGB image) into a display image (converted image). For example, the information processing device 100 acquires characteristic data related to the spectral sensitivity characteristics of the image capturing device 300. The information processing device 100 acquires image capturing-side spectral data related to the spectral distribution characteristics of a light source (e.g., image capturing-side lighting device 600A) in an image capturing environment where the image capturing device 300 captures an image. The information processing device 100 acquires color spectral data related to the spectral reflectance characteristics of a predetermined color (sample color). Using the acquired characteristic data, image capturing-side spectral data, and color spectral data, the information processing device 100 calculates RGB values that the image capturing device 300 will output when capturing an image of the sample color using the image capturing device 300.

[0029] Furthermore, the information processing device 100 acquires display-side spectral data relating to the spectral distribution characteristics of a light source (for example, a display-side illumination device 600B) in the display environment in which the display device 500 is displayed. The information processing device 100 uses the acquired display-side spectral data and color spectral data to calculate XYZ values when a sample color is displayed on the display device 500. The information processing device 100 calculates conversion information for converting the calculated RGB values into XYZ values.

[0030] Although the application of the color reproduction system to remote medical care has been described here as an example of its application, the application is not limited thereto. For example, the color reproduction system may be applied to an inspection system in a factory. More specifically, the color reproduction system technology proposed by the present disclosure may be applied to, for example, a case where the color of a product manufactured in a factory is to be checked from a remote location.

[0031] <1.3. Overview of information processing> 1B is a diagram for explaining an overview of information processing according to an embodiment of the present disclosure. The information processing is processing for generating conversion information used in conversion from an RGB image (captured image) to an XYZ image (display image) performed by the conversion device 200.

[0032] The information processing device 100 acquires image capturing side external light information from, for example, an external light sensor (not shown) mounted on the imaging device 300 (step S1). The image capturing side external light information may include, for example, information about the image capturing side illumination device 600A.

[0033] The information processing device 100 acquires display-side external light information from, for example, an external light sensor (not shown) mounted on the display device 500 (step S2). The display-side external light information may include, for example, information related to the display-side illumination device 600B.

[0034] The information processing device 100 acquires imaging-side spectral data relating to the spectral distribution characteristics of the imaging-side illumination device 600A from the imaging-side light source spectral data DB based on the acquired imaging-side external light information (step S3).

[0035] The information processing device 100 acquires display-side spectral data relating to the spectral distribution characteristics of the display-side illumination device 600B from the display-side light source spectral data DB based on the acquired display-side external light information (step S4).

[0036] The information processing device 100 calculates RGB values based on the characteristic data acquired from the imaging device spectral data DB, the image capture side spectral data acquired in step S3, and the color spectral data acquired from the sample color spectral data DB (step S5). Here, the characteristic data is information related to the spectral sensitivity characteristics of the imaging device 300. Furthermore, the color spectral data is information related to the spectral reflectance characteristics of a predetermined sample color. For example, the color spectral data may include information related to the spectral reflectance characteristics of a plurality of sample colors (e.g., 1,000 to 2,000 colors).

[0037] The information processing device 100 calculates, for example, based on the characteristic data, the image capturing side spectral data, and the color spectral data, the RGB values output by the image capturing device 300 when the subject 400 of a sample color is captured under the image capturing side illumination device 600A. The information processing device 100 calculates the RGB values for each of the multiple sample colors.

[0038] Next, the information processing device 100 calculates XYZ values based on the display-side spectral data acquired in step S4 and the color spectral data acquired from the sample color spectral data DB (step S6).

[0039] For example, based on the display-side spectral data and the color spectral data, the information processing device 100 calculates the XYZ values to be displayed on the display device 500 when displaying the subject 400 of a sample color under the display-side illumination device 600B. The information processing device 100 calculates the XYZ values for each of the multiple sample colors.

[0040] The information processing device 100 calculates conversion information for converting the RGB image into an XYZ image based on the RGB values calculated in step S5 and the XYZ values calculated in step S6 (step S7). The conversion information is, for example, information that associates the RGB values with the XYZ values for each of a plurality of sample colors. Details of the conversion information will be described later.

[0041] The conversion device 200 converts the RGB values of the image captured by the imaging device 300 into a converted image of XYZ values using the conversion information calculated by the information processing device 100 (step S8), and displays the converted image on the display device 500. Here, if the display device 500 is a device that displays an image of XYZ values, the conversion device 200 outputs the converted image of XYZ values to the display device 500. On the other hand, if the display device 500 is a device that displays an image of RGB values, the conversion device 200 may convert the converted image of XYZ values into a converted image of RGB values and output the converted image to the display device 500.

[0042] The conversion device 200 converts an RGB image into a converted image of XYZ values for each pixel of the RGB image on a frame-by-frame basis. On the other hand, the information processing device 100 may calculate the conversion information when, for example, the image capturing side illumination device 600A or the display side illumination device 600B is switched or when a color sample (color specimen) containing multiple sample colors is switched.

[0043] The color sample may change depending on the type of subject 400, such as the skin color of a person. In other words, when the type of subject imaged by the image capture device 300 changes, the color spectral data of the color sample may change; however, in general, the possibility of the color sample changing is low and it is considered to be fixed.

[0044] In addition, when adaptively responding to switching between the capturing-side illumination device 600A and the display-side illumination device 600B, the information processing device 100 performs information processing to calculate conversion information at a predetermined cycle, for example, every few seconds.

[0045] As described above, the information processing system 10 according to this embodiment captures the spectrum of the subject 400, including the capturing-side light source (capture-side illumination device 600A), using the imaging device 300 to acquire an RGB image. The information processing system 10 converts the acquired RGB image into absolute color coordinates XYZ under the display-side light source (display-side illumination device 600B) using conversion information. In this way, the information processing system 10 according to this embodiment does not estimate the spectral reflectance of the subject 400, and therefore can reduce the amount of calculation required for conversion.

[0046] Furthermore, the conversion information is calculated based on the color spectral data of the sample color, etc. As described above, the conversion information may be recalculated, for example, when the capturing-side illumination device 600A or the display-side illumination device 600B is switched. This reduces the amount of calculations performed by the information processing system 1 according to this embodiment, and makes it easier to reproduce the color of the subject 400 on the capturing side on the display device 500.

[0047] Furthermore, the information processing system 10 calculates the conversion information using, for example, image-capturing external light information and display-side external light information acquired from an external light sensor. Therefore, the information processing system 10 can easily update the conversion information even if the lighting device is switched after the system is installed, allowing for more flexible changes.

[0048] <<2. Example of information processing system configuration>> Next, a configuration example of each device of the information processing system 10 according to an embodiment of the present disclosure will be described.

[0049] [Lighting device 600] First, the imaging-side illumination device 600A and the display-side illumination device 600B will be described. When there is no need to particularly distinguish between the imaging-side illumination device 600A and the display-side illumination device 600B, they will also be simply referred to as illumination device 600. The illumination device 600 is used as a light source for the information processing system 10.

[0050] For example, an incandescent lamp, a fluorescent lamp, an LED (Light Emitting Diode), etc. may be used as the lighting device 600. Instead of the lighting device 600, sunlight may be used as the light source of the information processing system 10.

[0051] The shooting-side light source may be the same type of light source as the display-side light source, or may be a different type of light source. Furthermore, when the imaging device 300 and the display device 500 are placed in the same environment, such as in the same room, the shooting-side light source and the display-side light source may be the same.

[0052] [Imaging device 300] Next, the imaging device 300 of the information processing system 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example configuration of the imaging device 300 according to an embodiment of the present disclosure. The imaging device 300 shown in Fig. 2 includes a communication unit 310, an imaging unit 320, and an ambient light sensor 330.

[0053] (Communication unit 310) The communication unit 310 is a communication interface that communicates with an external device via a network, either wired or wirelessly, and is realized by, for example, a network interface card (NIC).

[0054] (Image capture unit 320) The imaging unit 320 captures an image of the subject 400 and generates a captured image (RGB image). The imaging unit 320 is, for example, an image sensor. The imaging unit 320 captures and generates, for example, a moving image or a still image. The imaging unit 320 outputs the captured image to the conversion device 200 via the communication unit 310.

[0055] (External light sensor 330) The external light sensor 330 is a device that acquires information about an image-capturing side light source. The external light sensor 330 has, for example, multiple color sensors (not shown). The multiple color sensors are, for example, sensors that separate and extract light (color components) of different wavelengths. The external light sensor 330 separates, for example, light in the human visible range (wavelengths of 380 nm to 780 nm) into multiple light components of different wavelengths using the multiple color sensors. In this way, the external light sensor 330 performs filter spectral processing to separate and extract light of predetermined wavelengths. The external light sensor 330 outputs the separation result to the information processing device 100, for example, via the communication unit 310.

[0056] Although the case where the ambient light sensor 330 performs spectroscopy using multiple color sensors has been described above, the ambient light sensor 330 is not limited to this. The ambient light sensor 330 may be a sensor that obtains the spectrum of the light source in more detail, such as a spectrometer. By using a spectrometer in this way, the light source estimation process by the information processing device 100, which will be described later, can be omitted. However, because spectrometers are expensive and uncommon, using a color sensor makes it possible to configure the ambient light sensor 330 more cheaply and easily.

[0057] Furthermore, although the imaging device 300 has been described as having the external light sensor 330, this is not limiting. It is sufficient that the external light sensor 330 acquires information about the light source on the imaging side. For example, the external light sensor 330 may be disposed independently on the imaging side, separate from the imaging device 300.

[0058] Next, the conversion device 200 and the display device 500 of the information processing system 10 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example configuration of the conversion device 200 and the display device 500 according to an embodiment of the present disclosure.

[0059] [Conversion device 200] The conversion device 200 is an information processing device that converts an RGB image acquired by the imaging device 300 into a converted image of absolute color coordinates XYZ values under a display-side light source, and displays the converted image on the display device 500. The conversion device 200 shown in FIG. 3 includes a communication unit 210, a storage unit 220, and a control unit 230.

[0060] (Communication unit 210) The communication unit 210 is a communication interface that communicates with external devices via a network, either wired or wirelessly, and is realized by, for example, a network interface card (NIC).

[0061] (Storage unit 220) The storage unit 220 is a data readable / writable storage device such as a DRAM, an SRAM, a flash memory, or a hard disk. The storage unit 220 functions as a storage means of the conversion device 200. The storage unit 220 stores conversion information used in the conversion process performed by the control unit 230, which will be described later.

[0062] (control unit 230) The control unit 230 controls each unit of the conversion device 200. The control unit 230 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), or the like executing a program stored inside the conversion device 200 using a random access memory (RAM) or the like as a working area. The control unit 230 is also realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0063] The control unit 230 includes a conversion information acquisition unit 231, a conversion processing unit 232, and a display control unit 233. Each block (conversion information acquisition unit 231 to display control unit 233) constituting the control unit 230 is a functional block indicating a function of the control unit 230. These functional blocks may be software blocks or hardware blocks. For example, each of the above-mentioned functional blocks may be a software module realized by software (including a microprogram), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The control unit 230 may be configured by functional units different from the above-mentioned functional blocks. The method of configuring the functional blocks is arbitrary.

[0064] The control unit 230 may be configured with functional units different from the above-mentioned functional blocks. Also, some or all of the operations of the blocks (conversion information acquisition unit 231 to display control unit 233) constituting the control unit 230 may be performed by another device. For example, some or all of the operations of the blocks constituting the control unit 230 may be performed by a control device realized by cloud computing.

[0065] (Conversion information acquisition unit 231) The conversion information acquisition unit 231 acquires conversion information from the information processing device 100 via the communication unit 210 and stores it in, for example, the storage unit 220. The conversion information acquisition unit 231 acquires the conversion information at a predetermined cycle, for example, every few seconds. The conversion information is information for converting an RGB image captured by the imaging device 300 into a converted image of absolute color coordinates XYZ values under a display-side light source, and is expressed by, for example, conversion coefficients or an LUT (Look Up Table), etc.

[0066] (Conversion processing unit 232) The conversion processing unit 232 executes conversion processing to convert the RGB image acquired from the imaging device 300 via the communication unit 210 into a converted image of absolute color coordinates XYZ values using the conversion information. The conversion processing unit 232 outputs the converted image to the display control unit 233. Details of the conversion processing will be described later.

[0067] (Display control unit 233) The display control unit 233 controls the display device 500 to display the converted image. If the display device 500 can display an image of XYZ values, the converted image of XYZ values generated by the conversion processing unit 232 is output to the display device 500. On the other hand, if the display device 500 cannot display an image of XYZ values and displays an image of RGB values, the display control unit 233 converts the converted image of XYZ values generated by the conversion processing unit 232 into a converted image of RGB values and outputs it to the display device 500. It is assumed that information for converting the converted image from XYZ values to RGB values is stored in advance in the storage unit 220.

[0068] [Display device 500] The display device 500 is, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The display device 500 displays a converted image output by the conversion device 200. When the conversion device 200 outputs a converted image of XYZ values, the display device 500 may convert the converted image of XYZ values into RGB values and then display the converted image on the display.

[0069] (External light sensor) The display device 500 shown in FIG. 3 also includes an ambient light sensor 510. The ambient light sensor 510 is a device that acquires information about a display-side light source. The ambient light sensor 510 has, for example, a plurality of color sensors (not shown). The plurality of color sensors are, for example, sensors that separate and extract light (color components) of different wavelengths. The ambient light sensor 510 separates, for example, light in the human visible range (wavelengths of 380 nm to 780 nm) into a plurality of light beams of different wavelengths using the plurality of color sensors. In this way, the ambient light sensor 510 performs filter spectral processing to separate and extract light of predetermined wavelengths. The ambient light sensor 510 outputs the separation result to the information processing device 100.

[0070] Although the case where the ambient light sensor 510 performs spectroscopy using multiple color sensors has been described above, the ambient light sensor 510 is not limited to this. The ambient light sensor 510 may be a sensor that obtains the spectrum of the light source in more detail, such as a spectrometer. By using a spectrometer in this way, the light source estimation process by the information processing device 100, which will be described later, can be omitted. However, since spectrometers are expensive and uncommon, the ambient light sensor 510 can be configured more cheaply and easily by using a color sensor.

[0071] Furthermore, although the display device 500 has been described here as having the external light sensor 510, this is not limiting. It is sufficient that information regarding the display-side light source is acquired by the external light sensor 510. For example, the external light sensor 510 may be independently disposed on the display side, separate from the display device 500, or when the conversion device 200 is disposed on the display side, the external light sensor 510 may be mounted on the conversion device 200.

[0072] [Information processing device 100] 4 is a block diagram showing an example configuration of an information processing device 100 according to an embodiment of the present disclosure. As shown in FIG. 4, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0073] [Communications Department 110] The communication unit 110 is a communication interface that communicates with an external device via a network, either wired or wirelessly, and is realized by, for example, a network interface card (NIC).

[0074] [Storage unit 120] The storage unit 120 is a data readable / writable storage device such as a DRAM, an SRAM, a flash memory, or a hard disk. The storage unit 120 functions as a storage means of the information processing device 100. The storage unit 120 has an image capture device spectral data DB 121, an image capture side light source spectral data DB 122, a display side light source spectral data DB 123, and a sample color spectral data DB 124.

[0075] (Imaging device spectral data DB121) The imaging device spectral data DB 121 holds characteristic data relating to the spectral sensitivity characteristics of the imaging device 300. The imaging device spectral data DB 121 holds, for example, characteristic data of a plurality of imaging devices 300. The information processing device 100 acquires characteristic data of the imaging device 300 actually installed in the information processing system 10 from the imaging device spectral data DB 121.

[0076] At this time, it is desirable that the imaging device spectral data DB 121 comprehensively stores characteristic data of all imaging devices 300 that can be employed in the information processing system 10.

[0077] If the imaging device 300 to be installed in the information processing system 10 is predetermined, the imaging device spectral data DB 121 may simply hold characteristic data of the predetermined imaging device 300.

[0078] (Photographer's light source spectral data DB122) The shooting-side light source spectral data DB 122 holds shooting-side (image-capturing-side) spectral data relating to the spectral distribution characteristics of shooting-side light sources. The shooting-side light source spectral data DB 122 holds, for example, shooting-side spectral data for multiple shooting-side lighting devices 600A. In this case, it is desirable that the shooting-side light source spectral data DB 122 comprehensively holds spectral data for light sources that can serve as shooting-side light sources, such as incandescent lamps, fluorescent lamps, LEDs, and sunlight.

[0079] (Display side light source spectral data DB123) The display-side light source spectral data DB 123 stores display-side (observation-side) spectral data relating to the spectral distribution characteristics of the display-side light source. The display-side light source spectral data DB 123 stores, for example, image-capture-side spectral data of a plurality of display-side illumination devices 600B. In this case, it is desirable that the display-side light source spectral data DB 123 comprehensively stores spectral data of light sources that can serve as display-side light sources, such as incandescent lamps, fluorescent lamps, LEDs, and sunlight.

[0080] 4, the light source spectral data DBs are separated for the image capturing side and the display side, but this is not limiting. For example, if spectral data relating to the same light source is stored on the image capturing side and the display side, the image capturing side light source spectral data DB 122 and the display side light source spectral data DB 123 may be configured as a single DB.

[0081] (Sample color spectrum data DB124) The sample color spectral data DB 124 stores color spectral data relating to the spectral reflectance characteristics of sample colors. The color spectral data includes, for example, the spectral reflectance of each of a plurality of sample colors included in a color sample (color specimen).

[0082] As will be described later, RGB values and XYZ values are calculated based on sample colors. Therefore, it is desirable that the sample colors included in the color spectrum data cover the color gamut required for the conversion process from RGB values to XYZ values. In other words, it is desirable that the color spectrum data include data on a large number of spectral reflectances that cover the color gamut required for the conversion process.

[0083] The sample color spectrum data DB 124 may store color spectrum data for each of a plurality of color samples (color specimens) according to, for example, the type of subject 400. For example, the sample color spectrum data DB 124 stores color spectrum data for each representative object in the natural world, based on color specimens for printing of approximately 1,000 to 2,000 colors.

[0084] In this case, the sample color spectral data DB 124 may be configured to hold color spectral data for sample colors that are densely populated around a color for which color reproduction accuracy is desired to be improved, i.e., for sample colors with a large number of samples around a color for which color reproduction accuracy is desired to be improved. In other words, the sample colors included in the color spectral data may be different for each corresponding object. This allows the information processing device 100 to further improve color reproduction accuracy and to some extent control the bias in color reproduction accuracy due to the sample colors for each object.

[0085] The spectral reflectance included in the color spectral data may be the actual spectral reflectance of the object, or may be a fictitious spectral reflectance (estimated spectral reflectance) created (estimated) as data based on the actual spectral reflectance.

[0086] [Control unit 130] The control unit 130 controls each unit of the information processing device 100. The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), or the like executing a program stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is also realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0087] The control unit 130 includes an ambient light information acquisition unit 131, a spectral data acquisition unit 132, an RGB value calculation unit 133, an XYZ value calculation unit 134, and a conversion information calculation unit 135. Each block constituting the control unit 130 (ambient light information acquisition unit 131 to conversion information calculation unit 135) is a functional block indicating a function of the control unit 130. These functional blocks may be software blocks or hardware blocks. For example, each of the above-described functional blocks may be a software module realized by software (including a microprogram), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The control unit 130 may be configured by functional units different from the above-described functional blocks. The method of configuring the functional blocks is arbitrary.

[0088] The control unit 130 may be configured with functional units different from the above-described functional blocks. Also, some or all of the operations of the blocks (external light information acquisition unit 131 to conversion information calculation unit 135) that configure the control unit 130 may be performed by another device. For example, some or all of the operations of the blocks that configure the control unit 130 may be performed by a control device realized by cloud computing.

[0089] (External light information acquisition unit 131) The external light information acquisition unit 131 acquires shooting-side external light information from the external light sensor 330 of the imaging device 300 via the communication unit 110. The external light information acquisition unit 131 acquires display-side external light information from the external light sensor 510 of the display device 500 via the communication unit 110. The external light information acquisition unit 131 outputs the acquired shooting-side external light information and display-side external light information to the spectral data acquisition unit 132.

[0090] (Spectroscopic data acquisition unit 132) The spectral data acquisition unit 132 acquires the shooting-side spectral data from the shooting-side light source spectral data DB 122 based on the shooting-side external light information acquired from the external light information acquisition unit 131. The spectral data acquisition unit 132 acquires the display-side spectral data from the display-side light source spectral data DB 123 based on the display-side external light information acquired from the external light information acquisition unit 131. Details of how the spectral data acquisition unit 132 acquires the shooting-side spectral data and the display-side spectral data will be described later with reference to FIG. 5.

[0091] (RGB value calculation unit 133) The RGB value calculation unit 133 calculates, for each sample color, the RGB values output by the image capture device 300 under the image capture side light source based on the characteristic data of the image capture device 300, the image capture side spectral data, and the color spectral data of the sample colors.

[0092] The RGB value calculation unit 133 acquires characteristic data of the imaging device 300 from the imaging device spectral data DB 121 according to the imaging device 300 that actually captures the image. The RGB value calculation unit 133 acquires the capturing-side spectral data from the spectral data acquisition unit 132. The RGB value calculation unit 133 acquires color spectral data according to the type of the subject 400 from the sample color spectral data DB 124, for example.

[0093] In this way, the RGB value calculation unit 133 calculates the RGB values by switching the characteristic data according to the image capture device 300. The RGB value calculation unit 133 also calculates the RGB values by switching the color spectral data according to the subject 400.

[0094] The RGB value calculation unit 133 outputs the calculated RGB values to the conversion information calculation unit 135 .

[0095] (XYZ value calculation unit 134) The XYZ value calculation unit 134 calculates the XYZ values under the display-side light source for each sample color based on the display-side spectral data and the color spectral data of the sample colors.

[0096] For example, the XYZ value calculation unit 134 acquires color spectral data according to the type of the subject 400 from the sample color spectral data DB 124. In this way, the XYZ value calculation unit 134 switches the color spectral data according to the subject 400 and calculates the RGB values.

[0097] The XYZ value calculation unit 134 outputs the calculated XYZ values to the conversion information calculation unit 135 .

[0098] (Conversion information calculation unit 135) The conversion information calculation unit 135 calculates conversion information using the RGB values calculated by the RGB value calculation unit 133 and the XYZ values calculated by the XYZ value calculation unit 134. The conversion information is, for example, information for associating RGB values with XYZ values for each sample color included in a color sample. Details of the conversion information and the calculation process of the conversion information will be described later.

[0099] The conversion information calculation unit 135 calculates (updates) the conversion information at a predetermined cycle (every few seconds), for example. The conversion information calculation unit 135 outputs the conversion information to the conversion device 200 via the communication unit 110.

[0100] [Details of the spectroscopic data acquisition unit 132] Next, a detailed configuration example of the spectral data acquisition unit 132 will be described with reference to Fig. 5. Fig. 5 is a diagram showing a configuration example of the spectral data acquisition unit 132 according to an embodiment of the present disclosure. Note that Fig. 5 describes a case where the spectral data acquisition unit 132 estimates the shooting-side light source, but the spectral data acquisition unit 132 also estimates the display-side light source in the same manner.

[0101] The spectral data acquisition unit 132 shown in FIG. 5 includes a filter spectral processing unit 1321 and a comparison estimation processing unit 1322.

[0102] The filter spectral processing unit 1321 acquires spectral data of multiple light sources held in the image-capturing-side spectral data DB as multiple spectral data candidates. The filter spectral processing unit 1321 extracts light of the same wavelength (color component) as that extracted by the ambient light sensor 330 from the spectral data candidates. That is, the filter spectral processing unit 1321 performs the same filter spectral processing as the ambient light sensor 330 on the spectral data candidates, and outputs the processing result to the comparison / estimation processing unit 1322. Note that the filter spectral processing unit 1321 performs the same filter spectral processing as the ambient light sensor 330, for example, by calculation.

[0103] The comparison estimation processing unit 1322 compares the filter spectral processing results for the multiple spectral data candidates with the image-capturing external light information acquired by the external light information acquisition unit 131. Generally, once the type of light source (for example, fluorescent lamp, LED, incandescent lamp, etc.) is determined, the shape of the spectrum is almost determined. Therefore, if the light source is the same, the result of filter spectral processing for the spectral data of the light source will theoretically be the same as the result of filter spectral processing for the actual light source.

[0104] Therefore, the comparison and estimation processing unit 1322 compares the filter spectral processing results for the plurality of spectral data candidates with the image capturing side external light information including the filter spectral processing results for the actual light source.

[0105] The comparison estimation processing unit 1322 estimates that the spectral data candidate corresponding to the filter spectral processing result that is closest to the filter spectral processing result for the actual light source is the imaging-side spectral data, and estimates that the light source corresponding to the imaging-side spectral data is the actual light source.

[0106] The comparison estimation processing unit 1322 compares the two filter spectral processing results by calculating, for example, using a technique such as regression analysis, how closely the filter spectral processing result for the candidate spectral data matches the filter spectral processing result for the actual light source.

[0107] The comparison estimation processing unit 1322 outputs the estimated light source spectral data candidates to the RGB value calculation unit 133 as imaging-side spectral data.

[0108] [Example of conversion information] (conversion factor) Next, an example of conversion information according to an embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a diagram for describing an example of conversion information according to an embodiment of the present disclosure. As described above, the conversion information is calculated by the conversion information calculation unit 135 (see Fig. 4).

[0109] In the example shown in FIG. 6, the conversion information calculation unit 135 compares the RGB values acquired from the RGB value calculation unit 133 with the XYZ values acquired from the XYZ value calculation unit 134, thereby mathematically calculating color coordinate conversion coefficients as conversion information.

[0110] As described above, the RGB value calculation unit 133 calculates the RGB value for each sample color. Fig. 6 shows the sample color, sample color ID, and 4-bit RGB value associated with each other. Furthermore, the XYZ value calculation unit 134 calculates the XYZ value for each sample color. Fig. 6 shows the sample color ID for identifying the sample color associated with the 4-bit RGB value.

[0111] The conversion information calculation unit 135 compares RGB values and XYZ values that have the same sample color ID, and calculates conversion coefficients for converting RGB values to XYZ values. For example, when converting RGB values to XYZ values using a matrix operation, the conversion information calculation unit 135 calculates a conversion matrix for converting RGB values to XYZ values.

[0112] 6, the conversion information calculation unit 135 calculates a conversion matrix x1 that converts the RGB value "2F1" having the sample color ID "0001" into the XYZ value "2D3." The conversion matrix x1 is, for example, a 3×3 matrix.

[0113] Similarly, the conversion information calculation unit 135 calculates the conversion matrices x2 to x2000 corresponding to the sample color IDs "0002" to "2000." In this way, the conversion information calculation unit 135 calculates, for example, the conversion matrices x corresponding to all the sample colors included in the color sample (color specimen).

[0114] The transformation matrix x calculated by the transformation information calculation unit 135 differs for each sample color. Therefore, the transformation information calculation unit 135 calculates, as transformation information, a representative transformation matrix X that can averagely represent the transformation from RGB values to XYZ values for all sample colors using, for example, the least squares method, and outputs the transformation information to the conversion device 200. This allows the conversion device 200 to convert from an RGB image to a converted image using a simple matrix operation, thereby reducing the amount of calculation required for the conversion process. This allows, for example, the conversion device 200 to be configured using simple hardware.

[0115] On the other hand, when all colors are converted using one representative transformation matrix X, variations in conversion accuracy occur depending on the color. One possible method for suppressing such variations is for the conversion device 200 to perform conversion processing using a transformation matrix x calculated for each sample color ID. That is, the conversion information calculation unit 135 generates conversion information using all transformation matrices x as the representative transformation matrix X. However, this method may result in a complex hardware configuration for the conversion device 200.

[0116] As such, as the number of representative transformation matrices increases, the amount of processing required for the transformation process increases. Therefore, the number of representative transformation matrices to be included in the transformation information can be set appropriately depending on, for example, the configuration of the transformation device 200, the required processing time, the required transformation accuracy, etc.

[0117] Note that even when calculating a plurality of representative transformation matrices (coefficients), the transformation information calculation unit 135 does not calculate a representative transformation matrix (coefficient) for each pixel, but sets several points according to color, brightness, etc., and calculates a representative transformation matrix (coefficient) for each of the set points. That is, the transformation information calculation unit 135 divides the sample colors into a plurality of groups according to color, brightness, etc., and calculates a representative transformation matrix for each group.

[0118] In this case, the conversion device 200 selects a representative transformation matrix according to each pixel value of the RGB image and performs transformation processing for each pixel. At this time, the conversion device 200 may perform corrections to the representative transformation matrix, such as weighting the selected representative transformation matrix according to the pixel value.

[0119] (conversion table) It should be noted that the conversion information calculated by the conversion information calculation unit 135 is not limited to a conversion matrix (conversion coefficients). For example, the conversion information calculation unit 135 may calculate a conversion table as the conversion information.

[0120] 7 is a diagram illustrating an example of conversion information calculated by the conversion information calculation unit 135 according to an embodiment of the present disclosure. As shown in Fig. 7, the conversion information calculation unit 135 calculates, as conversion information, a conversion table in which 4-bit RGB values and 4-bit XYZ values are associated with each other for each sample color.

[0121] The conversion information calculation unit 135 may calculate a conversion table that associates RGB values with XYZ values for all sample colors, or may calculate a conversion table that associates RGB values with XYZ values discretely for some sample colors. The number of sample colors included in the conversion table may be set appropriately depending on, for example, the configuration of the conversion device 200, the required processing time, the required conversion accuracy, etc.

[0122] [Conversion process example] Here, an example of the conversion process executed by the conversion device 200 when the conversion information is the above-described conversion table will be described with reference to Fig. 8. Fig. 8 is a diagram for describing an example of the conversion process by the conversion device 200 according to an embodiment of the present disclosure. Note that the conversion process is executed by the conversion processing unit 232 of the conversion device 200. Also, it is assumed that the conversion table is acquired from the information processing device 100 by the conversion information acquisition unit 231 (see Fig. 3) and stored in the storage unit 220 (see Fig. 3).

[0123] As shown in FIG. 8, the conversion processing unit 232 includes a read control unit 2321 and an interpolation processing unit 2322.

[0124] The read control unit 2321 controls the read processing of the conversion table according to each pixel value of the RGB image. If there is an RGB value that is the same as the pixel value, the read control unit 2331 performs the read processing of the conversion table so as to output the XYZ value that corresponds to the RGB value.

[0125] On the other hand, if there is no RGB value identical to the pixel value, the read control unit 2331 performs a read process of the conversion table so as to output XYZ values corresponding to RGB values close to the pixel value. In this case, the output XYZ values may be one or multiple (for example, two).

[0126] The interpolation processing unit 2322 performs interpolation processing using the XYZ values output by the read control unit 2321, converts the pixel values of the RGB image into XYZ values, and generates a converted image.

[0127] If interpolation is not required, that is, if the conversion table contains the same RGB value as the pixel value, the interpolation processing unit 2322 does not perform interpolation, but generates a converted image using the XYZ values corresponding to the RGB value as the pixel value.

[0128] For example, suppose the pixel value of a predetermined pixel in an RGB image is "004." In this case, the same RGB value is included in the conversion table, as shown in FIG. 8. Therefore, the read control unit 2321 performs a read process to output an XYZ value of "005" corresponding to the RGB value of "004." The interpolation processing unit 2322 generates a converted image in which the read XYZ value of "005" is used as the pixel value of the predetermined pixel.

[0129] On the other hand, if there is no RGB value corresponding to the pixel value of the RGB image, the interpolation processing unit 2322 performs interpolation processing on the output XYZ values to generate a converted image.

[0130] For example, suppose the pixel value of a specific pixel in an RGB image is "002." As shown in FIG. 8, the conversion table does not include the same RGB value. In this case, the read control unit 2321 performs a read process to output, for example, an XYZ value "001" corresponding to the RGB value "000" and an XYZ value "005" corresponding to the RGB value "004." The interpolation processing unit 2322 performs an interpolation process based on the read XYZ values "001" and "005," and sets the calculated XYZ values as the pixel value of the specific pixel.

[0131] In this way, the conversion processing unit 232 refers to the conversion table for each pixel of the RGB image and performs interpolation processing as necessary to generate a converted image in which each pixel value is converted into an XYZ value.

[0132] <<3. Information Processing>> <3.1. Conversion information calculation process> Next, a description will be given of information processing executed by the information processing device 100 according to an embodiment of the present disclosure. Fig. 9 is a flowchart showing the flow of an example of information processing executed by the information processing device 100 according to an embodiment of the present disclosure.

[0133] The information processing device 100 executes a conversion information calculation process for calculating conversion information as information processing. The information processing device 100 executes the conversion information calculation process, for example, at a predetermined interval.

[0134] 9, the information processing device 100 acquires external light information (step S101). The information processing device 100 acquires imaging-side external light information from the external light sensor 330 and acquires display-side external light information from the external light sensor 510.

[0135] Next, the information processing device 100 selects spectral data (step S102). Based on the imaging-side external light information, the information processing device 100 selects imaging-side spectral data corresponding to the imaging-side light source from the imaging-side light source spectral data DB 122. Based on the display-side external light information, the information processing device 100 also selects display-side spectral data corresponding to the display-side light source from the display-side light source spectral data DB 123.

[0136] The information processing device 100 calculates RGB values (step S103). The information processing device 100 calculates RGB values for each of a plurality of sample colors based on the characteristic data of the imaging device 300, the imaging-side spectral data, and the color spectral data.

[0137] The information processing device 100 calculates the XYZ values (step S104). The information processing device 100 calculates the XYZ values for each of the plurality of sample colors based on the display-side spectral data and the color spectral data.

[0138] The information processing device 100 calculates conversion information based on the RGB values and XYZ values (step S105). The conversion information is information used in the conversion process from RGB values to XYZ values executed by the conversion device 200, and includes, for example, conversion coefficients such as a conversion matrix and a conversion table.

[0139] <3.2. Conversion process> Next, a description will be given of the conversion process performed by the conversion device 200. Fig. 10 is a flowchart showing an example of the flow of the conversion process performed by the conversion device 200 according to an embodiment of the present disclosure. The conversion device 200 performs the conversion process while the imaging device 300 is capturing an image.

[0140] First, the conversion device 200 acquires the conversion information from the information processing device 100 (step S201). The conversion device 200 may acquire the conversion information before the imaging device 300 starts imaging.

[0141] Next, the conversion device 200 acquires an RGB image from the imaging device 300 (step S202). The conversion device 200 converts each pixel of the acquired RGB image from an RGB value to an XYZ value (step S203). The conversion device 200 calculates the pixel value of the XYZ value by multiplying each pixel value of the RGB image by a conversion coefficient. The conversion coefficient is included in, for example, the conversion information.

[0142] Conversion device 200 outputs the converted image generated in step S203 to display device 500 (step S204). As a result, the converted image is displayed on display device 500.

[0143] Next, the conversion device 200 determines whether the conversion information has been updated (step S205). For example, the conversion device 200 determines whether the conversion information has been updated depending on whether the conversion information has been acquired from the information processing device 100. Alternatively, the conversion device 200 may determine whether the conversion information has been updated depending on whether a predetermined period has elapsed.

[0144] If it is determined that the conversion information has been updated (step S205; Yes), the conversion device 200 updates the change information stored in the storage unit 220.

[0145] On the other hand, if it is determined that the conversion information has not been updated (step S205; No), the conversion device 200 determines whether or not the imaging of the imaging device 300 has ended (step S206). The conversion device 200 determines that the imaging has ended, for example, when it receives a notification that imaging has ended from the imaging device 300 or when communication with the imaging device 300 has ended.

[0146] If it is determined that the imaging has not ended (step S206; No), the process returns to step S202. On the other hand, if it is determined that the imaging has ended (step S206; Yes), the process ends.

[0147] As described above, in the information processing system 10 according to this embodiment, the information processing device 100 calculates RGB values when the sample color is captured by the imaging device 300 under a capture-side light source. The information processing device 100 also calculates XYZ values when the sample color is displayed on the display device 500 under a display-side light source. The information processing device 100 calculates conversion information for converting the calculated RGB values into XYZ values. The conversion device 200 uses this conversion information to convert the RGB image captured by the imaging device 300 into a converted image of XYZ values, and displays it on the display device 500. This allows the information processing system 10 to maintain color coordinates from capture to display. Therefore, the information processing system 10 can more easily reproduce the color of the subject 400 on the capture side on the display device 500 (monitor).

[0148] Furthermore, the information processing system 10 according to this embodiment is provided with external light sensors 330, 510 on the image capturing side and the display side to estimate the spectrum of the light source and acquire spectral data. Therefore, the observer can visually recognize the subject 400 displayed on the display device 500 without being aware of the spectrum of the light source or changes in the light source.

[0149] Furthermore, the information processing system 10 according to this embodiment only needs to update the conversion information when the spectrum of the light source changes, for example, every few seconds. In this way, the processing load for calculating the conversion information is low, so the information processing device 100 can be realized by software.

[0150] Furthermore, the information processing system 10 according to this embodiment calculates conversion information based on sample colors. This allows for a reduction in the amount of calculation per pixel compared to when conversion information is calculated for each pixel captured by the imaging device 300. This allows for a reduction in the hardware performance required to build the information processing system 10.

[0151] Furthermore, in the information processing system 10 according to this embodiment, the information processing device 100 is realized, for example, by cloud computing, and the conversion device 200 is located on the display side. Therefore, the imaging device 300 and the display device 500 can be easily connected via an existing network, for example, the Internet. In this manner, the information processing system 10 can be easily applied to a system in which captured images are viewed remotely. However, as described above, the information processing device 100 calculates RGB values using characteristic data of the imaging device 300. Therefore, when the imaging device 300 is located remotely from the information processing device 100, the display device 300, etc., the information processing device 100 acquires information about the imaging device 300, such as model information of the imaging device 300, from the imaging side (e.g., the imaging device 300) as, for example, metadata.

[0152] Although the above-described information processing system 10 illustrates a case where a captured image is transmitted from the imaging side to the display side, the present invention is not limited to this. For example, if the information processing system 10 includes an imaging device on the display side and a display device on the imaging side, the colors of the captured subject can be more easily reproduced and displayed even when images are transmitted and received bidirectionally.

[0153] <<4. Modifications>> <4.1. First modified example> In the above-described embodiment, the information processing device 100 estimates the light source based on the external light information acquired from the external light sensors 330 and 510, but this is not limiting. For example, the light source may be designated by a user who configures the information processing system 10, an image capturer who captures images, or an observer who views the converted image on the display device 500. In this case, the information processing device 100 acquires spectral data of the designated light source from the spectral data DB.

[0154] The specified light source may be different from the actual light source. For example, if the actual light source on the shooting side is a lighting device, the user (e.g., observer) can specify sunlight as the light source on the shooting side, thereby enabling the user to check the color of the subject 400 when it is outdoors.

[0155] By making it possible to specify a light source in this way, the information processing system 10 can present to the observer, for example, an image that would have been captured in an environment different from the actual capturing environment.

[0156] If the observer or the like specifies the light source, the external light sensors 330 and 510 may be omitted.

[0157] <4.2. Second Modification> In the above-described embodiment, the information processing device 100 is a device different from the conversion device 200. In this case, for example, the conversion device 200 may be arranged on the same display side as the display device 500. Furthermore, the information processing device 100 may be realized by, for example, cloud computing.

[0158] On the other hand, the information processing device 100 and the conversion device 200 may be located in the same place. For example, the functions of the information processing device 100 and the conversion device 200 may be realized by a single information processing device 700.

[0159] Fig. 11 is a block diagram showing a configuration example of an information processing device 700 according to a second modified example of the embodiment of the present disclosure. The information processing device 700 shown in Fig. 11 includes a communication unit 110, a storage unit 720, and a control unit 730. Note that, among the configuration of the information processing device 700 shown in Fig. 11, the same components as those of the information processing device 100 and the conversion device 200 are designated by the same reference numerals, and description thereof will be omitted.

[0160] The storage unit 720 stores, for example, the DBs stored in the storage unit 120 of the information processing device 100 and the information stored in the storage unit 220 of the conversion device 200.

[0161] The control unit 730 has, for example, each function of the control unit 130 of the information processing device 100 and each function of the control unit 230 of the conversion device 200.

[0162] In this way, the functions of the information processing device 100 and the conversion device 200 can be realized as a single device. For example, by placing the information processing device 700 on the same display side as the display device 500, it becomes easier to respond to instructions from the observer, such as the observer's specification of a light source (see the first modified example).

[0163] 11 shows a case where the information processing device 700 is arranged on the same display side as the display device 500, but the present invention is not limited to this. For example, the information processing device 700 may be realized by cloud computing.

[0164] <4.3.Third Modification> In the above-described embodiment and modified examples, the observer is on the display side, but this is not limiting. For example, there may also be an observer on the shooting side who directly observes the subject 400. Such a case will be described as a third modified example.

[0165] For example, there may be cases where the color of a prototype produced in a factory is to be checked simultaneously by the manufacturer in the factory and the orderer who is located away from the factory. In this case, it is desirable to match the color of the actual prototype with the color of the prototype displayed on the display device 500.

[0166] As such, it may be desirable to make the color of the subject 400 on the shooting side the same as the color of the subject 400 displayed on the display device 500. However, because there is variation in the spectral characteristics of humans, even if absolute color coordinates are maintained and displayed as in the information processing system 10 of the above embodiment, the shooting-side observer and the display-side observer may judge the colors to be different. Correcting such variation in the spectral characteristics of observers is difficult, and a mechanism for reducing the effects of variation in spectral characteristics using a method other than conversion processing is required.

[0167] For example, when measuring the colors of two objects with a colorimeter, the measurement results may show the same color even if the two objects have different spectral characteristics (a metameric pair). Alternatively, the colorimeter may show the same color even if the spectral characteristics of two light sources are different. Although a colorimeter mimics the spectral characteristics of the human eye on average, the spectral characteristics of the actual human eye vary from person to person. Therefore, in a metameric pair, different observers may not see the same color due to this individual variation.

[0168] One method for reducing the effects of this variation is to align the spectral characteristics of the imaging-side light source (imaging-side illumination device 600A) with the all-white spectral characteristics of the display device 500. Here, the all-white spectral characteristics refer to the spectral characteristics when a white image is displayed on the display device 500.

[0169] In this way, by aligning the all-white spectral characteristics of the shooting-side light source and the display device 500, the spectral characteristics seen by the shooting-side observer will have substantially the same shape as the spectral characteristics reproduced by the display device 500. Therefore, it is possible to reduce the influence of observer variability between the shooting-side observer and the display-side observer.

[0170] Assuming that the influence of this observer variation is metamerism, the influence of this variation is reduced as the spectral characteristics of the image-capturing light source and the display device 500 become closer. For example, by using the same device as the display device 500 as the image-capturing light source, it is possible to more easily construct an information processing system 10 that is less influenced by this variation.

[0171] In this way, for example, when constructing an information processing system 10 including a shooting-side light source, the influence of observer variation can be reduced by selecting the shooting-side light source in advance so that the spectral characteristics of the shooting-side light source and the all-white spectral characteristics of the display device 500 are close to each other.

[0172] As a specific method for aligning the spectral characteristics of the imaging-side light source (imaging-side illumination device 600A) with the all-white spectral characteristics of the display device 500, for example, a method of using the same device as the display device 500 as the imaging-side light source can be mentioned. In this case, by displaying white (all-white) on the same device as the display device 500, the device can be used as lighting (imaging-side light source).

[0173] In this way, by using the same device as the display device 500 as the imaging-side light source, it is possible to more easily build a system that can reduce the influence of observer variability.

[0174] Alternatively, for example, the information processing device 100 may acquire information from the display device 500 and present the acquired information to the shooting-side observer. The information processing device 100 may present information related to, for example, a display mounted on the imaging device 300 or a display device arranged on the shooting side. Alternatively, the information processing device 100 may present such information to the shooting-side observer by audio or the like. In this way, the information processing device 100 can prompt the shooting-side observer to change the shooting-side light source to a light source having spectral characteristics close to the all-white spectral characteristics of the display device 500.

[0175] The information processing device 100 acquires, for example, identification information for identifying the display device 500, such as a model number, from the display device 500. The information processing device 100 can present the acquired identification information to the capturing-side observer.

[0176] Alternatively, the information processing device 100 may present presentation information acquired using the identification information to the capturing-side observer. The presentation information may include, for example, at least one of the following information: Spectral information regarding the all-white spectral characteristics of the display device 500; Product information about the display device 500, such as product name, manufacturer, etc. Light source information relating to a light source having spectral characteristics close to the full-white spectral characteristics of the display device 500 (e.g., information relating to a lighting device such as a fluorescent lamp, an incandescent lamp, or an LED)

[0177] In addition to information about the lighting device, the light source information may also include information about a display device having an all-white spectral characteristic similar to that of the display device 500. In this way, by including in the presentation information information about a display device other than the display device 500 that is actually placed on the display side, the number of light sources that can be selected as the imaging-side light source increases.

[0178] The information processing device 100 acquires presentation information corresponding to the display device 500, for example, by searching a pre-constructed database. This database may be constructed by a system administrator, for example, or may be constructed based on information collected from product homepages or the like.

[0179] It should be noted that although the information processing device 100 presents information to the capturing-side observer here, this is not limiting. For example, the information processing device 100 may present information to the display-side observer, or may present information to both the display-side and capturing-side observers. For example, in cases where there is no device on the capturing side that presents information, the information processing device 100 may display such information on the display device 500. In this case, for example, the display-side observer who has received the presented information can communicate the information to the capturing-side observer, thereby changing the capturing-side light source.

[0180] Furthermore, although the shooting-side light source is changed to match the all-white spectral characteristics of the display device 500 in this example, the present invention is not limited to this. For example, the display device 500 may be changed to match the spectral characteristics of the shooting-side light source. For example, the display device 500 may be changed to the same display as one that can be prepared on the shooting side. In this case, the information processing device 100 presents information about the shooting-side light source to the display-side observer.

[0181] Alternatively, the information processing device 100 may present both information about the display device 500 and information about the shooting-side light source to both the display-side and shooting-side observers. In this case, the information processing device 100 can prompt the observer to change at least one of the devices (display device or lighting device) on the shooting side or the display side.

[0182] <4.4. Fourth Modification> In the above embodiment and modified examples, pixel values of an RGB image are converted from RGB values to XYZ values, which may result in errors during the conversion. For example, in business applications, there may be cases where information is required about the degree of error that occurs and the reliability of color reproduction.

[0183] Therefore, in the fourth modified example, an information processing device 100A that calculates such a conversion error will be described. Fig. 12 is a block diagram showing an example configuration of the information processing device 100A according to the fourth modified example of the embodiment of the present disclosure. Note that the same components as those of the information processing device 100 shown in Fig. 4 are assigned the same reference numerals and descriptions thereof will be omitted.

[0184] 12, control unit 130A of information processing device 100A has error calculation unit 136. Error calculation unit 136 estimates an error that occurs during conversion processing by conversion device 200, and causes display device 500 to display the estimation result, for example.

[0185] 13 is a block diagram showing a configuration example of the error calculation unit 136 according to the fourth modified example of the embodiment of the present disclosure. The error calculation unit 136 shown in FIG. 13 includes a conversion processing unit 1361, an error estimation unit 1362, and a display information generation unit 1363.

[0186] The conversion processing unit 1361 uses the RGB values calculated by the RGB value calculation unit 133 and the conversion information calculated by the conversion information calculation unit 135 to convert the RGB values calculated by the RGB value calculation unit 133 into XYZ values.

[0187] The error estimation unit 1362 compares the XYZ values converted by the conversion processing unit 1361 (hereinafter also referred to as converted XYZ values) with the XYZ values calculated by the XYZ value calculation unit 134 (hereinafter also referred to as calculated XYZ values) to estimate the error.

[0188] The error estimation unit 1362 calculates, for example, the difference (Δx) in the X component and the difference (Δy) in the Y component between the converted XYZ value and the calculated XYZ value as the error. Alternatively, the error estimation unit 1362 may use a color difference calculation program such as DE2000 to calculate the average value, maximum value, and standard deviation of all sample colors included in the color sample (color specimen) as the error.

[0189] The error estimation unit 1362 outputs the calculated error to the display information generation unit 1363 .

[0190] The display information generating unit 1363 generates display information to be displayed on the display device 500 based on the information about the error acquired from the error estimating unit 1362 .

[0191] The display information generating unit 1363 generates, as display information, at least one of the values calculated as errors by the error estimating unit 1362, such as the above-described Δx, Δy, average value, maximum value, and standard deviation. Alternatively, the display information generating unit 1363 may generate, as display information, a distribution diagram in which the calculated standard deviation σ is plotted on an xy chromaticity diagram, as shown in, for example, FIG. 14. FIG. 14 is a diagram illustrating an example of display information according to a fourth modified example of the embodiment of the present disclosure. In this way, the display information generating unit 1363 generates a distribution diagram as display information, allowing the viewer to grasp the coordinate reliability for each color at a glance.

[0192] Alternatively, the display information generation unit 1363 may generate display information in which error information is superimposed on the converted image to be displayed on the display device 500. Fig. 15 is a diagram showing another example of display information according to the fourth modified example of the embodiment of the present disclosure.

[0193] 15, for example, the display information generation unit 1363 acquires a converted image from the conversion device 200, and generates display information by superimposing and displaying the standard deviation σ on the converted image as a specific pattern. For example, the display information generation unit 1363 compares each pixel value of the converted image with the converted XYZ value (or the calculated XYZ value), determines the standard deviation σ corresponding to each pixel, and superimposes the determined standard deviation σ on the converted image as a specific pattern.

[0194] The display information generating unit 1363 generates display information in which the standard deviation σ is superimposed as a specific pattern, so that the viewer can grasp the coordinate reliability on the converted image at a glance.

[0195] Here, the display information generation unit 1363 acquires the converted image from the conversion device 200 and generates the display information, but this is not limiting. For example, the conversion device 200 may acquire information related to the error from the error calculation unit 136 and generate the display information.

[0196] Furthermore, the information processing device 100 may display the generated display information directly on the display device 500, or may display it via the conversion device 200. Furthermore, the display information may be presented to the shooting-side observer not only on the display device 500 but also on the display of the imaging device 300, for example.

[0197] <<5. Other embodiments>> The above-described embodiment and each modified example are merely examples, and various modifications and applications are possible.

[0198] For example, the control device that controls the information processing device 100 and the conversion device 200 of this embodiment may be realized by a dedicated computer system or a general-purpose computer system.

[0199] For example, a communication program for executing the above-described operations is stored in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk and distributed. Then, for example, the program is installed in a computer and the above-described processing is executed to configure a control device. In this case, the control device may be a device external to the information processing device 100 or the conversion device 200 (for example, a personal computer). Alternatively, the control device may be a device internal to the information processing device 100 or the conversion device 200 (for example, the control units 130 and 230).

[0200] The communication program may also be stored in a disk device provided in a server device on a network such as the Internet, and may be downloaded to a computer. The above-mentioned functions may also be realized by cooperation between an OS (Operating System) and application software. In this case, the parts other than the OS may be stored on a medium and distributed, or may be stored in a server device and downloaded to a computer.

[0201] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0202] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. This distribution and integration configuration may also be performed dynamically.

[0203] The above-described embodiments can be combined as appropriate within the scope of the present invention without causing any inconsistency in the processing content. The order of the steps shown in the sequence diagrams of the above-described embodiments can be changed as appropriate.

[0204] Furthermore, for example, this embodiment can also be implemented as any configuration that constitutes an apparatus or system, such as a processor as a system LSI (Large Scale Integration), a module using multiple processors, a unit using multiple modules, a set in which other functions are added to a unit, etc. (i.e., a configuration of a part of an apparatus).

[0205] In this embodiment, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0206] Furthermore, for example, this embodiment can be configured as a cloud computing system in which one function is shared and processed jointly by a plurality of devices via a network.

[0207] <<6. Conclusion>> Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0208] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0209] The present technology can also be configured as follows. (1) Acquire characteristic data relating to the spectral sensitivity characteristics of the imaging device; acquiring imaging-side spectral data relating to the spectral distribution characteristics of a light source in an imaging environment in which the imaging device captures an image; acquiring color spectral data relating to the spectral reflectance characteristics of a predetermined color; calculating RGB values output by the imaging device when imaging the predetermined color using the characteristic data, the imaging-side spectral data, and the color spectral data; acquiring display-side spectral data relating to the spectral distribution characteristics of a light source in a display environment in which the image data captured by the imaging device is displayed on a display device; calculating XYZ values when the predetermined color is displayed on the display device using the display-side spectral data and the color spectral data; a control unit that calculates conversion information for converting the RGB values into the XYZ values; An information processing device comprising: (2) The information processing device described in (1), wherein the control unit selects the display-side spectral data of the light source in the display environment from among a plurality of display-side spectral data corresponding to each of a plurality of light sources, based on display-side external light information acquired by a sensor in the display environment. (3) The information processing device described in (2), wherein the control unit selects the display-side spectral data of the light source in the display environment based on a comparison result between the result of the filter spectral processing by the sensor and the result of the filter spectral processing on the multiple display-side spectral data corresponding to each of the multiple light sources. (4) The information processing device described in any one of (1) to (3), wherein the control unit selects the imaging-side spectral data of the light source in the imaging environment from among a plurality of imaging-side spectral data corresponding to each of a plurality of light sources based on an instruction from a user. (5) The information processing device described in any one of (1) to (4), wherein the control unit selects the imaging-side spectral data of the light source in the imaging environment from among a plurality of imaging-side spectral data corresponding to each of a plurality of light sources, based on imaging-side external light information acquired by a sensor in the imaging environment. (6) The information processing device described in (5), wherein the control unit selects the image-capturing side spectral data of the light source in the image-capturing environment based on a comparison result between the result of the filter spectral processing by the sensor and the result of the filter spectral processing on the multiple image-capturing side spectral data corresponding to each of the multiple light sources. (7) The information processing device according to any one of (1) to (6), wherein the control unit generates presentation information for presenting information about the light source to be placed in the imaging environment in accordance with the display device. (8) The information processing device according to any one of (1) to (7), wherein the spectral distribution of the light source in the imaging environment and the spectral distribution of the display device are substantially the same. (9) The information processing device according to any one of (1) to (8), wherein the control unit calculates an error when the RGB values are converted to the XYZ values based on the conversion information, and generates presentation information for presenting information regarding the error. (10) The information processing device according to (9), wherein the control unit calculates the error by comparing the XYZ values with converted XYZ values obtained by converting the RGB values using the conversion information. (11) The information processing device according to any one of (1) to (10), wherein the control unit converts each pixel value of the captured image captured by the imaging device from the RGB value to the XYZ value using the conversion information. (12) Acquire characteristic data relating to the spectral sensitivity characteristics of the imaging device; acquiring imaging-side spectral data relating to the spectral distribution characteristics of a light source in an imaging environment in which the imaging device captures an image; acquiring color spectral data relating to the spectral reflectance characteristics of a predetermined color; calculating RGB values output by the imaging device when imaging the predetermined color using the characteristic data, the imaging-side spectral data, and the color spectral data; acquiring display-side spectral data relating to the spectral distribution characteristics of a light source in a display environment in which the image data captured by the imaging device is displayed on a display device; calculating XYZ values when the predetermined color is displayed on the display device using the display-side spectral data and the color spectral data; calculating conversion information for converting the RGB values into the XYZ values; Information processing methods. [Explanation of symbols]

[0210] 10 Information Processing Systems 100 Information processing device 110, 210, 310 Communications Department 120, 220 storage section 130, 230 control unit 131 External light information acquisition unit 132 Spectroscopic data acquisition unit 133 RGB value calculation section 134 XYZ value calculation section 135 Conversion information calculation unit 200 Conversion Device 231 Conversion information acquisition unit 232 Conversion processing section 233 Display control unit 300 Imaging device 320 Imaging unit 330, 510 ambient light sensor 400 subjects 500 display device 600 lighting equipment

Claims

1. Acquire characteristic data relating to the spectral sensitivity characteristics of the imaging device; acquiring imaging-side spectral data relating to the spectral distribution characteristics of a light source in an imaging environment in which the imaging device captures an image; acquiring color spectral data relating to the spectral reflectance characteristics of a predetermined color; calculating RGB values output by the imaging device when imaging the predetermined color using the characteristic data, the imaging-side spectral data, and the color spectral data; acquiring display-side spectral data relating to the spectral distribution characteristics of a light source in a display environment in which the image data captured by the imaging device is displayed on a display device; calculating XYZ values when the predetermined color is displayed on the display device using the display-side spectral data and the color spectral data; a control unit that calculates conversion information for converting the RGB values into the XYZ values by associating the RGB values with the XYZ values of the predetermined color; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the control unit selects the display-side spectral data of the light source in the display environment from among a plurality of display-side spectral data corresponding to a plurality of light sources, based on display-side external light information acquired by a sensor in the display environment.

3. 3. The information processing device according to claim 2, wherein the control unit selects the display-side spectral data of the light source in the display environment based on a comparison result between a result of the filter spectral processing by the sensor and a result of the filter spectral processing on a plurality of the display-side spectral data corresponding to each of a plurality of light sources.

4. The information processing apparatus according to claim 1 , wherein the control unit selects the image-capturing-side spectral data of the light source in the image-capturing environment from among a plurality of image-capturing-side spectral data corresponding to a plurality of light sources, based on an instruction from a user.

5. 2. The information processing device according to claim 1, wherein the control unit selects the image-capturing-side spectral data of the light source in the image-capturing environment from among a plurality of image-capturing-side spectral data corresponding to a plurality of light sources, based on image-capturing-side external light information acquired by a sensor in the image-capturing environment.

6. 6. The information processing device according to claim 5, wherein the control unit selects the image-capturing-side spectral data of the light source in the image-capturing environment based on a comparison result between a result of the filter spectral processing by the sensor and a result of the filter spectral processing on a plurality of the image-capturing-side spectral data corresponding to each of a plurality of light sources.

7. 2. The information processing device according to claim 1, wherein the control unit generates presentation information including at least one of spectral information regarding an all-white spectral characteristic of the display device, product information regarding the display device, and light source information regarding a light source having spectral characteristics corresponding to the all-white spectral characteristic of the display device.

8. The information processing apparatus according to claim 1 , wherein the spectral distribution of the light source in the image capturing environment and the spectral distribution of the display device are substantially the same.

9. The information processing apparatus according to claim 1 , wherein the control unit calculates an error when the RGB values are converted into the XYZ values based on the conversion information, and generates presentation information for presenting information relating to the error.

10. The information processing apparatus according to claim 9 , wherein the control unit calculates the error by comparing the XYZ values with converted XYZ values obtained by converting the RGB values using the conversion information.

11. The information processing device described in Claim 1, wherein the control unit converts each pixel value of the image captured by the imaging device from the RGB value to the XYZ value using the conversion information.

12. Acquire characteristic data relating to the spectral sensitivity characteristics of the imaging device; acquiring imaging-side spectral data relating to the spectral distribution characteristics of a light source in an imaging environment in which the imaging device captures an image; acquiring color spectral data relating to the spectral reflectance characteristics of a predetermined color; calculating RGB values output by the imaging device when imaging the predetermined color using the characteristic data, the imaging-side spectral data, and the color spectral data; acquiring display-side spectral data relating to the spectral distribution characteristics of a light source in a display environment in which the image data captured by the imaging device is displayed on a display device; calculating XYZ values when the predetermined color is displayed on the display device using the display-side spectral data and the color spectral data; calculating conversion information for converting the RGB values into the XYZ values by associating the RGB values with the XYZ values of the predetermined color; Information processing methods.

Images