Processing device, setting device, polyhedron, and program

Abstract

This processing device is provided with a virtual space construction unit and a determination unit. The virtual space construction unit disposes, in a target virtual space in which at least one virtual light source is set, a prescribed virtual member having the same shape as a prescribed member that is a polyhedron disposed in a target real space. The determination unit determines the settings of the at least one virtual light source that enable the reproduction of the light environment of the target real space in the target virtual space, on the basis of a camera image that shows a plurality of faces of the polyhedron in the light environment of the target real space and a virtual camera image that shows the prescribed virtual member as illuminated by the at least one virtual light source in the target virtual space.

Description

Processing device, setting device, polyhedron, and program

[0001] The present disclosure relates to a technology for reproducing a lighting environment in a real space in a virtual space.

[0002] Patent Document 1 describes a technique for estimating the position of a light source in an illumination space.

[0003] JP 2008-107087 A

[0004] A processing device, a setting device, a polyhedron, and a program are disclosed. In one embodiment, the processing device includes a virtual space construction unit and a determination unit. The virtual space construction unit places a virtual predetermined member having the same shape as a predetermined member, which is a polyhedron, to be placed in a target real space, in a target virtual space in which at least one virtual light source is set. The determination unit determines setting contents of the at least one virtual light source that can reproduce the lighting environment of the target real space in the target virtual space, based on a camera image that captures multiple faces of the polyhedron under a lighting environment of the target real space and a virtual camera image that captures the virtual predetermined member illuminated by the at least one virtual light source in the target virtual space.

[0005] In one embodiment, the setting device includes a setting unit that sets at least one virtual light source in the virtual space with the setting content determined by the determination unit included in the processing device.

[0006] In one embodiment, the polyhedron is a polyhedron used to set at least one virtual light source and has four or more faces visible to the camera.

[0007] In one embodiment, the program causes a computer device to function as the processing device.

[0008] In one embodiment, the program causes a computer device to function as the setting device.

[0009] FIG. 1 is a schematic diagram showing an example of the configuration of a processing device. FIG. 1 is a schematic diagram showing an example of a target real space and a target virtual space. FIG. 2 is a schematic diagram showing an example of a predetermined member. FIG. 3 is a schematic diagram showing an example of a predetermined member. FIG. 4 is a schematic diagram showing an example of a virtual predetermined member. FIG. 5 is a schematic diagram showing an example of the arrangement of a virtual light source. FIG. 6 is a schematic diagram showing an example of a processing device. FIG. 7 is a flowchart showing an example of the operation of the processing device. FIG. 8 is a flowchart showing an example of an XYZ orthogonal coordinate system set in the target virtual space. FIG. 9 is a flowchart showing an example of the operation of the processing device. FIG. 10 is a flowchart showing an example of the operation of the processing device. FIG. 11 is a schematic diagram showing an example of the configuration of a setting device.

[0010] 1 is a schematic diagram showing an example of the configuration of a processing device 1. The processing device 1 is, for example, a computer device. The processing device 1, for example, provides a plurality of virtual light sources in a virtual space on a computer and determines the settings of each virtual light source. The processing device 1 then sets each virtual light source according to the determined settings, thereby reproducing the lighting environment of the real space in the virtual space.

[0011] 1 , the processing device 1 includes, for example, a control unit 2, a storage unit 3, a communication unit 4, an input unit 5, and a display unit 6. The processing device 1 can also be considered, for example, a processing circuit. Note that the processing device 1 does not necessarily have to include at least one of the communication unit 4, the input unit 5, and the display unit 6.

[0012] The communication unit 4 can communicate with devices external to the processing device 1 (also referred to as external devices). The communication unit 4 can also be considered a communication circuit. The communication unit 4 may perform at least one of wired communication and wireless communication with at least one external device. The communication unit 4 may communicate directly with the external device, or may communicate with the external device via a network including at least one of a local area network (LAN) and the Internet. The at least one communication standard that the communication unit 4 complies with may include at least one of Bluetooth (registered trademark), USB (Universal Serial Bus), Wi-Fi, and Ethernet.

[0013] The control unit 2 can control the other components of the processing device 1 to provide overall management of the operation of the processing device 1. The control unit 2 can also be referred to as, for example, a control circuit. The control unit 2 can, for example, reproduce a lighting environment in a real space in a virtual space. The control unit 2 includes at least one processor to provide control and processing capabilities for performing various functions, as described in more detail below.

[0014] According to various embodiments, the at least one processor may be implemented as a single integrated circuit (IC) or as multiple communicatively connected integrated circuits ICs and / or discrete circuits. The at least one processor may be implemented according to various known techniques.

[0015] In one embodiment, a processor includes one or more circuits or units configured to perform one or more data computational procedures or processes, for example, by executing instructions stored in associated memory. In other embodiments, a processor may be firmware (e.g., discrete logic components) configured to perform one or more data computational procedures or processes.

[0016] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors, programmable logic devices, field programmable gate arrays, or any combination of these devices or configurations, or other known devices and configurations, to perform the functions described below.

[0017] The control unit 2 may include, for example, a CPU (Central Processing Unit) as a processor. The storage unit 3 may include a non-transitory recording medium readable by the CPU of the control unit 2, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 3 stores, for example, a program 30 for controlling the processing device 1. Various functions of the control unit 2 are realized, for example, by the CPU of the control unit 2 executing the program 30 in the storage unit 3.

[0018] The configuration of the control unit 2 is not limited to the above example. For example, the control unit 2 may include multiple CPUs. The control unit 2 may also include at least one DSP (Digital Signal Processor). All or some of the functions of the control unit 2 may be realized by a hardware circuit that does not require software to realize the function. The storage unit 3 may also include a computer-readable non-transitory recording medium other than ROM and RAM. The storage unit 3 may also include, for example, a small hard disk drive or SSD (Solid State Drive).

[0019] The input unit 5 can accept various inputs from the user. The input unit 5 may include, for example, a mouse and a keyboard. The input unit 5 may also include a touch sensor that accepts touch operations by the user. The input unit 5 may also include a microphone that accepts voice input by the user. The control unit 2 can recognize the content of the user input accepted by the input unit 5 based on the output signal from the input unit 5.

[0020] The display unit 6 is capable of displaying various types of information under the control of the control unit 2. The display unit 6 may be, for example, a liquid crystal display, an organic electroluminescence (EL) display, or a plasma display. Furthermore, if the input unit 5 includes a touch sensor, the touch sensor and the display unit 6 may form a touch panel display having a display function and a touch detection function.

[0021] The control unit 2 can reproduce the light environment in a target real space 100 set in real space in a target virtual space 200 in virtual space, which corresponds to the target real space 100. The light environment in the target real space 100 may be, for example, a lighting environment. Figure 2 is a schematic diagram showing an example of the target real space 100 and the target virtual space 200.

[0022] <Regarding the Target Real Space> The target real space 100 is, for example, a hemispherical, dome-shaped, or rectangular space. Furthermore, the target real space 100 does not have to have a predetermined shape. Light 161 is incident on the target real space 100 from, for example, one or more lighting devices 160 arranged on the ceiling 150 of a building. The lighting devices 160 may be, for example, fluorescent lamps or light-emitting diodes. Furthermore, natural light may be incident on the target real space 100, for example.

[0023] When the target real space 100 is hemispherical, the surface 101 of the hemispherical target real space 100 has a curved surface 102 and a bottom surface 103. A predetermined member 130 is arranged in the target real space 100 to determine the setting content of at least one virtual light source 210. The predetermined member 130 is irradiated with light 161 from, for example, multiple lighting devices 160. The predetermined member 130 may also be illuminated with natural light. The size and shape of the target real space 100 are set by the user of the processing device 1. Note that the shape of the target real space 100 may be other than hemispherical.

[0024] The predetermined member 130 is placed on, for example, the bottom surface 103 of the target real space 100. The predetermined member 130 is photographed by the camera 120. The camera 120 is, for example, a handheld camera that photographs the predetermined member 130 while being held in a person's hand. The camera 120 is, for example, a three-dimensional camera. The camera 120 photographs a photographing range including the predetermined member 130 to generate a camera image 125 (see FIG. 8 described later). The camera image 125 includes, for example, a color image 126 and a depth image 127 (see FIG. 8 described later). Each of the color image 126 and the depth image 127 can also be considered a camera image. The color image 126 can also be considered a color camera image 126, and the depth image 127 can also be considered a depth camera image 127. The camera 120 has a color photographing function that performs color photographing to generate the color image 126, and a depth image generating function that generates the depth image 127.

[0025] Each pixel value of the color image 126 includes, for example, an R component (red component), a G component (green component), and a B component (blue component). Such a color image 126 is also called an RGB image. A predetermined member 130 is depicted in the color image 126.

[0026] The depth image 127 is an image that two-dimensionally represents the distance from the camera 120 to each measurement point included in the shooting range. Each pixel value of the depth image 127 indicates the distance from the camera 120 to the measurement point corresponding to that pixel value. The depth image 127 is also called a distance image. The depth image 127 may be acquired, for example, by a stereo method, a ToF (Time of Flight) method, or another method.

[0027] Fig. 3 is a schematic top view showing an example of the predetermined member 130. Fig. 4 is a schematic side view showing an example of the predetermined member 130. Fig. 5 is a schematic perspective view showing an example of the predetermined member 130. The predetermined member 130 is placed, for example, on a table provided in real space. The top surface of the table, i.e., the placement surface on which the predetermined member 130 is placed, is located on the same plane as the bottom surface 103 of the target real space 100. The color image 126 generated by the camera 120 also captures the placement surface on which the predetermined member 130 is placed.

[0028] 3 to 5, the predetermined member 130 is a polyhedron. The predetermined member 130 is, for example, dome-shaped with a flat top. A mark 132 for specifying the position and orientation of the predetermined member 130 is attached to the flat upper surface 131 of the predetermined member 130. The mark 132 includes, for example, a white pattern 132a and a black pattern 132b. The mark 132, for example, is configured with the white pattern 132a and the black pattern 132b to form a checkerboard pattern.

[0029] The planar shape of the upper surface 131 of the predetermined member 130 may be, for example, a polygonal shape having four or more corners. The angle of each corner of the planar shape of the upper surface 131 may be greater than 90°. The sides of the upper surface 131 may be set to the same length. The corners of the upper surface 131 may be set to the same angle. The upper surface 131 may have an even number of corners.

[0030] The peripheral side surface 136 of the specified member 130 is composed of, for example, a plurality of flat surfaces 135. The normal directions of the surfaces 135 are different from one another. The surfaces 135 face in different directions from one another. Furthermore, the normal directions of the surfaces 135 and the top surface 131 are different from one another. The surfaces 135 and the top surface 131 face in different directions from one another. The peripheral side surface 136 has, for example, eight surfaces 135 in the circumferential direction and three surfaces 135 from the top surface 131 to the bottom surface 138, for a total of 24 surfaces 135. Each surface 135 has, for example, a trapezoidal shape. Hereinafter, each surface 135 may be referred to as a side surface 135. The surfaces 135 are connected to one another and to the top surface 131.

[0031] As shown in FIG. 2 , for example, the predetermined member 130 is placed on the bottom surface 103 of the target real space 100 so that the center of its bottom surface 138 coincides with the center 103 a of the bottom surface 103 of the target real space 100. The camera 120 is placed so as to be able to capture images of multiple surfaces of the predetermined member 130. In other words, the predetermined member 130 has a configuration such that multiple surfaces are captured by the camera 120. Furthermore, the predetermined member 130 is disposed so that multiple surfaces are captured by the camera 120. In the example of FIG. 2 , the camera 120 is located directly above the top surface 131 of the predetermined member 130, but the position of the camera 120 is not limited to this. For example, the camera 120 may be located so as to be able to simultaneously capture images of four or more surfaces of the predetermined member 130.

[0032] <Regarding the Target Virtual Space> The target virtual space 200 has, for example, the same size and shape as the target real space 100. A virtual predetermined member 230 corresponding to the predetermined member 130 is arranged in the target virtual space 200. The surface 201 of the hemispherical target virtual space 200 has a curved surface 202 and a bottom surface 203. Note that the target virtual space 200 may be set as a space having a different shape from the target real space 100.

[0033] The virtual predetermined member 230 is placed, for example, on the bottom surface 203 of the target virtual space 200. The relative position and orientation of the virtual predetermined member 230 with respect to the target virtual space 200 is the same as the relative position and orientation of the predetermined member 130 with respect to the target real space 100.

[0034] The virtual predetermined member 230 is photographed by a virtual camera 220 equivalent to the camera 120. The virtual camera 220 has, for example, a function equivalent to the color photographing function of the camera 120, but does not have a function equivalent to the depth image generation function of the camera 120. The virtual camera 220 can be said to be a two-dimensional camera that virtually photographs the photographing range in color. The virtual camera 220 generates a virtual color image equivalent to the color image 126 generated by the camera 120. The virtual color image is an image obtained by virtually photographing the virtual predetermined member 230 in color with the virtual camera 220.

[0035] The performance and shooting conditions for color photography by the virtual camera 220 are set to be the same as those for color photography by the camera 120. The relative position and orientation of the virtual camera 220 with respect to the target virtual space 200 are set to be the same as those of the camera 120 with respect to the target real space 100.

[0036] The virtual predetermined member 230 has the same size and shape as the predetermined member 130. The virtual predetermined member 230 may also have the same surface color as the predetermined member 130. The surface color may be, for example, gray, such as grayscale 18, which is black with 18% white. The virtual predetermined member 230 may also have the same surface condition as the predetermined member 130. FIG. 6 is a schematic perspective view showing an example of the virtual predetermined member 230. Like the predetermined member 130, the virtual predetermined member 230 has, for example, a dome shape with a flat top. A mark 232 for specifying the position and orientation of the virtual predetermined member 230 is attached to a flat upper surface 231 of the virtual predetermined member 230. The mark 232 has the same size and shape as the mark 132 attached to the predetermined member 130. Like the mark 132, the mark 232 includes, for example, a white pattern 232a and a black pattern 232b. The mark 232 is, for example, a checkered pattern made up of white patterns 232a and black patterns 232b.

[0037] The peripheral side surface 236 of the virtual predetermined member 230, like the peripheral side surface 136 of the predetermined member 130, is composed of, for example, a plurality of flat surfaces 235. The normal directions of the plurality of surfaces 235 are different from one another. Furthermore, the normal directions of the plurality of surfaces 235 and the top surface 231 are different from one another. The peripheral side surface 236 has, for example, eight surfaces 235 in the circumferential direction and three surfaces 235 from the top surface 231 to the bottom surface 238, for a total of 24 surfaces 235. The shape of each surface 235 is, for example, a trapezoid. Hereinafter, each surface 235 may be referred to as a side surface 235. It can also be said that each of the top surface 231 and the side surface 235 of the virtual predetermined member 230 is a virtual surface.

[0038] 2 , for example, the virtual predetermined member 230 is disposed on the bottom surface 203 of the target virtual space 200, similar to the predetermined member 130, so that the center of the bottom surface 238 of the virtual predetermined member 230 coincides with the center 203 a of the bottom surface 203 of the target virtual space 200. As in the example of FIG. 2 , for example, when the camera 120 is disposed directly above the top surface 131 of the predetermined member 130, the virtual camera 220 is disposed directly above the top surface 231 of the virtual predetermined member 230.

[0039] For example, a plurality of virtual light sources 210 capable of emitting light 211 into target virtual space 200 are arranged on surface 201 of target virtual space 200. Light emitted by virtual light sources 210 is irradiated onto virtual predetermined member 230. The plurality of virtual light sources 210 are arranged on curved surface 202 of target virtual space 200. Twenty-five virtual light sources 210 corresponding to top surface 231 and 24 side surfaces 235 of virtual predetermined member 230 are arranged on curved surface 202 of target virtual space 200. Each virtual light source 210 is capable of irradiating light 211 toward the corresponding surface of virtual predetermined member 230. Each virtual light source 210 is arranged such that its optical axis is directed toward the corresponding surface of virtual predetermined member 230. Each virtual light source 210 is arranged such that the normal to the location where virtual light source 210 is located on curved surface 202 of target virtual space 200 coincides with the optical axis of virtual light source 210. As will be described later, by appropriately setting the on / off status and brightness of each virtual light source 210, the light environment in target real space 100 is reproduced in target virtual space 200. Hereinafter, for convenience of explanation, virtual light source 210 corresponding to top surface 231 may be referred to as first virtual light source 210a, and virtual light source 210 corresponding to side surface 235 may be referred to as second virtual light source 210b.

[0040] 7 is a schematic diagram showing an example of the positional relationship between a certain side surface 235 (also referred to as side surface 235a) and second virtual light source 210b (also referred to as second virtual light source 210bb) corresponding to that certain side surface 235. As shown in Fig. 7, the position and orientation of second virtual light source 210bb relative to side surface 235a is set so that normal 235ab of side surface 235a, which passes through center 235aa of side surface 235a, and optical axis 212bb of second virtual light source 210bb are positioned on the same straight line. The positions and orientations of 24 second virtual light sources 210b corresponding to the 24 side surfaces 235, respectively, are set in the same manner as in Fig. 7.

[0041] The position and orientation of first virtual light source 210a relative to upper surface 231 of virtual predetermined member 230 are also set in a similar manner. That is, the position and orientation of first virtual light source 210a relative to upper surface 231 are set so that the normal to upper surface 231, which passes through the center of upper surface 231, and optical axis 212bb of first virtual light source 210a are positioned on the same straight line.

[0042] Hereinafter, the top surface 131 and the side surface 135 of the predetermined member 130 may be referred to as the actual top surface 131 and the actual side surface 135, respectively, and the top surface 231 and the side surface 235 of the imaginary predetermined member 230 may be referred to as the imaginary top surface 231 and the imaginary side surface 235, respectively.

[0043] Control unit 2 sets target virtual space 200, which corresponds to target real space 100, in the virtual space. Control unit 2 then realizes a plurality of virtual light sources 210, virtual cameras 220, and virtual predetermined members 230 in the virtual space. The plurality of virtual light sources 210, virtual cameras 220, and virtual predetermined members 230 are realized by processing executed by control unit 2. Processing device 1 can also be seen as including a plurality of virtual light sources 210, virtual cameras 220, and virtual predetermined members 230.

[0044] <Example of Operation of Control Unit> FIG. 8 is a schematic diagram showing an example of multiple functional blocks included in the control unit 2 and an example of information stored in the storage unit 3. As shown in FIG. 8, the storage unit 3 stores, for example, a camera image 125 generated by the camera 120. The camera image 125 includes a color image 126 and a depth image 127. The storage unit 3 also stores camera information 180 related to the camera 120 and predetermined member information 181 related to the predetermined member 130. The storage unit 3 also stores measured brightness information 182 indicating a measurement value of brightness in the target real space 100 (in other words, an actual measurement value), and target real space information 183 indicating the size and shape of the target real space 100. The camera image 125, the camera information 180, the predetermined member information 181, the measured brightness information 182, and the target real space information 183 can be said to be data related to the real environment (also referred to as real environment data).

[0045] The camera information 180 includes information necessary to realize a virtual camera 220 equivalent to the camera 120 in a virtual space. Realizing the virtual camera 220 in a virtual space can also be said to be reproducing the camera 120 in the virtual space. The camera information 180 indicates the performance and shooting conditions of the camera 120. The camera information 180 is generated by, for example, the camera 120. The camera information 180 may be an Exif file. Exif is an abbreviation for Exchangeable Image File Format. The camera information 180 includes, for example, camera body information indicating information about the body of the camera 120, lens information indicating information about the lens of the camera 120, and shooting condition information indicating the shooting conditions of the camera 120.

[0046] The camera body information indicates, for example, the size of the image sensor provided in the camera 120, the ISO sensitivity, the shutter speed, etc. The lens information indicates, for example, various lens parameters (e.g., distortion coefficient, etc.) The shooting condition information indicates, for example, the resolution and white balance settings, etc.

[0047] The target real space information 183 includes information necessary for setting, in the virtual space, a target virtual space 200 that corresponds to the target real space 100. The target real space information 183 indicates, for example, that the target real space 100 is hemispherical and the radius or diameter of the hemispherical target real space 100.

[0048] The predetermined component information 181 includes information necessary to realize a virtual predetermined component 230 corresponding to the predetermined component 130 in a virtual space. The realization of the virtual predetermined component 230 in a virtual space can also be said to be a reproduction of the predetermined component 130 in the virtual space. The predetermined component information 181 indicates, for example, the size, shape, reflectance, color, and material of the predetermined component 130. The predetermined component information 181 also indicates, for example, the position, size, shape, and color of a mark 132 affixed to the predetermined component 130.

[0049] The color information of the predetermined member information 181 may be the color temperature of the target real space 100. The color temperature in the target real space 100 is measured using, for example, a chromatic illuminance meter. The chromatic illuminance meter is disposed on, for example, the upper surface 131 of the predetermined member 130. In addition to the chromatic illuminance meter, the color temperature in the target real space 100 may also be determined based on the color image 126.

[0050] The measured brightness information 182 indicates a measurement value of brightness in the target real space 100. In this example, the brightness in the target real space 100 is measured using an illuminance meter. The illuminance meter is placed, for example, on the upper surface 131 of the specified member 130. In this case, the illuminance meter is placed, for example, on the upper surface 131 so that its light receiving unit faces the vertex of the hemispherical target real space 100. The illuminance meter measures, for example, the illuminance near the upper surface 131 of the specified member 130. The measured brightness information 182 includes the illuminance measured by the illuminance meter (also referred to as measured illuminance). Note that color information in addition to illuminance may also be acquired by a colorimeter.

[0051] The camera image 125, the camera information 180, the predetermined member information 181, the measured brightness information 182, and the target real space information 183 may be input to the control unit 2 via the communication unit 4, for example, and the control unit 2 may write the camera image 125, the camera information 180, the predetermined member information 181, the measured brightness information 182, and the target real space information 183 into the storage unit 3. Furthermore, the predetermined member information 181 and the target real space information 183 may be stored in advance in the storage unit 3.

[0052] 8, the control unit 2 includes, for example, a virtual space construction unit 20, a determination unit 21, and a setting unit 22. The virtual space construction unit 20, the determination unit 21, and the setting unit 22 are functional blocks formed in the control unit 2, for example, when the CPU of the control unit 2 executes a program 30 in the storage unit 3. Note that all or some of the functions of the virtual space construction unit 20 may be realized by a hardware circuit that does not require software to realize the function. The same applies to the determination unit 21 and the setting unit 22.

[0053] Virtual space construction unit 20 places in the virtual space a plurality of virtual light sources 210, virtual cameras 220, and virtual predetermined members 230. Virtual space construction unit 20 places in the virtual space a plurality of virtual light sources 210, virtual cameras 220, and virtual predetermined members 230, for example, based on camera image 125, camera information 180, predetermined member information 181, and target real space information 183 in storage unit 3.

[0054] Determination unit 21 determines the setting contents of multiple virtual light sources 210. Determination unit 21 determines the setting contents of multiple virtual light sources 210, for example, based on camera image 125 and measured brightness information 182 in storage unit 3 and a virtual color image (also referred to as a virtual camera image) obtained by virtual camera 220. Determination unit 21 determines, for example, whether multiple virtual light sources 210 are turned on or off, brightness, and color, as the setting contents of multiple virtual light sources 210.

[0055] Setting unit 22 sets virtual light source 210 and virtual camera 220. Setting unit 22 sets virtual camera 220 based on, for example, camera information 180 in storage unit 3. Setting unit 22 also sets a plurality of virtual light sources 210 in the setting content determined by determination unit 21. By setting at least one virtual light source 210 set in the virtual space in the setting content determined by determination unit 21, the lighting environment in target real space 100 is reproduced in target virtual space 200.

[0056] 9 is a flowchart showing an example of a process (also referred to as a virtual light environment reproduction process) in which the control unit 2 reproduces the light environment in the target real space 100 in the target virtual space 200. As shown in Fig. 9, in step s1, the virtual space construction unit 20 of the control unit 2 sets the target virtual space 200 in the virtual space based on the target real space information 183 in the storage unit 3. The virtual space construction unit 20 sets the target virtual space 200 in the virtual space having the same size and shape as the target real space 100 based on the target real space information 183.

[0057] Next, in step s2, virtual space construction unit 20 places virtual predetermined member 230 corresponding to predetermined member 130 on bottom surface 203 of target virtual space 200 based on predetermined member information 181 in storage unit 3. Virtual space construction unit 20 also places multiple virtual light sources 210 on surface 201 (e.g., curved surface 202) of target virtual space 200, as in the example of FIG.

[0058] When the arrangement of each virtual light source 210 is completed, virtual space construction unit 20 generates position identification information for each virtual light source 210 to identify the position of that virtual light source 210, and stores the information in storage unit 3. The position identification information may be, for example, information for identifying the position of virtual light source 210 in XYZ Cartesian coordinate system 500 set in target virtual space 200. Fig. 10 is a schematic diagram showing an example of XYZ Cartesian coordinate system 500 set in target virtual space 200.

[0059] The origin of the XYZ Cartesian coordinate system 500 is set, for example, at the center 203a of the bottom surface 203 of the target virtual space 200. The Z axis of the XYZ Cartesian coordinate system 500 is set, for example, to pass through the center 203a of the bottom surface 203 and to be perpendicular to the bottom surface 203. The XY plane of the XYZ Cartesian coordinate system 500 is set, for example, to be parallel to the bottom surface 203.

[0060] The position identification information may include, for example, distance information indicating the distance from center 203 a of bottom surface 203 to virtual light source 210. The distance information may include the distance from the origin of XYZ Cartesian coordinate system 500 (in other words, center 203 a of bottom surface 203) to the projection position of virtual light source 210 projected onto the XZ plane. This distance can be said to be the distance from the origin of XYZ Cartesian coordinate system 500 to virtual light source 210 when target virtual space 200 is viewed from the Y-axis direction as shown in FIG. 10 . The distance information may also include the linear distance (in other words, three-dimensional distance) from the origin of XYZ Cartesian coordinate system 500 to the position of virtual light source 210.

[0061] The position identification information may include angle information indicating the angle at which virtual light source 210 is viewed from center 203 a of bottom surface 203. The angle information may include, for example, a first angle formed by a vector extending from the origin of XYZ Cartesian coordinate system 500 to a projection position of virtual light source 210 projected onto the XZ plane, and the positive direction of the X axis. The first angle can be considered to be the angle of virtual light source 210 when target virtual space 200 is viewed from the Y axis direction. The angle information may also include a second angle formed by a vector extending from the origin of XYZ Cartesian coordinate system 500 to a projection position of virtual light source 210 projected onto the XY plane, and the positive direction of the X axis. The second angle can be considered to be the angle of virtual light source 210 when target virtual space 200 is viewed from the Z axis direction.

[0062] After step s2, in step s3, control unit 2 determines and sets the color of virtual light source 210. In step s3, determination unit 21 first determines the color of virtual light source 210 based on color image 126. In this example, for example, the colors of multiple virtual light sources 210 are set to be the same. Note that the colors of virtual light source 210 may be measured by a colorimeter provided in predetermined member 130.

[0063] The determining unit 21 extracts, from the color image 126, a partial image (also referred to as a top image) showing the top surface 131 of the predetermined member 130 to which the mark 132 is attached, based on the predetermined member information 181. In this example, a photographer using the camera 120 photographs the predetermined member 130 with the camera 120, for example, from directly above the top surface 131 of the predetermined member 130. Therefore, the color image 126 shows the entire top surface 131 and peripheral side surface 136 of the predetermined member 130.

[0064] Next, the determination unit 21 extracts a white pattern image showing the white pattern 132a of the mark 132 from the extracted top surface image based on the predetermined member information 181. Then, the determination unit 21 determines the surface color of the white pattern 132a irradiated with the light 161 from the lighting device 160 based on the white pattern image, and determines the determined surface color as the color of each virtual light source 210. The surface color is also called a reflective object color.

[0065] The surface color determined by the determination unit 21 may be expressed as a color temperature. In this case, the determination unit 21 determines, for example, the average values ​​of the R component, the G component, and the B component of the pixel values ​​of the plurality of pixels constituting the white pattern image. The determination unit 21 may then convert the average values ​​of the R component, the average value of the G component, and the average value of the B component into a color temperature, and use the determined color temperature as the surface color of the white pattern 132a.

[0066] The surface color determined by the determination unit 21 may be expressed by saturation and hue angle. In this case, the determination unit 21 may convert the average values ​​of the R component, the G component, and the B component into lightness L, saturation C, and hue angle h (i.e., convert RGB to LCh), and use the determined saturation C and hue angle h as the surface color of the white pattern 132a.

[0067] When determination unit 21 determines the color of virtual light source 210, setting unit 22 sets the color of each virtual light source 210 to the color determined by determination unit 21. At this point, although the color of each virtual light source 210 has been set, each virtual light source 210 is not lit.

[0068] After step s3, in step s4, control unit 2 determines and sets the brightness of first virtual light source 210a located directly above upper surface 231 of virtual predetermined member 230. In step s4, determination unit 21 first determines the brightness of first virtual light source 210a based on measured brightness information 182 in storage unit 3.

[0069] Here, the brightness of virtual light source 210 is represented, for example, by the illuminance at the location where virtual light source 210 exists. Hereinafter, the brightness of virtual light source 210 refers to the illuminance at the location where virtual light source 210 exists.

[0070] For example, determination unit 21 determines the brightness of first virtual light source 210a to be the same illuminance as the measured illuminance included in measured brightness information 182. When determination unit 21 determines the brightness of first virtual light source 210a, setting unit 22 sets the brightness of first virtual light source 210a to the brightness determined by determination unit 21. For example, setting unit 22 sets the brightness of first virtual light source 210a to the same illuminance as the measured illuminance included in measured brightness information 182.

[0071] After step s4, in step s5, determination unit 21 determines to turn on first virtual light source 210a, and setting unit 22 turns on first virtual light source 210a. The color of turned-on first virtual light source 210a is the color determined in step s3, and the brightness of turned-on first virtual light source 210a is the brightness determined in step s4. Setting unit 22 associates the position identification information of turned-on first virtual light source 210a, brightness information (illuminance in this example) indicating the brightness of first virtual light source 210a, and color information (e.g., color temperature) indicating the color of first virtual light source 210a with each other and stores them in storage unit 3. Note that, at the time step s4 is executed, only first virtual light source 210a is turned on among the multiple virtual light sources 210.

[0072] After step s5, in step s6, the control unit 2 positions and sets the virtual camera 220. In step s6, the virtual space construction unit 20 identifies the positional relationship between the camera 120 and the predetermined member 130 based on the depth image 127 in the storage unit 3. Then, the virtual space construction unit 20 positions the virtual camera 220 in the virtual space so that the positional relationship between the virtual camera 220 and the virtual predetermined member 230 in the virtual space is the same as the positional relationship between the camera 120 and the predetermined member 130 in the real space.

[0073] In step s6, the setting unit 22 sets the virtual camera 220 based on the camera information 180 in the storage unit 3. The setting unit 22 sets the virtual camera 220 based on the camera information 180 so that the performance and shooting conditions of the virtual camera 220 are the same as the performance and shooting conditions of color shooting by the camera 120.

[0074] After step s6, in step s7, virtual camera 220 takes an image. In step s7, virtual camera 220 takes a virtual color image with the same capabilities and conditions as those for color imaging by camera 120, with first virtual light source 210a turned on. In step s6, virtual camera 220 takes a color image and generates a virtual color image showing virtual predetermined member 230. After step s7, step s8 is executed.

[0075] In step S8, the determination unit 21 determines whether or not there is a virtual side 235 among the multiple virtual sides 235 of the virtual specified member 230 that corresponds to the second virtual light source 210b that is not currently lit and is darker than the actual side 135 of the specified member 130.

[0076] Here, it is assumed that a real side surface 135 and a virtual side surface 235 corresponding to the real side surface 135 correspond to each other. If the brightness of the virtual side surface 235 is darker than the brightness of the real side surface 135 corresponding to the real side surface 235, the virtual side surface 235 becomes a virtual side surface 235 darker than the real side surface 135. Whether the virtual side surface 235 is darker than the real side surface 135 may be determined, for example, by whether the difference in brightness between the real side surface 135 and the virtual side surface 235 is equal to or less than a predetermined threshold. In other words, if the difference is equal to or less than the threshold, there is no virtual side surface 235 darker than the real side surface 135, and if the difference is equal to or less than the threshold, there is a virtual side surface 235 darker than the real side surface 135. The threshold may be set, for example, based on human perception, the robustness of a learning model, or user definition.

[0077] Hereinafter, a virtual side surface 235 corresponding to unlit second virtual light source 210b may be referred to as an unlit-corresponding virtual side surface 235. A virtual side surface 235 corresponding to unlit second virtual light source 210b and darker than real side surface 135 may be referred to as a specific unlit-corresponding virtual side surface 235. A real side surface 135 corresponding to an unlit-corresponding virtual side surface 235 may be referred to as an unlit-corresponding real side surface 135. In step s8, the determination unit 21 determines whether or not there is an unlit-corresponding virtual side surface 235 that is darker than the real side surface 135 among the multiple virtual side surfaces 235. In other words, in step s8, the determination unit 21 determines whether or not there is a specific unlit-corresponding virtual side surface 235 among the multiple virtual side surfaces 235.

[0078] When step s8 is executed for the first time after the start of the virtual lighting environment reproduction process, all of second virtual light sources 210b are not lit, and therefore all of virtual side surfaces 235 are unlit corresponding virtual side surfaces 235. Step s8 will be described in detail below.

[0079] In step s8, the determination unit 21 acquires brightness information of each unlit corresponding virtual side surface 235 based on, for example, the virtual color image obtained in step s7. The determination unit 21 also acquires brightness information of each unlit corresponding real side surface 135 of the specified member 130 based on the color image 126 in the storage unit 3.

[0080] When acquiring brightness information of the unlit-corresponding virtual side surface 235, the determination unit 21 first extracts a partial image (also referred to as a specific virtual side surface image) showing the unlit-corresponding virtual side surface 235 from the virtual color image based on the predetermined component information 181. Next, the determination unit 21 calculates the average values ​​of the R, G, and B components of the pixel values ​​of multiple pixels constituting the extracted specific virtual side surface image. Next, the determination unit 21 converts the average values ​​of the R, G, and B components of the specific virtual side surface image into lightness L, chromaticity a, and chromaticity b (i.e., converts RGB to Lab). The determination unit 21 then sets the lightness L calculated from the specific virtual side surface image as the lightness of the unlit-corresponding virtual side surface 235 shown in the specific virtual side surface image. In this example, for example, the lightness of the unlit-corresponding virtual side surface 235 is used as brightness information of the unlit-corresponding virtual side surface 235. The determination unit 21 similarly acquires brightness information of each unlit-corresponding virtual side surface 235.

[0081] Furthermore, when acquiring brightness information of the unlit-corresponding real side surface 135, the determination unit 21 first extracts a partial image (also referred to as a specific real side surface image) showing the unlit-corresponding real side surface 135 from the color image 126 based on the predetermined component information 181. Next, the determination unit 21 calculates the average values ​​of the R, G, and B components of the pixel values ​​of the multiple pixels constituting the extracted specific real side surface image. Next, the determination unit 21 converts the average values ​​of the R, G, and B components of the specific real side surface image into lightness L, chromaticity a, and chromaticity b (i.e., converts RGB to Lab). The determination unit 21 then sets the lightness L calculated from the specific real side surface image as the lightness of the unlit-corresponding real side surface 135 shown in the specific real side surface image. In this example, for example, the lightness of the unlit-corresponding real side surface 135 is used as brightness information of the unlit-corresponding real side surface 135. The determination unit 21 similarly acquires brightness information of each unlit-corresponding real side surface 135.

[0082] The determination unit 21 determines a first brightness difference value obtained by subtracting the brightness of the unlit-corresponding real side surface 135 corresponding to the unlit-corresponding virtual side surface 235 from the brightness of the unlit-corresponding virtual side surface 235. The determination unit 21 determines the first brightness difference value for each unlit-corresponding virtual side surface 235. Then, for each unlit-corresponding virtual side surface 235, if the first brightness difference value determined for the unlit-corresponding virtual side surface 235 is equal to or greater than a positive threshold value, the determination unit 21 determines that the unlit-corresponding virtual side surface 235 is an unlit-corresponding virtual side surface 235 that is darker than the real side surface 135, i.e., a specific unlit-corresponding virtual side surface 235.

[0083] If the determination unit 21 determines that the plurality of virtual sides 235 includes at least one specific unlit corresponding virtual side 235 (YES in step s8), step s9 is executed. On the other hand, if the determination unit 21 determines that the plurality of virtual sides 235 does not include the specific unlit corresponding virtual side 235 (NO in step s8), the virtual lighting environment reproduction process ends.

[0084] In step s9, determination unit 21 determines second virtual light source 210b to be turned on from second virtual light sources 210b that are currently off (in other words, second virtual light source 210b to be turned on). If only one specific unlit corresponding virtual side surface 235 was identified in step s8, determination unit 21 determines to turn on second virtual light source 210b corresponding to the identified specific unlit corresponding virtual side surface 235. On the other hand, if multiple specific unlit corresponding virtual side surfaces 235 were identified in step s8, determination unit 21 identifies, from the multiple specific unlit corresponding virtual side surfaces 235, the specific unlit corresponding virtual side surface 235 having the largest first brightness difference value calculated in step s8. Then, determination unit 21 determines to turn on second virtual light source 210b corresponding to the specific unlit corresponding virtual side surface 235 having the largest first brightness difference value. Hereinafter, second virtual light source 210b determined to be turned on in step s8 may be referred to as turned-on second virtual light source 210b.

[0085] After step s9, in step s10, control unit 2 determines and sets the brightness of second virtual light source 210b that was determined to be turned on in step s8 (i.e., second virtual light source 210b to be turned on). In step s10, determination unit 21 first determines the brightness of second virtual light source 210b to be turned on, based on the first brightness difference value calculated for unlit corresponding virtual side surface 235 that corresponds to second virtual light source 210b to be turned on.

[0086] The determination unit 21 determines the brightness (also referred to as the required brightness) of the lighting-determined second virtual light source 210b that is required so that, when the lighting-determined second virtual light source 210b is turned on, the brightness of the virtual side surface 235 corresponding to the lighting-determined second virtual light source 210b increases by the first brightness difference value determined for the virtual side surface 235. The determination unit 21 can determine the required brightness of the lighting-determined second virtual light source 210b, for example, based on the camera information 180, the specified member information 181, the positional relationship between the virtual camera 220 and the virtual specified member 230, and the position of the lighting-determined second virtual light source 210b. Then, the determination unit 21 determines to turn on the lighting-determined second virtual light source 210b at the required brightness thus determined. This determines the brightness of the lighting-determined second virtual light source 210b. As described above, the brightness of the lighting-determined second virtual light source 210b is expressed, for example, by the illuminance at the location where the lighting-determined second virtual light source 210b is present.

[0087] When determination unit 21 determines the brightness of second virtual light source 210 b to be turned on, setting unit 22 sets the brightness of second virtual light source 210 b to the brightness determined by determination unit 21 .

[0088] After step s10, in step s11, setting unit 22 turns on determined-to-turn-on second virtual light source 210b. The color of determined-to-turn-on second virtual light source 210b that is turned on is the color determined in step s3, and the brightness of determined-to-turn-on second virtual light source 210b that is turned on is the brightness determined in step s10. Setting unit 22 stores in storage unit 3 the position identification information of determined-to-turn-on second virtual light source 210b that is turned on, brightness information indicating the brightness of determined-to-turn-on second virtual light source 210b, and color information indicating the color of determined-to-turn-on second virtual light source 210b, in association with each other.

[0089] After step s11, step s6 is executed again, and virtual camera 220 captures an image. In this capture, the number of lit virtual light sources 210 is increased by one compared to the previous capture in step s6. After step s6, control unit 2 operates in the same manner.

[0090] When there is no unlit corresponding virtual side surface 235 (i.e., specific unlit corresponding virtual side surface 235) that is darker than the real side surface among the multiple virtual side surfaces 235 that virtual predetermined member 230 has, step s8 is determined to be NO, and the virtual lighting environment reproduction process ends. In this example, second virtual light source 210b, which was not determined to be lit in the virtual lighting environment reproduction process, is ultimately determined to be unlit. In other words, by not determining to turn on second virtual light source 210b, determiner 21 determines that second virtual light source 210b is to be unlit.

[0091] As can be understood from the above description, the processing of steps s7 to s11 is repeatedly executed to generate a plurality of virtual camera images showing virtual predetermined member 230 illuminated by a plurality of different combinations of virtual light sources 210. Then, determination unit 21 determines the setting contents of a plurality of virtual light sources 210 based on the plurality of virtual camera images.

[0092] As described above, control unit 2 operates to set each virtual light source 210, so that the brightness of virtual upper surface 231 and each virtual side surface 235 of virtual predetermined member 230 arranged in target virtual space 200 becomes closer to the brightness of upper surface 131 and each side surface 135 of predetermined member 130 arranged in target real space 100. As a result, the light environment in target real space 100 is reproduced in target virtual space 200.

[0093] When the virtual environment reproduction process is completed, storage unit 3 stores setting information indicating the setting contents of multiple virtual light sources 210 for reproducing the lighting environment in target real space 100 in target virtual space 200. The setting information includes position specifying information, brightness information, and color information of each virtual light source 210 for which it has been determined that it should be turned on. Furthermore, the setting information includes position specifying information of each virtual light source 210 for which it has not been determined that it should be turned on, i.e., each virtual light source 210 for which it has been determined that it should not be turned on. By referring to the setting information, it is possible to understand which virtual light source 210 should be turned on, and at what color and brightness, in order to reproduce the lighting environment in target real space 100 in target virtual space 200.

[0094] As described above, in this example, determination unit 21 determines the setting contents of at least one virtual light source that can reproduce the light environment of target real space 100 in target virtual space 200, based on camera image 125 that captures multiple faces of a polyhedron serving as predetermined member 130 under the light environment of target real space 100 and a virtual camera image that captures virtual predetermined member 230 illuminated by at least one virtual light source 210 in target virtual space 200. This makes it possible to appropriately reproduce the light environment in target real space 100 in target virtual space 200 using at least one virtual light source 210.

[0095] The target virtual space 200, in which the lighting environment in the target real space 100 is reproduced, can be used in various situations. For example, if the target real space 100 is a workspace in which a robot performs work, the robot's working environment, including the lighting environment, can be reproduced in the target virtual space 200. This makes it possible to appropriately use the target virtual space 200 to simulate, for example, various robot operations (such as object recognition operations and object holding operations) taking the lighting environment into consideration.

[0096] Furthermore, in this example, determination unit 21 acquires brightness information of real side surface 135 based on color image 126, and acquires brightness information of virtual side surface 235 corresponding to real side surface 135 based on the virtual color image. Then, determination unit 21 determines whether to turn on or off second virtual light source 210b corresponding to virtual side surface 235 (in other words, second virtual light source 210b that emits light toward virtual side surface 235) based on the brightness information of real side surface 135 and the brightness information of virtual side surface 235 corresponding to real side surface 135. This makes it possible to appropriately determine whether to turn on or off second virtual light source 210b when reproducing the lighting environment in target real space 100 in target virtual space 200.

[0097] In addition, in the above example, the determination unit 21 determines the brightness of the second virtual light source 210b corresponding to the virtual side surface 235 based on the brightness information of the real side surface 135 and the brightness information of the virtual side surface 235 corresponding to the real side surface 135, and therefore can appropriately determine the brightness of the second virtual light source 210b when reproducing the lighting environment in the target real space 100 in the target virtual space 200.

[0098] Furthermore, in the above example, the determination unit 21 determines the color of the virtual light source 210 based on the white pattern image containing the white pattern 132a of the mark 132 included in the color image 126, and therefore the color of the virtual light source 210 can be appropriately set when reproducing the light environment in the target real space 100 in the target virtual space 200.

[0099] Note that the structure of predetermined member 130 is not limited to the above example. For example, peripheral side surface 136 of predetermined member 130 may have 14 faces 135 in the circumferential direction, and three faces 135 from top surface 131 to bottom surface 138, for a total of 42 faces 135. In this way, increasing the number of faces of predetermined member 130 increases the number of multiple virtual light sources 210, and therefore the light environment in target real space 100 can be more appropriately reproduced in target virtual space 200.

[0100] In the above example, one camera image 125 is used in the virtual lighting environment reproduction process, but the virtual lighting environment reproduction process may also be performed using multiple camera images 125 generated by the camera 120 by capturing images of the predetermined member 130 at different angles. Figure 11 is a flowchart showing an example of the virtual lighting environment reproduction process in this case. Hereinafter, the top surface 131 and the peripheral side surface 136 of the predetermined member 130 may be collectively referred to as the target surface.

[0101] In this example, the photographer using the camera 120 photographs the entire target surface of the predetermined member 130 not in one shot but multiple shots. The photographer uses the camera 120 to photograph the predetermined member 130 multiple times from different angles to photograph the entire target surface of the predetermined member 130. At this time, the photographer always photographs the top surface 131 with the mark 132 on it in each shot.

[0102] In the plurality of color images 126 contained in the plurality of camera images 125 generated by capturing images multiple times with the camera 120, the predetermined member 130 is captured in different orientations. Although only a portion of the target surface of the predetermined member 130 is captured in each color image 126, the entire target surface of the predetermined member 130 is captured when the plurality of color images 126 are viewed as a whole. Furthermore, at least the top surface 131 of the predetermined member 130 is captured in each of the plurality of color images 126.

[0103] A plurality of camera images 125 generated by the camera 120 are stored in the storage unit 3. Furthermore, the camera information 180 in the storage unit 3 includes an Exif file for each of the plurality of camera images 125.

[0104] In the virtual lighting environment reproduction process of this example, as shown in FIG. 11, the above-mentioned steps s1 and s2 are executed to set the target virtual space 200 and arrange the virtual predetermined member 230 and the plurality of virtual light sources 210.

[0105] Next, in step s23, the control unit 2 determines and sets the color of the virtual light source 210. In step s23, the determination unit 21 first combines the multiple color images 126 in the storage unit 3 to generate a single composite color image that captures the entire target surface of the predetermined member 130 (i.e., the top surface 131 and the peripheral side surfaces 136). The composite color image is an image that appears as if the camera 120 captured the entire target surface of the predetermined member 130 in a single capture. For example, the composite color image appears as if it were captured by the camera 120 located directly above the top surface 131 of the predetermined member 130, as shown in FIG. 2. The determination unit 21 generates a single composite color image that captures the entire target surface of the predetermined member 130 from the multiple color images 126, based on the Exif files for the multiple camera images 125 included in the camera information 180.

[0106] Next, determination unit 21 determines the color of virtual light source 210 based on the generated composite color image. In this example, for example, the colors of the multiple virtual light sources 210 are set to be the same. Determination unit 21 can determine the color of virtual light source 210 in the same manner as in step s3 described above. That is, determination unit 21 determines the color of virtual light source 210 using the composite color image instead of color image 126. Once determination unit 21 determines the color of virtual light source 210, setting unit 22 sets the color of each virtual light source 210 to the color determined by determination unit 21.

[0107] After step s23, the above-mentioned step s4 is executed to determine and set the brightness of first virtual light source 210a corresponding to upper surface 231 of virtual predetermined member 230. Then, the above-mentioned step s5 is executed to turn on first virtual light source 210a.

[0108] Next, in step s26, the control unit 2 positions and sets the virtual camera 220. In step s26, the virtual space construction unit 20 determines one of the multiple camera images 125 in the storage unit 3 as a processing target. Then, based on the depth image 127 included in the camera image 125 to be processed, the virtual space construction unit 20 specifies the positional relationship between the camera 120 and the predetermined member 130 at the time of capturing the camera image 125 to be processed. Then, the virtual space construction unit 20 positions the virtual camera 220 in the virtual space so that the positional relationship between the virtual camera 220 and the virtual predetermined member 230 in the virtual space is the same as the positional relationship between the camera 120 and the predetermined member 130 at the time of capturing the camera image 125 to be processed.

[0109] Furthermore, in step s26, the setting unit 22 sets the virtual camera 220 based on the Exif file of the camera image 125 to be processed, which is included in the camera information 180 in the storage unit 3. The setting unit 22 sets the virtual camera 220 based on the Exif file of the camera image 125 to be processed, so that the performance and shooting conditions of the virtual camera 220 are the same as the performance and shooting conditions of color shooting by the camera 120 when the camera image 125 to be processed was captured.

[0110] After step s26, in step s27, the virtual camera 220 takes color photographs and generates a virtual color image corresponding to the color image 126 included in the camera image 125 to be processed.

[0111] After step s27, in step s41, the virtual space construction unit 20 determines whether all of the multiple camera images 125 in the storage unit 3 have been set as processing targets. If the determination in step s41 is NO, step s26 is executed again. On the other hand, if the determination in step s41 is YES, step s42 is executed.

[0112] In step s26, the virtual space construction unit 20 determines, as a new processing target, a camera image 125 that has not been set as a processing target among the multiple camera images 125 in the storage unit 3. Then, similar to the above, the virtual space construction unit 20 determines the positional relationship between the camera 120 and the predetermined member 130 at the time of capturing the camera image 125 to be processed, based on the depth image 127 included in the newly set processing target camera image 125. Then, the virtual space construction unit 20 places the virtual camera 220 in the virtual space so that the positional relationship between the virtual camera 220 and the virtual predetermined member 230 in the virtual space is the same as the positional relationship between the camera 120 and the predetermined member 130 at the time of capturing the camera image 125 to be processed. Also, in step s26, the setting unit 22 sets the virtual camera 220 based on the Exif file of the camera image 125 of the predetermined target, which is included in the camera information 180 in the storage unit 3, in the same manner as described above. After step s26, steps s27 and s41 are executed in the same manner.

[0113] When step s41 returns YES, a plurality of virtual color images corresponding to the plurality of color images 126 included in the plurality of camera images 125 stored in the storage unit 3 have been generated. In step s42, the determination unit 21 combines the generated virtual color images to generate a composite virtual color image corresponding to the composite color image generated in step s23. The composite virtual color image is an image that appears as if the virtual camera 220 had captured the entire target surface of the virtual predetermined member 230 in a single capture. For example, the composite virtual color image appears as if it had been captured by the virtual camera 220 positioned directly above the top surface 231 of the virtual predetermined member 230, as shown in FIG. 2 . The determination unit 21 can generate a single composite virtual color image that captures the entire target surface of the virtual predetermined member 230 from the plurality of virtual color images, based on the Exif files for the plurality of camera images 125 included in the camera information 180.

[0114] After step s42, step s28 is executed. In step s28, similar to step s8, the determination unit 21 determines whether or not a specific unlit-corresponding virtual side surface 235 exists among the multiple virtual side surfaces 235. The determination unit 21 can determine whether or not a specific unlit-corresponding virtual side surface 235 exists in the same manner as step s8. That is, the determination unit 21 determines whether or not a specific unlit-corresponding virtual side surface 235 exists by using a composite virtual color image and a composite color image instead of the virtual color image and the color image 126, respectively.

[0115] In step s28, if the determination unit 21 determines that the plurality of virtual sides 235 includes at least one specific unlit corresponding virtual side 235 (YES in step s28), step s29 is executed. On the other hand, if the determination unit 21 determines that the plurality of virtual sides 235 does not include the specific unlit corresponding virtual side 235 (NO in step s28), the virtual lighting environment reproduction process ends.

[0116] In step s29, determination unit 21 determines second virtual light source 210b to be turned on from second virtual light source 210b that is not currently turned on, in the same manner as in step s9 described above.

[0117] Next, in step s30, the control unit 2 determines and sets the brightness of the second virtual light source 210b (also called the second virtual light source 210b determined to be turned on) that was determined to be turned on in step s29, in the same manner as in step s10 described above.

[0118] Next, in step s31, setting unit 22 turns on second virtual light source 210b. Then, step s26 is executed again. In step s26 immediately after step s31, virtual space construction unit 20 resets the previous determination of the processing target and newly determines one of the multiple camera images 125 in storage unit 3 as the processing target. Then, based on depth image 127 included in camera image 125 of the processing target, virtual space construction unit 20 places virtual camera 220 in the virtual space so that the positional relationship between virtual camera 220 and virtual predetermined member 230 in the virtual space is the same as the positional relationship between camera 120 and predetermined member 130 when camera image 125 of the processing target was captured. Thereafter, control unit 2 operates in the same manner.

[0119] Thereafter, if step s41 is judged as YES and step s42 is executed, multiple virtual color images are synthesized that were taken in a state where the number of virtual light sources 210 that was lit was one more than when the multiple virtual color images synthesized in the previous step s42 were taken.

[0120] In this way, even if the photographer photographs the entire target surface of the specified member 130 in multiple shots rather than just one, the light environment in the target real space 100 can be appropriately reproduced in the target virtual space 200.

[0121] 11, multiple virtual color images obtained by taking multiple images with the virtual camera 220 are synthesized, but a virtual color image equivalent to the synthesized color image generated in step s23 may be generated by taking a single image with the virtual camera 220. Fig. 12 is a flowchart showing an example of virtual light environment reproduction processing in this case.

[0122] In the example of Fig. 12, steps s1, s2, s23, s4, and s5 are executed in this order, as in the example of Fig. 11. Next, in step s56, the control unit 2 positions and sets the virtual camera 220. In step s56, the virtual space construction unit 20 positions the virtual camera 220 so that the positional relationship between the virtual camera 220 and the virtual predetermined member 230 is the same as the positional relationship between the camera 120 and the predetermined member 130 when the camera 120 captures a color image similar to the composite color image generated in step s23. Then, the setting unit 22 sets the performance and shooting conditions of the virtual camera 220 so that they are the same as the performance and shooting conditions of the camera 120 when the camera 120 captures a color image similar to the composite color image generated in step s23.

[0123] After step s56, in step s57, the virtual camera 220 takes an image and generates a virtual color image corresponding to the composite color image. In step s58, the control unit 2 determines whether the specific unlit corresponding virtual side surface 235 exists by using the color image generated in step s57 instead of the composite virtual color image.

[0124] In step s58, if the determination unit 21 determines that the plurality of virtual sides 235 includes at least one specific unlit corresponding virtual side 235 (YES in step s58), step s59 is executed. On the other hand, if the determination unit 21 determines that the plurality of virtual sides 235 does not include the specific unlit corresponding virtual side 235 (NO in step s58), the virtual lighting environment reproduction process ends.

[0125] In step s59, determination unit 21 determines second virtual light source 210b to be turned on from second virtual light sources 210b that are not currently turned on, in the same manner as in step s9 described above. Next, in step s60, control unit 2 determines and sets the brightness of second virtual light source 210b that was determined to be turned on in step s59 (also referred to as "turn-on determined second virtual light source 210b"), in the same manner as in step s10 described above. Next, in step s61, setting unit 22 turns on turn-on determined second virtual light source 210b. Thereafter, step s57 is executed again, and control unit 2 performs the same process thereafter.

[0126] After the virtual lighting environment reproduction process is completed, control unit 2 may perform an adjustment process to adjust the settings of virtual light source 210 that is turned on. Fig. 13 is a flowchart showing an example of the adjustment process. Hereinafter, virtual light source 210 that determination unit 21 has determined to be turned on in the virtual lighting environment reproduction process may be referred to as turned-on virtual light source 210. In this example, turned-on determined virtual light source 210 includes first virtual light source 210a that has been determined to be turned on in step s5 and second virtual light source 210b that has been determined to be turned on in step s9 or the like.

[0127] In this example, in the real space, a virtual placement surface corresponding to the top surface of the table on which the predetermined member 130 is placed, i.e., the placement surface on which the predetermined member 130 is placed, is placed in the target virtual space 200 in step s2 of the virtual light environment reproduction process before the adjustment process is performed. The virtual placement surface is placed on the bottom surface 203 of the target virtual space 200. The predetermined member information 181 in the storage unit 3 includes placement surface information that indicates the size, shape, reflectance, color, and material of the placement surface on which the predetermined member 130 is placed. Based on this placement surface information, the virtual space construction unit 20 places the virtual placement surface corresponding to the placement surface on which the predetermined member 130 is placed on the bottom surface 203 of the target virtual space 200.

[0128] After the virtual lighting environment reproduction process is completed, as shown in FIG. 13 , in step s71, the control unit 2 acquires a virtual color image (also referred to as a specific virtual color image) to be used in the adjustment process. For example, consider a case where the adjustment process is executed after the virtual lighting environment reproduction process of the example of FIG. 9 . In this case, the same process as step s5 is executed after the virtual lighting environment reproduction process, and the virtual camera 220 captures an image. At this time, the virtual camera 220 captures an image with all of the lighting-determined virtual light sources 210 turned on, and the virtual color image generated thereby is designated as the specific virtual color image. Hereinafter, the adjustment process executed after the virtual lighting environment reproduction process of the example of FIG. 9 may be referred to as the first adjustment process.

[0129] As another example, consider a case where an adjustment process is performed after the virtual lighting environment reproduction process of the example of FIG. 11 . In this case, the same processes as steps s26, s27, s41, and s42 are performed after the virtual lighting environment reproduction process to generate a composite virtual color image. At this time, a plurality of virtual color images captured with all of the lighting-determined virtual light sources 210 turned on are combined to generate a composite virtual color image. This composite virtual color image is designated as a specific virtual color image. Hereinafter, the adjustment process performed after the virtual lighting environment reproduction process of the example of FIG. 11 may be referred to as a second adjustment process.

[0130] As another example, consider a case where an adjustment process is performed after the virtual lighting environment reproduction process of the example of Fig. 12. In this case, the same process as step s57 is performed after the virtual lighting environment reproduction process, and virtual camera 220 takes an image. At this time, virtual camera 220 takes an image with all lighting-determined virtual light sources 210 turned on, and the virtual color image generated thereby is designated as a specific virtual color image. Hereinafter, the adjustment process performed after the virtual lighting environment reproduction process of the example of Fig. 12 may be referred to as a third adjustment process.

[0131] After step s71, in step s72, the determination unit 21 acquires shadow information related to the shadow of the predetermined member 130 formed in the target real space 100. In step s72 of the first adjustment process, the determination unit 21 acquires the shadow information based on the color image 126 in the storage unit 3. For example, the determination unit 21 extracts, from the color image 126, a shadow image in which the shadow of the predetermined member 130 formed on the placement surface on which the predetermined member 130 is placed is reflected. For example, the determination unit 21 sets the extracted shadow image as the shadow information.

[0132] In step s72 of the second adjustment process and the third adjustment process, the determination unit 21 acquires shadow information based on the composite color image generated in step s23. For example, the determination unit 21 extracts a shadow image from the composite color image, in which the shadow of the specified member 130 formed on the placement surface on which the specified member 130 is placed is reflected. For example, the determination unit 21 sets the extracted shadow image as the shadow information.

[0133] When the shadow information is acquired in step s72, step s73 is executed. In step s73, the determination unit 21 acquires virtual shadow information related to the virtual shadow of the virtual specified member 230 formed in the target virtual space 200 based on the specific virtual color image acquired in step s71. The specific virtual color image shows the virtual specified member 230 and the virtual placement surface in a state where all of the lighting-determined virtual light sources 210 determined to be turned on in the virtual light environment reproduction processing are turned on. For example, the determination unit 21 extracts, from the specific virtual color image, a virtual shadow image showing the virtual shadow of the virtual specified member 230 formed on the virtual placement surface on which the virtual specified member 230 is placed. For example, the determination unit 21 sets the extracted virtual shadow image as virtual shadow information.

[0134] After step s73, in step s74, determination unit 21 adjusts the settings of each lit-on determined virtual light source 210 based on the shadow information acquired in step s72 and the virtual shadow information acquired in step s73.

[0135] In step s74, determination unit 21 adjusts, for example, the setting value of the position of each lighting-determined virtual light source 210 based on the shadow information and the virtual shadow information. When the virtual lighting environment execution process is completed, the position of each lighting-determined virtual light source 210 is set as shown in the above-described Fig. 7. Based on the shadow information and the virtual shadow information, determination unit 21 changes and adjusts the setting value of the position of each lighting-determined virtual light source 210 from the setting value shown in Fig. 7 so that the shape of the virtual shadow of virtual specified member 230 formed on the virtual placement surface approaches the shape of the shadow of specified member 130 formed on the placement surface on which specified member 130 is placed.

[0136] For example, the determination unit 21 calculates a shape difference between the shape of a shadow shown in a shadow image as shadow information and the shape of a virtual shadow shown in a virtual shadow image as virtual shadow information. This shape difference indicates a current difference between the shape of the virtual shadow of the virtual predetermined member 230 formed on the virtual placement surface and the shape of the shadow of the predetermined member 130 formed on the placement surface on which the predetermined member 130 is placed. Then, based on the calculated shape difference and the current positional relationship between each lighting-determined second virtual light source 210b and the virtual predetermined member 230, the determination unit 21 calculates a position of each lighting-determined virtual light source 210 such that the shape of the virtual shadow of the virtual predetermined member 230 formed on the virtual placement surface approaches the shape of the shadow of the predetermined member 130 formed on the placement surface on which the predetermined member 130 is placed. In other words, the determination unit 21 calculates a position of each lighting-determined virtual light source 210 such that the difference between the shape of the virtual shadow of the virtual predetermined member 230 formed on the virtual placement surface and the shape of the shadow of the predetermined member 130 formed on the placement surface on which the predetermined member 130 is placed is smaller than the current shape. Then, determination unit 21 determines the obtained positions as the setting values ​​of the final positions of each lighting-determined virtual light source 210. In this way, the setting values ​​of the positions of each lighting-determined virtual light source 210 are adjusted.

[0137] When determination unit 21 adjusts the setting value of the position of each lighting-determined virtual light source 210, setting unit 22 moves each lighting-determined virtual light source 210 so that the position of each lighting-determined virtual light source 210 becomes the adjusted setting value. As a result, the setting of the position of each lighting-determined virtual light source 210 is adjusted, and the shape of the virtual shadow of virtual predetermined member 230 formed on the virtual placement surface becomes closer to the shape of the shadow of predetermined member 130 formed on the placement surface on which predetermined member 130 is placed. Therefore, the lighting environment in target real space 100 is more appropriately reproduced in target virtual space 200. When determination unit 21 moves the position of each lighting-determined virtual light source 210, it updates the position identification information of each lighting-determined virtual light source 210 included in the setting information in storage unit 3 in accordance with the movement.

[0138] In this example, for example, lighting-determined second virtual light source 210b moves on curved surface 202 of target virtual space 200. Furthermore, even after lighting-determined second virtual light source 210b moves, the optical axis of lighting-determined second virtual light source 210b after the movement coincides with the normal to the location on curved surface 202 where lighting-determined second virtual light source 210b after the movement is located.

[0139] In this way, the setting value of the position of the lighting determination virtual light source 210 is adjusted based on shadow information regarding the shadow of the specified member 130 formed in the target real space 100 and virtual shadow information regarding the virtual shadow of the virtual specified member 230 formed in the target virtual space 200, so that the setting value of the position of the lighting determination virtual light source 210 can be appropriately adjusted in order to reproduce the lighting environment in the target real space 100 in the target virtual space 200.

[0140] Note that when the position of lighting-determined virtual light source 210 moves, the brightness of virtual surface 235 corresponding to lighting-determined virtual light source 210 changes. Before the adjustment process is performed, lighting-determined virtual light source 210 is positioned as shown in Fig. 7 above, and therefore light 211 from lighting-determined virtual light source 210 is irradiated perpendicularly onto virtual surface 235 corresponding to lighting-determined virtual light source 210. In contrast, when the position of lighting-determined virtual light source 210 is moved in the adjustment process, light 211 from lighting-determined virtual light source 210 is irradiated obliquely onto virtual surface 235 corresponding to lighting-determined virtual light source 210, and the brightness of virtual surface 235 becomes darker.

[0141] Therefore, in step s74, determination unit 21 may adjust the setting value of the position of lighting-determined virtual light source 210 and then adjust the setting value of the brightness of that lighting-determined virtual light source 210. For example, in step s74 of the first adjustment process, after the position of each lighting-determined virtual light source 210 is moved, virtual camera 220 takes an image and generates a virtual color image. Then, similar to step s8 described above, determination unit 21 determines the brightness of imaginary side surface 235 corresponding to that lighting-determined virtual light source 210 and the brightness of real side surface 135 corresponding to that virtual side surface 235, based on the generated virtual color image and color image 126 in storage unit 3. Next, determination unit 21 determines a second brightness difference value obtained by subtracting the brightness of real side surface 135 corresponding to that virtual side surface 235 from the brightness of virtual side surface 235 corresponding to that lighting-determined virtual light source 210. Next, determination unit 21 determines the brightness of lighting-determined virtual light source 210 such that the brightness of imaginary side surface 235 corresponding to lighting-determined virtual light source 210 is increased by the second brightness difference value. Determination unit 21 then sets the determined brightness as the final setting value of the brightness of lighting-determined virtual light source 210. In this way, the setting value of the brightness of lighting-determined virtual light source 210 is adjusted. Determination unit 21 adjusts the setting value of the brightness of each lighting-determined virtual light source 210 in the same manner.

[0142] In step s74 of the second adjustment process, after the position of each lighting-determined virtual light source 210 has been moved, steps s26, s27, and s41 are executed again. If step s41 returns YES and step s42 is executed, a composite virtual color image is obtained, showing virtual predetermined member 230 in a state in which each lighting-determined virtual light source 210 is turned on after the movement. Similar to step s28, determination unit 21 calculates the brightness of virtual side surface 235 corresponding to lighting-determined virtual light source 210 and the brightness of real side surface 135 corresponding to virtual side surface 235, based on the generated composite virtual color image and the composite color image generated in step s23 before the second adjustment process. Thereafter, control unit 2 operates similarly to step s74 of the first adjustment process to determine the final setting value of the brightness of each lighting-determined virtual light source 210 and adjust the setting value of the brightness of each lighting-determined virtual light source 210.

[0143] In step s74 of the third adjustment process, after the position of each lighting-determined virtual light source 210 has moved, virtual camera 220 takes an image and generates a virtual color image. Then, similar to step s58 described above, determination unit 21 calculates the brightness of imaginary side surface 235 corresponding to lighting-determined virtual light source 210 and the brightness of real side surface 135 corresponding to virtual side surface 235, based on the generated virtual color image and the composite color image generated in step s23 before the third adjustment process. Thereafter, control unit 2 operates similarly to step s74 of the first adjustment process to determine the final setting value of the brightness of each lighting-determined virtual light source 210 and adjust the setting value of the brightness of each lighting-determined virtual light source 210.

[0144] When determination unit 21 adjusts the brightness setting value of each lighting-determined virtual light source 210, setting unit 22 adjusts the brightness of each lighting-determined virtual light source 210 so that the brightness of each lighting-determined virtual light source 210 becomes the adjusted setting value. This adjusts the brightness setting of each lighting-determined virtual light source 210, and the brightness of virtual side surface 235 corresponding to each lighting-determined virtual light source 210 becomes closer to the brightness of the corresponding real side surface 135. Thus, the lighting environment in target real space 100 is more appropriately reproduced in target virtual space 200. When determination unit 21 adjusts the brightness setting of each lighting-determined virtual light source 210, it updates the brightness information of each lighting-determined virtual light source 210 included in the setting information in storage unit 3 in accordance with the adjustment.

[0145] In this way, after the setting value of the position of the lighting determination virtual light source 210 is adjusted, the setting value of the brightness of the lighting determination virtual light source 210 is adjusted, so that the setting value of the brightness of the lighting determination virtual light source 210 can be appropriately adjusted in order to reproduce the lighting environment in the target real space 100 in the target virtual space 200.

[0146] Note that if the positional relationship between the camera 120 and the predetermined member 130 is fixed, for example because the camera 120 is fixed to a stand on which the predetermined member 130 is placed, the camera 120 may be a two-dimensional color camera rather than a three-dimensional camera. In this case, the memory unit 3 stores positional relationship information indicating the positional relationship between the camera 120 and the predetermined member 130, and the control unit 2 determines the positional relationship between the camera 120 and the predetermined member 130 based on this positional relationship information.

[0147] Furthermore, the setting information (also referred to as setting information 600) stored in storage unit 3 of processing device 1 may be input to a processing device other than processing device 1. In this case, the other processing device may set a plurality of virtual light sources 210 arranged in a virtual space in the same manner as described above with the setting contents indicated by setting information 600, and reproduce the light environment in target real space 100 in target virtual space 200 on the other processing device.

[0148] 14 is a schematic diagram showing an example of a processing device 900 to which setting information 600 has been input. The processing device 900 is, for example, a computer device. The processing device 900 includes, for example, a control unit 920, a storage unit 930, a communication unit 940, an input unit 950, and a display unit 960, similar to the processing device 1. The control unit 920, the storage unit 930, the communication unit 940, the input unit 950, and the display unit 960 have the same configurations as, for example, the control unit 2, the storage unit 3, the communication unit 4, the input unit 5, and the display unit 6 of the processing device 1, respectively.

[0149] The storage unit 930 stores a program 931 for controlling the processing device 900. The various functions of the control unit 920 are realized, for example, by a CPU included in the control unit 920 executing the program 931 in the storage unit 930. The storage unit 930 also stores setting information 600 generated by the processing device 1. For example, the communication unit 940 receives the setting information 600 from the processing device 1. The control unit 920 stores the setting information 600 received by the communication unit 940 in the storage unit 930.

[0150] The control unit 920 includes a setting unit 921. The setting unit 921 is a functional block formed in the control unit 920, for example, by the CPU of the control unit 920 executing a program 931 in the storage unit 930. Note that all or some of the functions of the setting unit 921 may be realized by a hardware circuit that does not require software to realize the function.

[0151] The setting unit 921 sets multiple virtual light sources in the virtual space according to the setting content indicated by the setting information 600. The processing device 900 can be said to be a setting device 900 that sets multiple virtual light sources. Based on the position specification information of each virtual light source 210 included in the setting information 600, the setting unit 921 arranges the multiple virtual light sources in the virtual space such that the positional relationship of the multiple virtual light sources, the same number as the multiple virtual light sources 210, is the same as the positional relationship of the multiple virtual light sources 210. Then, the setting unit 921 sets the multiple virtual light sources in the virtual space according to the setting content indicated by the setting information 600. As a result, the light environment in the target real space 100 is reproduced in the target virtual space 200 in the setting device 900.

[0152] As described above, the processing device and the setting device have been described in detail, but the above description is merely an example in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be applied in combination as long as they are not mutually contradictory. It is understood that countless examples not illustrated can be envisioned without departing from the scope of this disclosure.

[0153] This disclosure includes the following:

[0154] In one embodiment, (1) a processing device includes a virtual space construction unit that places a virtual predetermined member having the same shape as a predetermined member, which is a polyhedron, to be placed in a target real space in a target virtual space in which at least one virtual light source is set, and a determination unit that determines the setting contents of the at least one virtual light source that can reproduce the light environment of the target real space in the target virtual space based on a camera image that captures multiple faces of the polyhedron under the light environment of the target real space and a virtual camera image that captures the virtual predetermined member illuminated by the at least one virtual light source in the target virtual space.

[0155] (2) In the processing device of (1) above, the at least one virtual light source includes a plurality of virtual light sources, and the determination unit determines the setting content based on a plurality of virtual camera images in which the virtual specified component is illuminated by a plurality of virtual light sources in different combinations.

[0156] (3) In the processing device of (1) or (2) above, the at least one virtual light source includes a plurality of virtual light sources, and the determination unit determines whether to turn on or off the plurality of virtual light sources.

[0157] (4) In the processing device of (3) above, the determination unit determines the brightness of a virtual light source that has been determined to be turned on, among the plurality of virtual light sources.

[0158] (5) In the processing device of (3) or (4) above, the determination unit acquires brightness information of a first surface included in the surface of the specified member based on the camera image, acquires brightness information of a first virtual surface included in the surface of the virtual specified member and corresponding to the first surface based on the virtual camera image, and determines whether to turn on or off a first virtual light source that emits light toward the first virtual surface based on the brightness information of the first surface and the brightness information of the first virtual surface.

[0159] (6) In the processing device of (5) above, the determination unit acquires brightness information of a second surface included in the surface of the specified member based on the camera image, acquires brightness information of a second virtual surface included in the surface of the virtual specified member, which corresponds to the second surface, based on the virtual camera image showing the virtual specified member in a state where the first virtual light source that has been decided to be turned on is turned on, and decides whether to turn on or off the second virtual light source that emits light toward the second virtual surface based on the brightness information of the second surface and the brightness information of the second virtual surface.

[0160] (7) In the processing device of (5) above, the determination unit determines the brightness of the first virtual light source that has been determined to be turned on based on brightness information of the first surface and brightness information of the first virtual surface.

[0161] (8) In the processing device of (6) above, the determination unit determines the brightness of the first virtual light source that has been determined to be turned on based on brightness information of the first surface and brightness information of the first virtual surface, and determines the brightness of the second virtual light source that has been determined to be turned on based on brightness information of the second surface and brightness information of the second virtual surface.

[0162] (9) In the processing device according to any one of (1) to (8) above, the determination unit determines a color of the at least one virtual light source.

[0163] (10) In the processing device of (9) above, the surface of the specified member includes a white pattern, and the determination unit determines the color of the at least one virtual light source based on a white pattern image in which the white pattern is captured and included in the camera image.

[0164] (11) In any one of the processing devices (2) to (8) above, the determination unit acquires shadow information regarding the shadow of the specified component formed in the target real space from the camera image, acquires virtual shadow information regarding the virtual shadow of the virtual specified component formed in the target virtual space based on the virtual camera image in which the virtual specified component is captured when at least one virtual light source among the multiple virtual light sources that has been determined to be turned on is turned on, and adjusts the setting value of the position of the at least one virtual light source that has been determined to be turned on based on the shadow information and the virtual shadow information.

[0165] (12) In the processing device of (11) above, the determination unit adjusts the setting value of the position of the at least one virtual light source that has been determined to be lit, and then adjusts the setting value of the brightness of the at least one virtual light source.

[0166] (13) In the processing device according to any one of (1) to (12) above, the camera image is a composite camera image obtained by combining a plurality of camera images in which the predetermined member is captured in different postures.

[0167] (14) In any one of the processing devices (1) to (13) above, the processing device further includes a setting unit that sets the at least one virtual light source according to the setting content determined by the determination unit.

[0168] (15) The setting device includes a setting unit that sets at least one of a plurality of virtual light sources in a virtual space with the setting content determined by the determination unit provided in any one of the processing devices (1) to (14) above.

[0169] (16) The polyhedron is a polyhedron used to set at least one virtual light source and has four or more faces visible to the camera.

[0170] (17) The polyhedron of (16) above, having a flat surface disposed at the vertex.

[0171] (18) The polyhedron of (17) above, having a white pattern arranged on the plane.

[0172] (19) The program is a program for causing a computer device to function as any one of the processing devices (1) to (14) above.

[0173] (20) The program is a program for causing a computer device to function as the setting device described above in (15).

[0174] REFERENCE SIGNS LIST 1 Processing device 20 Virtual space construction unit 21 Determination unit 30, 931 Program 100 Target real space 126 Color image (color camera image) 130 Predetermined member 131, 231 Top surface 132a White pattern 135, 235 Side surface 200 Target virtual space 210 Virtual light source 230 Virtual predetermined member 900 Setting device (processing device) 921 Setting unit

Claims

1. a virtual space construction unit that places a virtual predetermined member having the same shape as a predetermined member that is a polyhedron and is placed in the target real space, in the target virtual space in which at least one virtual light source is set; a determination unit that determines setting details of the at least one virtual light source capable of reproducing the light environment of the target real space in the target virtual space, based on a camera image capturing a plurality of faces of the polyhedron under a light environment of the target real space and a virtual camera image capturing the virtual predetermined member illuminated by the at least one virtual light source in the target virtual space; A processing device comprising:

2. 2. The processing device according to claim 1, the at least one virtual light source includes a plurality of virtual light sources; The determination unit determines the setting content based on a plurality of virtual camera images in which the virtual predetermined member is illuminated by a plurality of virtual light sources in different combinations.

3. 2. The processing device according to claim 1, the at least one virtual light source includes a plurality of virtual light sources; The processing device wherein the determination unit determines whether to turn on or off the plurality of virtual light sources.

4. 4. The processing device according to claim 3, The determination unit determines the brightness of a virtual light source that has been determined to be turned on, from among the plurality of virtual light sources.

5. 4. The processing device according to claim 3, The determination unit acquiring brightness information of a first surface included in the surface of the predetermined member based on the camera image; acquiring brightness information of a first virtual surface that is included in the surface of the virtual predetermined member and corresponds to the first surface based on the virtual camera image; a processing device that determines whether to turn on or off a first virtual light source that emits light toward the first virtual surface based on brightness information of the first surface and brightness information of the first virtual surface.

6. 6. The processing device according to claim 5, The determination unit acquiring brightness information of a second surface included in the surface of the predetermined member based on the camera image; acquiring brightness information of a second virtual surface, which is included in a surface of the virtual predetermined member and corresponds to the second surface, based on the virtual camera image in which the virtual predetermined member is captured in a state in which the first virtual light source determined to be turned on is turned on; a processing device that determines whether to turn on or off a second virtual light source that emits light toward the second virtual surface based on brightness information of the second surface and brightness information of the second virtual surface.

7. 6. The processing device according to claim 5, The processing device wherein the determination unit determines the brightness of the first virtual light source that has been determined to be turned on, based on brightness information of the first surface and brightness information of the first virtual surface.

8. 7. The processing device according to claim 6, The determination unit determining a brightness of the first virtual light source determined to be turned on based on brightness information of the first surface and brightness information of the first virtual surface; a processing device that determines the brightness of the second virtual light source that has been determined to be turned on, based on brightness information of the second surface and brightness information of the second virtual surface.

9. 2. The processing device according to claim 1, The processing device, wherein the determination unit determines a color of the at least one virtual light source.

10. 10. The processing device according to claim 9, the surface of the predetermined member includes a white pattern; The determination unit determines the color of the at least one virtual light source based on a white pattern image that is included in the camera image and that captures the white pattern.

11. 3. The processing device according to claim 2, The determination unit acquiring, from the camera image, shadow information relating to a shadow of the predetermined member formed in the target real space; acquire virtual shadow information relating to a virtual shadow of the virtual specified member formed in the target virtual space based on the virtual camera image in which the virtual specified member is captured in a state in which at least one virtual light source determined to be turned on among the plurality of virtual light sources is turned on; a processing device that adjusts a setting value of a position of the at least one virtual light source that has been determined to be lit, based on the shadow information and the virtual shadow information.

12. 12. The processing device according to claim 11, The processing device wherein the determination unit adjusts a setting value of a position of the at least one virtual light source that has been determined to be turned on, and then adjusts a setting value of brightness of the at least one virtual light source that has been determined to be turned on.

13. 2. The processing device according to claim 1, The processing device, wherein the camera image is a composite camera image obtained by combining a plurality of camera images in which the predetermined member is captured in different postures.

14. 2. The processing device according to claim 1, a setting unit that sets the at least one virtual light source according to the setting content determined by the determination unit.

15. A setting device comprising: a setting unit that sets at least one virtual light source set in a virtual space with the setting content determined by the determination unit included in the processing device according to claim 1 .

16. A polyhedron used to define at least one virtual light source, A polyhedron with four or more faces visible to the camera.

17. 17. The polyhedron of claim 16, A polyhedron with flat surfaces at its vertices.

18. 18. The polyhedron of claim 17, A polyhedron having a white pattern disposed on the plane.

19. A program for causing a computer device to function as the processing device according to any one of claims 1 to 14.

20. A program for causing a computer device to function as the setting device according to claim 15.