Information processing device, information processing method, and computer program

Abstract

This information processing device 1 comprises: a first generation unit C3 that generates a target image by imaging a target; a second generation unit C that generates a first projection image by imaging the target on which a first light pattern having a first wavelength is projected, and generates a second projection image by imaging the target on which a second light pattern having a second wavelength different from the first wavelength is projected; a measurement unit 11 that uses the first projection image and measures a first three-dimensional shape and uses the second projection image and measures a second three-dimensional shape; an output unit 20 that outputs a three-dimensional shape; and a selection unit 12 that selects the three-dimensional shape output by the output unit 20 on the basis of the target image, the first three-dimensional shape, and the second three-dimensional shape.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] Patent Document 1 discloses a technique for determining a three-dimensional shape by combining one or more light sources and a camera. Patent Document 2 discloses a technique for calculating relative phase values ​​at each part of an object based on luminance values ​​of an image of the object onto which a first light pattern whose luminance changes with a first period is projected, calculating absolute phase values ​​at each part of the object based on luminance values ​​and relative phase values ​​of an image of the object onto which a second light pattern whose period is longer than the first period is projected, and calculating three-dimensional coordinates at each part of the object based on the absolute phase values. Note that other prior art documents related to this disclosure include Patent Documents 3 to 6.

[0003] JP 2001-12925 A, International Publication No. 2019 / 177066 A, JP 2004-295813 A, JP 2018-173731 A, JP 2003-208601 A, JP 2006-163871 A

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of the information processing device includes a first generation unit that generates an object image by capturing an image of the object, a second generation unit that generates a first projected image by capturing an image of the object onto which a first light pattern having a first wavelength is projected, and generates a second projected image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected, a measurement means that measures a first three-dimensional shape using the first projected image and measures a second three-dimensional shape using the second projected image, an output unit that outputs the three-dimensional shape, and a selection means that selects the three-dimensional shape to be output by the output unit based on the object image, the first three-dimensional shape, and the second three-dimensional shape.

[0006] One aspect of the information processing method includes generating an object image by capturing an image of the object, generating a first projected image by capturing an image of the object onto which a first light pattern having a first wavelength is projected, generating a second projected image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected, measuring a first three-dimensional shape using the first projected image, measuring a second three-dimensional shape using the second projected image, and selecting a three-dimensional shape to be output by an output unit that outputs the three-dimensional shape based on the object image, the first three-dimensional shape, and the second three-dimensional shape.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method that generates an object image by capturing an image of the object, generates a first projected image by capturing an image of the object onto which a first light pattern having a first wavelength is projected, generates a second projected image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected, measures a first three-dimensional shape using the first projected image, measures a second three-dimensional shape using the second projected image, and selects a three-dimensional shape to be output by an output unit that outputs the three-dimensional shape based on the object image, the first three-dimensional shape, and the second three-dimensional shape.

[0008] FIG. 1 is a block diagram showing the configuration of an information processing device in a first embodiment. FIG. 2 is a schematic diagram showing the configuration of an information processing system in a second embodiment. FIG. 3 is a block diagram showing the configuration of an information processing device in the second embodiment. FIG. 4 is a flowchart showing the flow of information processing operations of the information processing device in the second embodiment. FIG. 5 is a diagram showing examples of four types of first periodic light patterns projected onto a target T. FIG. 6 is a conceptual diagram of information processing operations performed by the information processing device in the second embodiment. FIG. 7 is a schematic diagram showing the configuration of an information processing system in a third embodiment. FIG. 8 is a block diagram showing the configuration of an information processing device in the third embodiment. FIG. 9 is a flowchart showing the flow of information processing operations of the information processing device in the third embodiment. FIG. 10 is a conceptual diagram of information processing operations performed by the information processing device in the third embodiment. FIG. 11 is a block diagram showing the configuration of an information processing device in a fourth embodiment. FIG. 12 is a diagram showing an example of a periodic light pattern projected onto a target T. FIG. 13 is a schematic diagram showing the configuration of an information processing system in a fifth embodiment. FIG. 14 is a block diagram showing the configuration of an information processing device in the fifth embodiment. FIG. 15 is a flowchart showing the flow of information processing operations of the information processing device in the fifth embodiment. Fig. 16 is a block diagram showing the configuration of an information processing device in the sixth embodiment, and Fig. 17 is a flowchart showing the flow of information processing operations of the information processing device in the sixth embodiment.

[0009] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 to which the first embodiment of the information processing device, the information processing method, and the recording medium is applied. [1-1: Configuration of Information Processing Device 1]

[0011] FIG. 1 is a block diagram showing the configuration of an information processing device 1 according to a first embodiment. As shown in FIG. 1, the information processing device 1 includes a first generation unit C3, a second generation unit C, a measurement unit 11, a selection unit 12, and an output unit 20. The first generation unit C3 generates an object image by capturing an image of the object. The second generation unit C generates a first projected image by capturing an image of the object onto which a first light pattern having a first wavelength is projected, and generates a second projected image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected. The measurement unit 11 measures a first three-dimensional shape using the first projected image and measures a second three-dimensional shape using the second projected image. The output unit 20 outputs the three-dimensional shape. The selection unit 12 selects a three-dimensional shape to be output by the output unit 20 based on the object image, the first three-dimensional shape, and the second three-dimensional shape. [1-2: Technical Effects of the Information Processing Device 1]

[0012] The information processing device 1 in the first embodiment selects one of a plurality of measured three-dimensional shapes based on an image of the object, and therefore can acquire an appropriate three-dimensional shape that is likely to be the object. [2: Second Embodiment]

[0013] Next, a second embodiment of the information processing device, the information processing method, and the recording medium will be described. Hereinafter, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2 to which the second embodiment of the information processing device, the information processing method, and the recording medium is applied. [2-1: Measurement of the three-dimensional shape of an object]

[0014] The three-dimensional shape of an object can be measured by various methods, for example, measurement using a stereo camera, measurement using a light pattern projection method, etc. The measured three-dimensional shape of a face is used for person matching as exemplified in Patent Documents 3 and 4 above, and is used for computer graphics (CG) in multimedia content as exemplified in Patent Documents 5 and 6 above.

[0015] The object to be measured must be stationary during measurement of its three-dimensional shape. When the object to be measured is a face, the face moves, so it is necessary to be able to measure it in a short time (and / or at high speed). Furthermore, high measurement accuracy is also required for person matching and realizing realistic CG. As a method for achieving these, methods using a sine wave pattern as a light pattern have been proposed, as disclosed in the above-mentioned Patent Documents 1 and 2.

[0016] The above-mentioned Patent Document 1 discloses a method for resolving the ambiguity and uniquely determining a three-dimensional shape by adding one or more projectors that project light patterns or cameras that photograph the object to be measured. The above-mentioned Patent Document 2 discloses a method for projecting two types of light patterns with different periods and measuring the three-dimensional shape of an object to be measured with high accuracy and in a short time using a simpler system. The above-mentioned Patent Document 2 shows that when a sine wave is used as the short-period light pattern and a brightness gradient pattern is used as the long-period pattern as an example of two types of light patterns with different periods, it is possible to achieve both high speed, which allows the three-dimensional shape to be determined with a small number of projections, and high accuracy, which reduces the projection angle dependency of measurement error.

[0017] The measurement method disclosed in Patent Document 2 does not require phase unwrapping processing to smoothly connect relative phase values, and can calculate absolute phase values ​​for each pixel of a camera image capturing a pattern. However, when a luminance gradient pattern is used, the measurement may be affected by multiple reflections of the projected light pattern. For example, in recessed areas of an object (concave-shaped areas), the measurement may be affected by secondary and multiple reflections of the light pattern. In this case, the calculation of the absolute phase value may be erroneous. For example, when measuring the three-dimensional shape of a person's head and neck, there is a concave shape extending from the face to the neck, and the measurement may be affected by secondary and multiple reflections of the light pattern hitting the chin and the area from the chin to the neck, which may result in an erroneous calculation of the absolute phase value of the relevant area.

[0018] To deal with portions where the calculation of the absolute phase value is erroneous, such as a concave shape, a phase unwrapping process that smoothly connects relative phase values ​​may be used in combination with a measurement method that projects two types of light patterns with different periods. That is, an absolute phase value may be calculated using a phase value smoothly connected by the phase unwrapping process and an integer (n) multiple of the period (2π) of the periodic light pattern calculated by the measurement method that projects two types of light patterns with different periods.

[0019] However, even when a phase unwrapping process is used in combination with a measurement method that projects two types of light patterns with different periods, erroneous phase unwrapping may occur in areas with stepped, discontinuous shapes. For example, when measuring the three-dimensional shape of a person's head and neck, erroneous phase unwrapping may occur in areas with concave shapes extending from the face to the neck. For example, the neck may protrude in front of the face by an amount corresponding to a depth that is an integer (n) times the period (2π) of the light pattern. [2-2: Configuration of Information Processing System S2]

[0020] Fig. 2 illustrates the overall configuration of an information processing system S2 including an information processing device 2 to which the second embodiment is applied. Fig. 2 is a schematic diagram illustrating the configuration of an information processing system S2 including an information processing device 2 to which the second embodiment is applied.

[0021] As shown in FIG. 2 , the information processing system S2 in the second embodiment includes an information processing device 2, a projector P, a camera C as a second generation unit, and a third camera C3 as a first generation unit. The projector P and the camera C are each arranged to face the direction of the target T. The third camera C3 is arranged to face the direction of the target T from a different direction from the camera C. The third camera C3 generates a facing image of the target T captured from the side facing the target T. The facing image may be an example of a target image. In the second embodiment, the target T whose three-dimensional shape is measured may be the head and neck of a person. Furthermore, the target T whose three-dimensional shape is measured may be a part other than the head and neck, such as a fingerprint or the shape of a finger.

[0022] The projector P is configured to be able to project a periodic light pattern onto the target T. The projector P is not particularly limited, and may be, for example, a DLP (Digital Light Processing) projector, a liquid crystal projector, or the like. A DLP projector or a liquid crystal projector can project any light pattern at high speed, which is preferable for shortening the time required to measure the shape of the target T. A shorter measurement time is particularly advantageous for measuring the three-dimensional shape of a moving object (moving body), such as when performing face recognition of a person. Details of the light pattern projected onto the target T by the projector P will be described later.

[0023] The camera C and the third camera C3 are configured to be able to capture images of the target T. The camera C may include a solid-state imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera C may also include an optical system that forms an image of the subject on the imaging surface of the solid-state imaging element, a signal processing circuit that processes the output of the solid-state imaging element to obtain a luminance value for each pixel, and the like. [2-3: Configuration of the Information Processing Device 2]

[0024] 3 is a block diagram showing the configuration of an information processing device 2 in the second embodiment. As shown in FIG. 3, the information processing device 2 includes a calculation device 21 and a storage device 22. The information processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not necessarily have to include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0025] The arithmetic device 21 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (e.g., an input device 24 described later) not shown in the drawings that is included in the information processing device 2. The arithmetic device 21 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the information processing device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the information processing device 2 are realized within the arithmetic device 21. In other words, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing operations (in other words, processes) to be performed by the information processing device 2. The arithmetic device 21 may output information to devices (not shown), such as other computers or cloud servers, that are provided outside the information processing device 2, via the communication device 23 (or other communication devices).

[0026] 3 shows an example of logical functional blocks implemented within the arithmetic device 21 to perform information processing operations. As shown in FIG. 3 , the arithmetic device 21 implements a measurement unit 211, which is a specific example of a "measurement means" described in the appendix below, a selection unit 212, which is a specific example of a "selection means" described in the appendix below, a light pattern control unit 213, and a camera control unit 214. However, at least one of the light pattern control unit 213 and the camera control unit 214 does not have to be implemented within the arithmetic device 21. The measurement unit 211 may include a relative phase calculation unit 2111, an absolute phase calculation unit 2112, and a coordinate calculation unit 2113. Details of the operations of the measurement unit 211, the selection unit 212, the light pattern control unit 213, and the camera control unit 214 will be described later with reference to FIGS. 4 to 6 .

[0027] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the information processing device 2 stores long-term. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0028] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The communication device 23 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus).

[0029] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0030] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-4: Information Processing Operation Performed by the Information Processing Device 2]

[0031] The flow of information processing operations performed by the information processing device 2 will be described with reference to Figs. 4 to 6. In this embodiment, a case will be described in which four types of first periodic light patterns are used as the multiple periodic light patterns. Fig. 4 is a flowchart showing the flow of information processing operations performed by the information processing device 2. Fig. 5 is a diagram showing an example of four types of first periodic light patterns projected onto the target T. Fig. 6 is a conceptual diagram of information processing operations performed by the information processing device 2.

[0032] 4 , the light pattern control unit 213 generates a light pattern to be projected onto the target T (step S20). The light pattern control unit 213 may store the generated light pattern in the storage device 22. The light pattern control unit 213 transmits the generated light pattern to the projector P.

[0033] The light pattern control unit 213 generates each of four types of first periodic light patterns. The first periodic light pattern is a short-period light pattern with a relatively short period. The first periodic light pattern may be a sinusoidal grating light pattern (hereinafter referred to as a sinusoidal pattern) in which the luminance varies sinusoidally. In this embodiment, the information processing device 2 measures the three-dimensional shape of the target T using a sinusoidal grating phase shift method. The sinusoidal grating phase shift method is a method in which a sinusoidal pattern is projected onto the target T while shifting the phase by a predetermined value, and the three-dimensional shape of the target T is identified based on an image captured of the target T onto which the light pattern is projected.

[0034] 5A and 5B are diagrams illustrating examples of four types of first periodic light patterns. Fig. 5A illustrates a light pattern including a pattern repeated 9 times in a captured image. Fig. 5B illustrates a light pattern including a pattern repeated 10 times in a captured image. Fig. 5C illustrates a light pattern including a pattern repeated 11 times in a captured image. Fig. 5D illustrates a light pattern including a pattern repeated 12 times in a captured image. That is, each of the four types of first periodic light patterns has a different period.

[0035] The light pattern control unit 213 generates a plurality of light patterns with different phases for each of the four types of first periodic light patterns. For example, the light pattern control unit 213 may generate four light patterns for each of the four types of first periodic light patterns, with the phases shifted by a quarter wavelength in the direction in which the pattern repeats. Hereinafter, the amount of phase shift in the direction in which the pattern repeats will be referred to as a "gap."

[0036] The light pattern control unit 213 generates a second periodic pattern having a longer period than each of the first periodic patterns. The light pattern control unit 213 may generate a light pattern including one pattern in the captured image as the second periodic pattern. The second periodic light pattern may be a luminance gradient pattern in which the luminance value changes linearly at a constant rate.

[0037] The camera control unit 214 acquires an image of the target T onto which the periodic light pattern is projected (referred to as a "pattern projection image") (step S21). The camera control unit 214 may acquire four pattern projection images onto which one of the four types of sine wave patterns is projected, and a pattern projection image onto which a luminance gradient pattern is projected. The camera control unit 214 instructs the light pattern control unit 213 to cause the projector P to project the light pattern. The light pattern control unit 213 controls the operation of the projector P so that the projector P projects the light pattern. The camera control unit 214 controls the operation of the camera C and controls the timing of shooting by the camera C, etc. In this way, the camera control unit 214 acquires the image captured by the camera C.

[0038] The relative phase calculation unit 2111 calculates a phase value θ for each pixel (x, y) based on the four pattern projection images onto which the sine wave pattern is projected (step S22). The relative phase calculation unit 2111 calculates a phase value θ at each part of the target T corresponding to each pixel (x, y) based on the brightness value of each pixel (x, y) of the four pattern projection images onto which the sine wave pattern is projected. [Calculation of Phase Value θ]

[0039] When an object T onto which a sine wave pattern is projected is photographed by a camera C, the brightness value I(x, y, t) of the gap t at the (x, y) coordinates of the obtained image is expressed by the following formula (1), where A is the amplitude of the sine wave, θ is the phase value, and B is the bias (center value of the sine wave): [Formula (1)]

[0040] The light pattern projected by the projector P has a different phase value θ for each angle viewed from the projector P, so if the phase value θ at the coordinates (x, y) can be determined, the three-dimensional position corresponding to the coordinates (x, y) can be determined.

[0041] Since the unknown quantities in the above equation (1) are the amplitude A, the phase value θ, and the bias B, the phase value θ can be calculated with a minimum of three pattern projection images. If four or more pattern projection images can be acquired, the phase value θ can be calculated with higher accuracy by the least squares method or the like.

[0042] When pattern projection images corresponding to gap t=0, gap t=π / 2, gap t=π, and gap t=3π / 2 are acquired, the brightness value I(x, y, t) of the coordinates (x, y) in each pattern projection image is expressed by the following formulas (2) to (5): [Formula (2)] [Formula (3)] [Formula (4)] [Formula (5)]

[0043] From the above formulas (2) to (5), the amplitude A, the phase value θ, and the bias B can be expressed by the following formulas (6) to (8) by the least squares method. [Formula (6)] [Formula (7)] [Formula (8)]

[0044] The phase value θ is a value for each period of the sine wave pattern, i.e., a value between -π and π. In other words, the phase value θ is a relative phase value, and coordinates cannot be uniquely determined by the phase value θ alone. Therefore, in order to find the absolute phase value when a light pattern including a pattern with multiple periods as exemplified in FIG. 5 is projected, it is necessary to estimate the position of the pattern on the image. In this embodiment, the variable ω is used to estimate the position of the pattern on the image. [Calculation of variable ω]

[0045] The calculation operation of the variable ω will be described using an example in which the second periodic light pattern is a luminance gradient pattern. When an object T onto which a luminance gradient pattern is projected is photographed by a camera C, the luminance value K(x, y, t) of the gap t at the (x, y) coordinate of the obtained image is expressed as in the following formula (9). Here, A" is the amplitude, B" is the bias, and ω is a variable whose value changes linearly in the range of -1≦ω≦1. [Formula (9)]

[0046] When a projector capable of projecting any light pattern, such as a liquid crystal projector or a DLP projector, is used as the projector P, it is possible to rapidly switch between a sine wave pattern and a luminance gradient pattern and project them using a single projector P. When a sine wave pattern and a luminance gradient pattern generated from light emitted from the same light source are projected using a single projector P, it can be assumed that the basic physical characteristics of the projector P when projecting these light patterns are the same. In other words, when the same projector P projects a sine wave pattern and a luminance gradient pattern onto the target T, the following equation (10) holds. Therefore, the variable ω can be found by simply photographing the target T once onto which the luminance gradient pattern is projected. [Equation (10)] A=A", B=B"

[0047] When an image of the target T onto which the luminance gradient pattern is projected is captured only once, the luminance value K(x, y, t) and the variable ω are expressed by the following equations (11) and (12): [Equation (11)] [Formula (12)]

[0048] Alternatively, the object T onto which the luminance gradient pattern is projected may be photographed twice and the variable ω may be calculated by the least squares method.

[0049] The absolute phase calculation unit 2112 calculates an absolute phase value for each pixel (x, y) based on the variable ω and the phase value θ as the relative phase value calculated by the relative phase calculation unit 2111 (step S23). The absolute phase calculation unit 2112 estimates the position on the image of a corresponding pattern of a light pattern including multiple periodic patterns based on the variable ω, and calculates the absolute phase value at each part of the target T corresponding to each pixel (x, y).

[0050] The coordinate calculation unit 2113 calculates the three-dimensional coordinates of the projection point (X, Y, Z) on the target T corresponding to the pixel (x, y) based on the absolute phase value for each pixel (x, y) calculated by the absolute phase calculation unit 2112 (step S24). The coordinate calculation unit 2113 calculates the absolute coordinate value in three-dimensional space of the projection point on the target T corresponding to the pixel by the principle of triangulation based on the absolute phase value of the pixel and the optical arrangement of the projector P and the camera C. In other words, the measurement unit 211 measures the three-dimensional shape of the target T based on the pattern projection image of the target T onto which the periodic light pattern is projected.

[0051] As described above, when the shape of a person's head and neck has a step, for example, the calculation of the absolute phase value at the step portion may be erroneous. In other words, phase unwrapping may fail at the step portion.

[0052] Furthermore, even when the same object T is projected and photographed from the same direction, the absolute phase value calculated by the projected light pattern may differ depending on the shape of the object T. In other words, even when the head and neck of the same person are projected and photographed from the same direction, the absolute phase value calculated by the projected light pattern may differ depending on the degree of difference in level between the face and neck of the person, and the measured three-dimensional shape may differ.

[0053] Therefore, in this embodiment, a plurality of periodic light patterns are used to measure the three-dimensional shape of the target T corresponding to each of the plurality of periodic light patterns, thereby making it possible to acquire a three-dimensional shape that accurately represents the shape of the target T.

[0054] The measurement unit 211 determines whether or not the three-dimensional shapes corresponding to all four types of first periodic light patterns have been measured (step S25). If there is a first periodic light pattern for which the three-dimensional shape has not yet been measured (step S25: No), the measurement unit 211 measures the three-dimensional shapes of the first periodic light patterns for which the three-dimensional shape has not yet been measured. That is, the measurement unit 211 measures the three-dimensional shapes of the multiple targets T corresponding to each of the multiple (four types in this embodiment) periodic light patterns.

[0055] When each of the three-dimensional shapes corresponding to each of the four types of periodic light patterns has been measured (step S25: Yes), the selection unit 212 selects one of the four types of three-dimensional shapes of the object T based on each of the four types of three-dimensional shapes of the object T and an image of the object T (step S26).

[0056] The absolute phase value is a value obtained by adding or subtracting an integer multiple of the period of the first periodic light pattern to or from the relative phase value. Therefore, in other words, the selection unit 212 selects a three-dimensional shape that does not include a deviation related to an integer multiple of the period of the first periodic light pattern corresponding to the three-dimensional shape of the object (face) to be measured (or, even if a deviation is included, the deviation is very small).

[0057] FIG. 6 is a conceptual diagram of the operation of step S26. FIG. 6(a) illustrates a two-dimensional image captured by the third camera C3, and FIGS. 6(b) to 6(d) illustrate projected images of the measured three-dimensional shape projected onto a plane perpendicular to the optical axis of the third camera C3. The selection unit 212 may select the corresponding three-dimensional shape, for example, when the projected image of the three-dimensional shape matches (or is very similar to) the two-dimensional image. In the example illustrated in FIG. 6, FIGS. 6(a) and 6(b) match (or are very similar), but FIGS. 6(a) and 6(c) and 6(d) are not similar. In FIG. 6(c), the neck is positioned closer to the subject than in FIG. 6(a). In FIG. 6(d), the neck is positioned further back than in FIG. 6(a). Therefore, in the example illustrated in FIG. 6, the selection unit 212 selects the three-dimensional shape corresponding to the projected image illustrated in FIG. 6(b). For example, the selection unit 212 may calculate the similarity score of the images and select the three-dimensional shape corresponding to the image with the highest score. Note that, for the sake of explanation, the left and right sides are illustrated equally clearly in Figures 6(b) to 6(d), but in reality, the part that is not included in the shooting range of camera C is not included in the projected image either.

[0058] Alternatively, instead of the two-dimensional image captured by the third camera C3, a single captured image captured by camera C may be used to estimate the depth. The depth of the face and the depth of the neck may be estimated from a single captured image, and the resulting three-dimensional shape with the smallest depth difference may be selected by comparing the depths. For example, the depth of the captured image may be estimated using an estimation model trained by deep learning. For example, the technology disclosed in the document “[2012.04012] Learning an Animatable Detailed 3D Face Model from In-The-Wild Images (arxiv.org)” may be employed to estimate the depth of the captured image.

[0059] Alternatively, the three-dimensional shape of the object (face) to be measured may be selected by combining the three-dimensional shape of the first side and the three-dimensional shape of the second side. First, the three-dimensional shape of the first side is measured using a projected image captured from the first side of the target T, and the face and neck portions are separated. Similarly, the three-dimensional shape of the second side is measured using a projected image captured from the second side of the target T, and the face and neck portions are separated. The three-dimensional shape of the first side and the three-dimensional shape of the second side are combined to generate a combined three-dimensional shape so that the three-dimensional shape and depth of the face portion on the first side match the three-dimensional shape and depth of the face portion on the second side. The selection unit 212 may compare the depth of the neck portion on the first side and the neck portion on the second side, and select a combined three-dimensional shape that minimizes the difference in depth.

[0060] Although the present embodiment describes a case in which each of the three-dimensional shapes of a plurality of targets T corresponding to each of a plurality of first periodic light patterns is measured, each of the three-dimensional shapes of a plurality of targets T corresponding to each of a plurality of second light patterns may also be measured. That is, a relative phase value at each portion of the target T may be calculated based on the target T onto which the first periodic light pattern is projected, and multiple absolute phase values ​​may be calculated from the luminance value and relative phase value at each pixel of the image of the target T onto which the multiple second light patterns are projected. In this case, the multiple second light patterns may each have a different shape. For example, one of the second light patterns may be a sine wave pattern. Another of the second light patterns may be a sawtooth wave pattern. Another of the second light patterns may be a triangular wave pattern. The multiple periodic light patterns may include a predetermined light pattern and a light pattern having a different phase from the predetermined light pattern. Light patterns having different phases will be described in detail in another embodiment.

[0061] Note that the number of types of periodic light patterns is four, but this is merely an example, and the number of types may be more or less than four. The number of types can be determined depending on the application of the information processing device 2. [2-4: Technical Effects of the Information Processing Device 2]

[0062] When calculating absolute phase values ​​after calculating relative phase values, as in the above-described sinusoidal grating phase shift method, the calculation of absolute phase values ​​is often erroneous in areas where the depth position changes discontinuously, compared to areas where the depth position changes continuously. For example, in the case of the head and neck of a human body, the quality of the three-dimensional shape is often poor in areas where the depth position changes discontinuously, such as the boundary between the chin and neck. In other words, in the three-dimensional shape, the depth of the neck relative to the face may be inaccurate. For example, in the three-dimensional shape, the neck may appear to be closer than the face.

[0063] The information processing device 2 in the second embodiment can acquire a highly accurate three-dimensional shape even when the object T for three-dimensional shape measurement has an uneven portion, because it selects an appropriate three-dimensional shape from three-dimensional shapes measured based on a plurality of different light patterns. [3: Third Embodiment]

[0064] Next, a third embodiment of the information processing device, the information processing method, and the recording medium will be described. Hereinafter, the third embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 3 to which the third embodiment of the information processing device, the information processing method, and the recording medium is applied. [3-1: Configuration of Information Processing System S3]

[0065] Fig. 7 illustrates the overall configuration of an information processing system S3 including an information processing device 3 to which the third embodiment is applied. Fig. 7 is a schematic diagram illustrating the configuration of an information processing system S3 including an information processing device 3 to which the third embodiment is applied.

[0066] As shown in Fig. 7, an information processing system S3 in the third embodiment includes a measuring device M in addition to an information processing device 3, a projector P, and a camera C. The measuring device M measures the distance to a target T. The measuring device M measures the distance to the target T using any distance measuring mechanism such as a distance sensor or a TOF (Time of Flight) sensor. [3-2: Configuration of Information Processing Device 3]

[0067] The configuration of the information processing device 3 in the third embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the information processing device 3 in the third embodiment.

[0068] As shown in FIG. 8 , the information processing device 3 in the third embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 3 in the second embodiment. Furthermore, the information processing device 3 in the third embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment. However, the information processing device 3 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 3 in the third embodiment differs from the information processing device 2 in the second embodiment in that the calculation device 21 further includes a division unit 315, which is a specific example of the "division means" described in the appendix below, a distance information acquisition unit 316, which is a specific example of the "acquisition means" described in the appendix below, and a correction unit 317, which is a specific example of the "correction means" described in the appendix below. The division unit 315 may include a chin detection unit 3151, which is a specific example of the "detection means" described in the appendix below, and an estimation unit 3152, which is a specific example of the "estimation means" described in the appendix below. Other features of the information processing device 3 may be the same as other features of the information processing device 2. Therefore, in the following, only the parts that differ from the already described embodiment will be described in detail, and the description of other overlapping parts will be omitted as appropriate. [3-3: Information Processing Operation Performed by the Information Processing Device 3]

[0069] The flow of the information processing operation performed by the information processing device 3 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the information processing operation performed by the information processing device 3. In the third embodiment, after a three-dimensional shape is selected, a correction operation for the three-dimensional shape is performed.

[0070] 9 , in the third embodiment, after the operation of step S26, the chin detection unit 3151 detects the chin of the target T included in the two-dimensional image (step S30). The two-dimensional image may be the image used in step S26.

[0071] The estimation unit 3152 estimates the skin-colored region vertically above the chin detected in step S30 as the face region. The estimation unit 3152 also estimates the skin-colored region vertically above the chin detected in step S30 as the neck region (step S31).

[0072] Based on the estimation in step S31, the dividing unit 315 divides the object T included in the two-dimensional image into a face region and a neck region (step S32). The dividing unit 315 extracts a face portion corresponding to the face region and a neck portion corresponding to the neck region from the three-dimensional shape selected by the selecting unit 212.

[0073] The estimation of the face region and the neck region and the segmentation of the face region and the neck region may be performed using a model trained by deep learning. For example, the segmentation may be performed by employing a region segmentation technique such as semantic segmentation.

[0074] The distance information acquisition unit 316 acquires distance measurement information that measures the distance to the face region and the distance to the neck region (step S33). The measurement device M measures the distance to the face region and the distance to the neck region. The distance information acquisition unit 316 acquires each distance measured by the measurement device M.

[0075] The correction unit 317 determines whether or not correction of the positional relationship is necessary based on the positional relationship between the face portion and the neck portion extracted from the three-dimensional shape and the distance measurement information acquired in step S33 (step S34). If correction of the positional relationship is necessary (step S34: Yes), the correction unit 317 corrects the positional relationship between the face portion and the neck portion extracted from the three-dimensional shape (step S35). The correction unit 317 corrects the positional relationship between the face portion and the neck portion based on the distance measurement information. The correction unit 317 corrects the positional relationship by an amount related to an integer (n) multiple of the period (2π) of the periodic light pattern corresponding to the three-dimensional shape.

[0076] Incorrect positional relationships between the face and neck are often caused by errors in calculation of the absolute phase value. As described above, the absolute phase value is a value obtained by adding or subtracting an integer (n) multiple of the period (2π) of the periodic light pattern to or from the relative phase value. That is, incorrect positional relationships between the face and neck are often caused by an error in estimating the integer (n) by the absolute phase calculation unit 2112. Therefore, an accurate three-dimensional shape can be obtained by the correction unit 317 correcting the positional relationship by an amount related to an integer (n) multiple of the period (2π) of the periodic light pattern corresponding to the three-dimensional shape.

[0077] FIG. 10 is a conceptual diagram of the operation of step S35. FIG. 10(a) illustrates an example of a two-dimensional image captured by camera C. FIG. 10(b) illustrates a case where, compared with the distance measurement information, the position of neck portion N is closer to the face portion F in the three-dimensional shape. In this case, the correction unit 317 corrects the position of neck portion N to the back by an amount related to an integer (n) multiple of the period (2π). FIG. 10(c) illustrates a case where, compared with the distance measurement information, the position of neck portion N is closer to the face portion F in the three-dimensional shape. In this case, the correction unit 317 corrects the position of neck portion N to the front by an amount related to an integer (n) multiple of the period (2π). [3-4: Technical Effects of Information Processing Device 3]

[0078] The information processing device 3 in the third embodiment corrects the positional relationship between the face and neck portions of a three-dimensional shape, thereby enabling an accurate three-dimensional shape to be acquired. The information processing device 3 corrects the positional relationship between the face and neck portions using an amount related to an integer multiple of the period of the periodic light pattern corresponding to the three-dimensional shape. Therefore, even if calculation of the absolute phase value at the boundary between the face and neck portions fails, a three-dimensional shape with an accurate positional relationship between the face and neck portions can be acquired. Furthermore, the information processing device 3 corrects the positional relationship between the face and neck portions of the three-dimensional shape based on distance measurement information measuring the distance to the face region and the distance to the neck region, thereby enabling accurate correction of the positional relationship. Furthermore, the information processing device 3 estimates the skin-colored region vertically above the chin in the image as the face region, and estimates the skin-colored region vertically above the chin as the neck region, thereby enabling accurate distances to the face region and the neck region to be acquired. [4: Fourth Embodiment]

[0079] Next, a fourth embodiment of the information processing device, the information processing method, and the recording medium will be described. Hereinafter, the fourth embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 4 to which the fourth embodiment of the information processing device, the information processing method, and the recording medium is applied. [4-1: Configuration of the information processing device 4]

[0080] The configuration of the information processing device 4 in the fourth embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the configuration of the information processing device 4 in the fourth embodiment.

[0081] As shown in FIG. 11 , the information processing device 4 in the fourth embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. Furthermore, the information processing device 4 in the fourth embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. However, the information processing device 4 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 4 in the fourth embodiment differs from the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment in that a feature detection unit 418 is further implemented within the calculation device 21. Furthermore, the operation of the light pattern control unit 413 differs from the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. Other features may be the same as other features of at least one of the information processing device 2 and the information processing device 3. Therefore, in the following, only the differences from the previously described embodiment will be described in detail, and the description of other overlapping parts will be omitted as appropriate. [4-2: Information Processing Operation Performed by Information Processing Device 4]

[0082] The information processing operation performed by the information processing device 4 will be described with reference to Fig. 12. Fig. 12 is a diagram showing an overview of the information processing operation performed by the information processing device 4.

[0083] The feature detection unit 418 detects feature parts of the target T from the image of the target T. As illustrated in Fig. 12(a), the feature detection unit 418 may detect an Adam's apple Fc, a chin Fd, and a mole Fe as feature parts of the target T. Note that the Adam's apple, the chin, and a mole are examples of feature parts, and the feature detection unit 418 may detect any feature part suitable for the application, such as the tip of the nose or any end of the ear.

[0084] The measuring device M measures the distance to each of the characteristic features of the target T, and the distance information acquisition unit 416 acquires distance measurement information measuring the distance to each of the characteristic features of the target T. The distance information acquisition unit 416 calculates the three-dimensional coordinates of the characteristic features of the target T based on the positions and distances of the characteristic features of the target T in the image.

[0085] The light pattern control unit 413 generates a light pattern such that a predetermined phase of the light pattern is projected onto the characteristic feature detected by the feature detection unit 418. Specifically, the correspondence between the three-dimensional coordinates obtained by the measurement device M and the two-dimensional coordinates of the image (light pattern) projected by the projector P is determined in advance by calibration using an object with known dimensions, and a pattern is generated in which a predetermined phase of the light pattern is projected onto the characteristic feature based on the correspondence. The light pattern control unit 413 may generate a light pattern such that the boundary between adjacent patterns is projected onto the Adam's apple Fc, as exemplified in FIG. 12( c). The light pattern control unit 413 may also generate a light pattern such that the boundary between adjacent patterns is projected onto the chin Fd, as exemplified in FIG. 12( d). The light pattern control unit 413 may also generate a light pattern such that the boundary between adjacent patterns is projected onto a mole Fe, as exemplified in FIG. 12( e). That is, the light pattern control unit 413 may generate a light pattern that has a phase different from that of the light pattern exemplified in FIG. 12B, depending on the characteristic portion.

[0086] The light pattern control unit 413 may generate a plurality of light patterns with different phases for each of the four types of light patterns shown in Figures 12(b), 12(c), 12(d), and 12(e). For example, the light pattern control unit 413 may generate four light patterns for each of the four types of light patterns, with the phases shifted by a quarter wavelength in the direction in which the pattern is repeated.

[0087] The measurement unit 211 measures the three-dimensional shape for each of the four types of light patterns shown in FIGS. 12(b), 12(c), 12(d), and 12(e).

[0088] While the present embodiment describes measuring the three-dimensional shapes of multiple targets T corresponding to multiple first periodic light patterns, it is also possible to measure the three-dimensional shapes of multiple targets T corresponding to multiple second light patterns. That is, the multiple second periodic light patterns may have different phases so that the predetermined phases of the multiple second periodic light patterns are projected onto characteristic portions of the target T. In this case, a relative phase value for each portion of the target T may be calculated based on the target T onto which the first periodic light pattern is projected, and multiple absolute phase values ​​may be calculated from the luminance value and relative phase value for each pixel of an image of the target T onto which the multiple second light patterns are projected. [4-3: Technical Effects of the Information Processing Device 4] The information processing device 4 in the fourth embodiment measures the three-dimensional shape of the target T based on a light pattern projection image captured by projecting a light pattern so that the predetermined phase of the light pattern is projected onto the characteristic portions of the target T, thereby improving the likelihood of acquiring an accurate three-dimensional shape. [5: Fifth Embodiment]

[0089] Next, a fifth embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, the fifth embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 5 to which the fifth embodiment of the information processing device, the information processing method, and the recording medium is applied. [5-1: Configuration of Information Processing System S5]

[0090] Fig. 13 illustrates the overall configuration of an information processing system S5 including an information processing device 5 to which the fifth embodiment is applied. Fig. 13 is a schematic diagram illustrating the configuration of an information processing system S5 including an information processing device 5 to which the fifth embodiment is applied.

[0091] 13, an information processing system S5 in the fifth embodiment includes an information processing device 5 and a projection photography apparatus 100. The projection photography apparatus 100 includes a first projector P1, a first camera C1, a second projector P2, a second camera C2, and a third camera C3.

[0092] The first projector P1 projects a periodic light pattern onto the target T from a first side of the target T. The first projector P1 may project a periodic light pattern onto the target T from the first side relative to the midline of the target T. FIG. 13 illustrates a case where the first projector P1 projects onto the target T from the left side as viewed from the target T. Below, as an example, a case where the first projector P1 projects onto the target T from the left side as viewed from the target T will be described.

[0093] The first camera C1 generates an image of the target T photographed from a first side of the target T. The first camera C1 may generate an image of the target T photographed from the first side relative to the midline of the target T. FIG. 13 illustrates an example in which the first camera C1 photographs the target T from the left side as viewed from the target T. Below, as an example, a description will be given of a case in which the first camera C1 photographs the target T from the left side as viewed from the target T.

[0094] The second projector P2 projects a periodic light pattern onto the target T from a second side of the target T. The second projector P2 may project a periodic light pattern onto the target T from the second side relative to the midline of the target T. FIG. 13 illustrates a case where the second projector P2 projects onto the target T from the right side as viewed from the target T. Below, as an example, a case where the second projector P2 projects onto the target T from the right side as viewed from the target T will be described.

[0095] The second camera C2 generates an image of the target T photographed from the second side of the target T. The second camera C2 may generate an image of the target T photographed from the second side relative to the midline of the target T. Fig. 13 illustrates an example in which the second camera C2 photographs the target T from the right side as viewed from the target T. Below, as an example, a description will be given of a case in which the second camera C2 photographs the target T from the right side as viewed from the target T.

[0096] The third camera C3 generates a facing image by capturing an image of the target T from the side facing the target T. Fig. 13 illustrates a case where the third camera C3 captures an image of the target T from the front as viewed from the target T. [5-2: Configuration of the information processing device 5]

[0097] The configuration of the information processing device 5 in the fifth embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the configuration of the information processing device 5 in the fifth embodiment.

[0098] As shown in FIG. 14 , the information processing device 5 in the fifth embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment to the information processing device 4 in the fourth embodiment. Furthermore, the information processing device 5 in the fifth embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment to the information processing device 4 in the fourth embodiment. However, the information processing device 5 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 5 in the fifth embodiment differs from the information processing device 2 in the second embodiment to the information processing device 4 in the fourth embodiment in that a generation unit 519 is further implemented within the calculation device 21. Furthermore, because the projection photography apparatus 100 includes two projectors and three cameras, the operation of the light pattern control unit 513 and the camera control unit 514 differs from the information processing device 2 in the second embodiment to the information processing device 4 in the fourth embodiment. Other features of the information processing device 5 may be the same as other features of at least one of the information processing devices 2, 3, and 4. Therefore, in the following, only differences from the already described embodiments will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [5-3: Information Processing Operation Performed by Information Processing Device 5]

[0099] The flow of information processing operations performed by the information processing device 5 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the information processing operations performed by the information processing device 5.

[0100] 15 , the generation unit 519 acquires a three-dimensional shape of the left side (step S50). The three-dimensional shape acquired in step S50 is a selected left-side three-dimensional shape from among multiple left-side three-dimensional shapes measured based on a left-side light pattern projection image of the target T onto which a periodic light pattern is projected from the left side of the midline of the target T. The left-side three-dimensional shape may have been corrected by the correction unit 317.

[0101] The generation unit 519 acquires a right-side three-dimensional shape (step S51). The three-dimensional shape acquired in step S51 is a right-side three-dimensional shape selected from multiple right-side three-dimensional shapes measured based on a right-side light pattern projection image of the target T on which a light pattern is projected from the right side of the midline of the target T. The right-side three-dimensional shape may have been corrected by the correction unit 317.

[0102] 15 illustrates an example in which the three-dimensional shape of the left side is acquired first, followed by the three-dimensional shape of the right side. However, the order of the operations required to acquire the three-dimensional shape of the left side and the operations required to acquire the three-dimensional shape of the right side does not have to be the order illustrated in FIG. 15. For example, the light pattern control unit 513 may control the first projector P1 and the second projector P2 to project the same light pattern at the same time. The camera control unit 514 may control the first camera C1 and the second camera C2 to capture images of the target T onto which the same light pattern is projected at the same time. In other words, the left light pattern projection image for measuring the three-dimensional shape of the left side and the right light pattern projection image for measuring the three-dimensional shape of the right side may be captured simultaneously.

[0103] The generation unit 519 generates one three-dimensional shape based on the three-dimensional shape on the left side and the three-dimensional shape on the right side (step S52). Since one three-dimensional shape includes both the three-dimensional shape on the left side and the three-dimensional shape on the right side, it can be used to generate two-dimensional images capturing the object T from various angles. For example, when one three-dimensional shape is projected onto a plane parallel to the front of the object T, an image equivalent to the image obtained by photographing the object T with the third camera C3 can be obtained. [5-4: Technical Effects of the Information Processing Device 5]

[0104] The information processing device 5 in the fifth embodiment can acquire a single accurate three-dimensional shape that includes the three-dimensional shape of the left side of the target T and the three-dimensional shape of the right side of the target T. By using this three-dimensional shape, two-dimensional images of the target T viewed from various angles can be acquired. [6: Sixth Embodiment]

[0105] Next, a sixth embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, the sixth embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 6 to which the sixth embodiment of the information processing device, the information processing method, and the recording medium is applied.

[0106] In the second to fifth embodiments, multiple three-dimensional shapes on the left side of the target T are measured based on a pattern projection image of the target T onto which a light pattern is projected from the left side, and one of the multiple three-dimensional shapes is selected. In contrast, in the sixth embodiment, multiple face data corresponding to each of multiple periodic light patterns is generated, and one of the face data is selected. The face data is a single piece of data generated based on the three-dimensional shape on the left side and the three-dimensional shape on the right side. [6-1: Configuration of Information Processing System S6]

[0107] Fig. 16 illustrates the overall configuration of an information processing system S6 including an information processing device 6 to which the sixth embodiment is applied. Fig. 16 is a schematic diagram illustrating the configuration of an information processing system S6 including an information processing device 6 to which the sixth embodiment is applied.

[0108] 16, an information processing system S6 in the sixth embodiment includes an information processing device 6 and a projection photography apparatus 100. The projection photography apparatus 100 may be the same as the projection photography apparatus 100 described in the fifth embodiment. [6-2: Configuration of the Information Processing Device 6]

[0109] The configuration of the information processing device 6 in the sixth embodiment will be described with reference to FIG. 16 . As shown in FIG. 16 , the information processing device 6 in the sixth embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment to the information processing device 5 in the fifth embodiment. Furthermore, the information processing device 6 in the sixth embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment to the information processing device 5 in the fifth embodiment. However, the information processing device 6 does not need to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 6 in the sixth embodiment differs from the information processing device 2 in the second embodiment to the information processing device 5 in the fifth embodiment in that the measurement unit 611 implemented in the calculation device 21 includes a generation unit 6114. Other features of the information processing device 6 may be the same as at least one other feature of the information processing device 2 to the information processing device 5. Therefore, the following will describe in detail the differences from the already-described embodiments, and will omit appropriate descriptions of other overlapping features. [6-3: Information Processing Operation Performed by Information Processing Device 6]

[0110] The flow of information processing operations performed by the information processing device 6 will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the information processing operations performed by the information processing device 6.

[0111] 17 , the generation unit 6114 measures the three-dimensional shape of the left side of the target T based on a pattern projection image of the left side of the target T onto which one type of periodic light pattern from among the multiple types of periodic light patterns is projected from the left side (step S60). The generation unit 6114 measures the three-dimensional shape of the right side of the target T based on a pattern projection image of the right side of the target T onto which one type of periodic light pattern from among the multiple types of periodic light patterns is projected from the right side (step S61). The generation unit 6114 generates three-dimensional face data of the target T based on the three-dimensional shape of the left side and the three-dimensional shape of the right side (step S62).

[0112] The measurement unit 611 determines whether or not three-dimensional face data corresponding to all of the multiple types of periodic light patterns has been generated (step S63). If three-dimensional face data corresponding to all of the multiple types of periodic light patterns has been generated (step S63: Yes), the camera control unit 614 controls the third camera C3 to capture a front image of the target T (step S64). The selection unit 612 selects one of the three-dimensional face data of the multiple targets T based on the three-dimensional face data of the multiple targets T and the front image of the target T (step S65).

[0113] The dividing unit 615 divides the object T included in the front image into a face region and a neck region (step S66). The distance information acquiring unit 616 acquires distance measurement information that measures the distance to the face region and the distance to the neck region (step S67).

[0114] The correction unit 617 determines whether or not correction of the positional relationship is necessary based on the distance measurement information acquired in step S67 and the positional relationship between the face portion and the neck portion in the three-dimensional face data (step S68). If correction of the positional relationship is necessary (step S68: Yes), the correction unit 617 corrects the positional relationship between the face portion and the neck portion in the three-dimensional face data (step S69). The correction unit 617 corrects the positional relationship based on the distance measurement information. The correction unit 617 corrects the positional relationship using an amount related to an integer (n) multiple of the period (2π) of the periodic light pattern corresponding to the three-dimensional face data. [6-4: Technical Effects of Information Processing Device 6]

[0115] The information processing device 6 in the sixth embodiment, like the information processing device 5 in the fifth embodiment, can acquire one accurate face data set including the three-dimensional shape of both the left side and the right side of the target T. By using this face data, two-dimensional images of the target T viewed from various angles can be acquired. [7: Supplementary Note]

[0116] The following supplementary notes are further disclosed regarding the above-described embodiments: [Supplementary Note 1] An information processing device comprising: a first generation unit that generates an object image by capturing an image of an object; a second generation unit that generates a first projected image by capturing an image of the object onto which a first light pattern having a first wavelength is projected, and generates a second projected image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected; measurement means that measures a first three-dimensional shape using the first projected image and measures a second three-dimensional shape using the second projected image; an output unit that outputs the three-dimensional shapes; and selection means that selects a three-dimensional shape to be output by the output unit based on the object image, the first three-dimensional shape, and the second three-dimensional shape. [Supplementary Note 2] The imaging system includes a first-side projection unit that projects a periodic light pattern onto the target from a first side of the target, and a second-side projection unit that projects a periodic light pattern onto the target from a second side of the target, wherein the second generation unit includes a first-side capture unit that captures an image of the target from the first side to generate an image of the first side, and a second-side capture unit that captures an image of the target from the second side to generate an image of the second side, wherein the measurement unit measures the first three-dimensional shape of the first side using the first projected image captured from the first side, and measures the second three-dimensional shape of the first side using the second projected image captured from the first side, and the selection unit selects one of a plurality of three-dimensional shapes based on the target image, the first three-dimensional shape of the first side, and the second three-dimensional shape of the first side, The information processing device according to Appendix 1, further comprising: the measuring means measures the first three-dimensional shape of the second side using the second projection image captured from the second side; and measures the second three-dimensional shape of the second side using the second projection image captured from the second side; the selecting means selects one of a plurality of three-dimensional shapes based on the target image, the first three-dimensional shape of the second side, and the second three-dimensional shape of the second side; and a generating means that generates one three-dimensional shape based on the three-dimensional shape of the first side and the three-dimensional shape of the second side selected by the selecting means.[Supplementary Note 3] The information processing device according to Supplementary Note 1, comprising: a first-side projection unit that projects a periodic light pattern onto the target from a first side of the target; and a second-side projection unit that projects a periodic light pattern onto the target from a second side of the target; wherein the second generation unit includes a first-side capture unit that captures an image of the target from the first side to generate an image of the first side, and a second-side capture unit that captures an image of the target from the second side to generate an image of the second side; and the measurement means measures a first three-dimensional shape using the first projected image captured from the first side and the second projected image captured from the first side, and measures a second three-dimensional shape using the first projected image captured from the second side and the second projected image captured from the second side. [Supplementary Note 4] The information processing device according to any one of Supplements 1 to 3, wherein the plurality of periodic light patterns including the first light pattern and the second light pattern include at least one of a first light pattern, a second light pattern having a different wavelength from the first light pattern, a third light pattern having a different phase from the first light pattern, and a fourth light pattern having a different shape from the first light pattern. [Supplementary Note 5] The information processing device according to any one of Supplements 1 to 3, wherein the target is a head and neck of a person, and the information processing device comprises: a dividing means for dividing the target included in an image of the target into a face region and a neck region; and a correcting means for correcting a positional relationship between a face portion corresponding to the face region and a neck portion corresponding to the neck region in a selected three-dimensional shape selected by the selecting means. [Supplementary Note 6] The information processing device according to the Supplementary Note, wherein the correcting means corrects the positional relationship using an amount related to an integer multiple of a period of the periodic light pattern corresponding to the selected three-dimensional shape. [Supplementary Note 7] The information processing device according to Supplementary Note 5, further comprising: an acquisition means for acquiring distance measurement information that measures the distance to the face region and the distance to the neck region, wherein the correction means corrects the positional relationship based on the distance measurement information. [Supplementary Note 8] The information processing device according to Supplementary Note 5, further comprising: a detection means for detecting a chin of the subject included in the image of the subject; and an estimation means for estimating a skin-colored region vertically above the chin to be the face region, and estimating another skin-colored region vertically above the chin to be the neck region.[Supplementary Note 9] The information processing device according to Supplementary Note 4, further comprising: feature detection means for detecting a feature portion of the object from an image of the object, wherein a predetermined phase of the third light pattern is projected onto the feature portion. [Supplementary Note 10] An information processing method comprising: generating an object image by capturing an image of the object; generating a first projected image by capturing an image of the object onto which a first light pattern having a first wavelength is projected; generating a second projected image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected; measuring a first three-dimensional shape using the first projected image; measuring a second three-dimensional shape using the second projected image; and selecting a three-dimensional shape to be output by an output unit that outputs three-dimensional shapes, based on the object image, the first three-dimensional shape, and the second three-dimensional shape. [Supplementary Note 11] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method comprising: generating an object image by capturing an image of the object; generating a first projection image by capturing an image of the object onto which a first light pattern having a first wavelength is projected; generating a second projection image by capturing an image of the object onto which a second light pattern having a second wavelength different from the first wavelength is projected; measuring a first three-dimensional shape using the first projection image; measuring a second three-dimensional shape using the second projection image; and selecting a three-dimensional shape to be output by an output unit that outputs three-dimensional shapes based on the object image, the first three-dimensional shape, and the second three-dimensional shape. [Supplementary Note 12] An information processing device comprising: a measurement means that measures the three-dimensional shape of an object based on a light pattern projection image of the object onto which periodic light patterns are projected, and measures each of a plurality of three-dimensional shapes corresponding to each of the plurality of periodic light patterns; and a selection means that selects one of the plurality of three-dimensional shapes based on each of the plurality of three-dimensional shapes and an image of the object. [Supplementary Note 13] An information processing method comprising: measuring a three-dimensional shape of an object based on a light pattern projection image of the object onto which a periodic light pattern is projected; measuring each of a plurality of three-dimensional shapes corresponding to each of the plurality of periodic light patterns; and selecting one of the plurality of three-dimensional shapes based on each of the plurality of three-dimensional shapes and an image of the object.[Supplementary Note 14] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, which comprises: measuring a three-dimensional shape of an object based on a light pattern projection image of the object onto which a periodic light pattern is projected; measuring each of a plurality of three-dimensional shapes corresponding to each of the plurality of periodic light patterns; and selecting one of the plurality of three-dimensional shapes based on each of the plurality of three-dimensional shapes and an image of the object.

[0117] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. Information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0118] 1, 2, 3, 4, 5, 6 Information processing device 11, 211, 611 Measurement unit 12, 212, 612 Selection unit 20 Output unit S2, S3, S5, S6 Information processing system P Projector C Camera 2111 Relative phase calculation unit 2112 Absolute phase calculation unit 2113 Coordinate calculation unit 213, 413, 513, 613 Light pattern control unit 214, 514, 614 Camera control unit M Measuring device 315, 615 Division unit 3151 Jaw detection unit 3152 Estimation unit 316, 616 Distance information acquisition unit 317, 617 Correction unit 418 Feature detection unit 100 Projection photography device P1 First projector C1 First camera P2 Second projector C2 Second camera C3 Third camera 519,6114 Generation part

Claims

1. A first generation unit that generates an image of the target by capturing an image of the target, A second generation unit generates a first projection image by imaging the object onto which a first light pattern having a first wavelength is projected, and generates a second projection image by imaging the object onto which a second light pattern having a second wavelength different from the first wavelength is projected. A measuring means for measuring a first three-dimensional shape using the first projection image and measuring a second three-dimensional shape using the second projection image, A selection means for selecting a three-dimensional shape to be output based on the aforementioned target image, the first three-dimensional shape, and the second three-dimensional shape, An output unit that outputs the three-dimensional shape of the output target, An information processing device equipped with the following features.

2. A first-side projection unit that projects a periodic light pattern onto the target from the first side of the target, and The object is further equipped with a second projection unit that projects a periodic light pattern onto the object from its second side. The second generation unit includes a first side imaging unit that photographs the object from the first side and generates an image of the first side, and a second side imaging unit that photographs the object from the second side and generates an image of the second side. The measurement means measures the first three-dimensional shape of the first side using the first projection image captured from the first side, and measures the second three-dimensional shape of the first side using the second projection image captured from the first side. The selection means selects the three-dimensional shape of the first side based on the target image, the first three-dimensional shape of the first side, and the second three-dimensional shape of the first side. The measurement means measures the first three-dimensional shape of the second side using the second projection image captured from the second side, and measures the second three-dimensional shape of the second side using the second projection image captured from the second side. The selection means selects the third-dimensional shape of the second side based on the target image, the first third-dimensional shape of the second side, and the second third-dimensional shape of the second side. The system includes a generation means that generates a single three-dimensional shape based on the first three-dimensional shape and the second three-dimensional shape selected by the selection means. The information processing apparatus according to claim 1.

3. A first-side projection unit that projects a periodic light pattern onto the target from the first side of the target, and The object is further equipped with a second projection unit that projects a periodic light pattern onto the object from its second side. The second generation unit includes a first side imaging unit that photographs the object from the first side and generates an image of the first side, and a second side imaging unit that photographs the object from the second side and generates an image of the second side. The measurement means measures the first three-dimensional shape using the first projection image taken from the first side and the second projection image taken from the first side, and measures the second three-dimensional shape using the first projection image taken from the second side and the second projection image taken from the second side. The information processing apparatus according to claim 1.

4. A plurality of periodic light patterns, including the first light pattern and the second light pattern, A second light pattern having a different wavelength from the first light pattern, A third light pattern having a different phase from the first light pattern, and This includes at least one of a fourth light pattern having a different shape from the first light pattern. An information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned subject is the head and neck of a person. A division means that divides the subject included in the image of the subject into a face region and a neck region, Correction means for correcting the positional relationship between the face portion corresponding to the face region and the neck portion corresponding to the neck region in the three-dimensional shape selected by the selection means. An information processing apparatus according to any one of claims 1 to 3, comprising

6. The correction means corrects the positional relationship using a quantity relating to an integer multiple of the period of the periodic light pattern corresponding to the selected three-dimensional shape. The information processing apparatus according to claim 5.

7. The system includes means for acquiring distance measurement information, which measures the distance to the face region and the distance to the neck region. The correction means corrects the positional relationship based on the distance measurement information. The information processing apparatus according to claim 5.

8. The division means includes a detection means for detecting the jaw portion of the target included in the target image, and an estimation means for estimating the skin-colored region vertically above the jaw portion as the face region, and the skin-colored region vertically above the jaw portion as the neck region. The information processing apparatus according to claim 5.

9. The process of generating an image of a target by capturing an image of that target. A first projection image is generated by imaging the object onto which a first light pattern having a first wavelength is projected. A second projection image is generated by imaging the object onto which a second light pattern having a second wavelength different from the first wavelength is projected. To measure the first three-dimensional shape using the first projection image, Using the second projection image, measure the second three-dimensional shape. Based on the aforementioned target image, the first three-dimensional shape, and the second three-dimensional shape, a three-dimensional shape to be output is selected, and This includes obtaining the three-dimensional shape of the output target, The information processing method performed by computers.

10. On the computer, The process of generating an image of a target by capturing an image of that target. A first projection image is generated by imaging the object onto which a first light pattern having a first wavelength is projected. A second projection image is generated by imaging the object onto which a second light pattern having a second wavelength different from the first wavelength is projected. To measure the first three-dimensional shape using the first projection image, Using the second projection image, measure the second three-dimensional shape. Based on the aforementioned target image, the first three-dimensional shape, and the second three-dimensional shape, a three-dimensional shape to be output is selected, and This includes outputting the three-dimensional shape of the output target. A computer program that executes an information processing method.