Information processing system, information processing method, and recording medium

Abstract

This information processing system S1 comprises: a projection unit 101 that projects a light pattern onto a target; a photographing unit 102 that photographs the target on which the light pattern is projected to generate a target image; an identification unit 11 that identifies a target region on the basis of brightness information relating to brightness of each of a plurality of partial images included in the target image; and a measurement unit 13 that measures the three-dimensional shape of the target on the basis of the target region.

Description

Information processing system, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing system, an information processing method, and a recording medium.

[0002] Patent Document 1 discloses a technology in which relative phase values ​​at each part of a measured object are calculated based on the luminance values ​​of an image of the measured object onto which a first light pattern whose luminance changes with a first period is projected, absolute phase values ​​at each part of the measured object are calculated based on the luminance values ​​and relative phase values ​​of an image of the measured object onto which a second light pattern whose period is longer than the first period is projected, and three-dimensional coordinates at each part of the measured object are calculated based on the absolute phase values.

[0003] International Publication No. 2019 / 177066

[0004] An object of the present disclosure is to provide an information processing system, an information processing method, and a recording medium that aim to improve upon the techniques disclosed in prior art documents.

[0005] One aspect of the information processing system includes a projection means for projecting a light pattern onto an object, an imaging means for imaging the object onto which the light pattern is projected to generate an object image, an identification means for identifying an area of ​​the object based on luminance information regarding the luminance of each of a plurality of partial images included in the object image, and a measurement means for measuring the three-dimensional shape of the object based on the area of ​​the object.

[0006] One aspect of the information processing method involves projecting a light pattern onto an object, photographing the object onto which the light pattern is projected to generate an image of the object, and for each of multiple partial images included in the object image, identifying the area of ​​the object based on brightness information regarding the brightness of the partial image, and measuring the three-dimensional shape of the object.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method that projects a light pattern onto an object, photographs the object onto which the light pattern is projected to generate an image of the object, and, for each of a plurality of partial images included in the object image, identifies an area of ​​the object based on brightness information regarding the brightness of the partial image, and measures the three-dimensional shape of the object.

[0008] FIG. 1 is a block diagram showing an example of a configuration of an information processing system according to the present disclosure. FIG. 2 is a schematic diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 3 is a schematic diagram showing an example of a configuration of an information processing system according to the present disclosure. FIG. 4 is a schematic diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 5 is a flowchart showing an example of a processing operation of an information processing device according to the present disclosure. FIG. 6 is a schematic diagram showing an example of a configuration of an information processing system according to the present disclosure. FIG. 7 is a schematic diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 8 is a flowchart showing an example of a processing operation of an information processing device according to the present disclosure. FIG. 9 is a conceptual diagram of an example of a processing operation of an information processing system according to the present disclosure. FIG. 10 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 11 is a flowchart showing an example of a processing operation of an information processing device according to the present disclosure. FIG. 12 is a conceptual diagram of an example of a processing operation of an information processing system according to the present disclosure.

[0009] Hereinafter, an information processing system, an information processing method, and a recording medium according to an embodiment will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing system, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing system, the information processing method, and the recording medium will be described using an information processing system S1 according to the present disclosure. [1-1: Configuration of the information processing system S1]

[0011] 1 is a block diagram showing the configuration of an information processing system S1 according to the present disclosure. As shown in FIG. 1, the information processing system S1 includes a projection unit 101, an image capture unit 102, and an identification unit 11.

[0012] The projection unit 101 projects a light pattern onto a target. The imaging unit 102 images the target onto which the light pattern is projected, and generates a target image. The identification unit 11 identifies a region of the target based on luminance information relating to the luminance of each of a plurality of partial images included in the target image. The measurement unit 13 measures the three-dimensional shape of the target based on the region of the target. [1-2: Technical Effects of the Information Processing System S1]

[0013] The information processing system S1 according to the present disclosure identifies the target for each of a plurality of partial images included in the target image, and therefore can measure the three-dimensional shape of the target with high accuracy.

[0014] A second embodiment of an information processing system, an information processing method, and a recording medium will be described below. Hereinafter, the second embodiment of an information processing system, an information processing method, and a recording medium will be described using an information processing system S2 according to the present disclosure. [2-1: Configuration of the information processing system S2]

[0015] FIG. 2 is a schematic diagram showing the configuration of an information processing system S2 according to the present disclosure. As shown in FIG. 2, the information processing system S2 includes a projector P, a camera C, and an information processing device 2. The projector P and the camera C are each arranged to face the direction of an object T. In the second embodiment, the object T whose three-dimensional shape is measured may be a person's head and neck, mainly consisting of the person's face. The person's head and neck is the part of the person from the shoulders up. The person's head and neck includes the person's eye area. Furthermore, the object T whose three-dimensional shape is measured may be a part other than the head and neck, such as a fingerprint or the shape of a finger.

[0016] The projector P is configured to be able to project a light pattern onto the target T. The projector P is not particularly limited, but may be, for example, a DLP (Digital Light Processing) projector, a liquid crystal projector, or the like. DLP projectors and liquid crystal projectors are capable of projecting any light pattern at high speed, which is preferable for shortening the time required to measure the shape of the target T. Shortening the measurement time is particularly advantageous for measuring the three-dimensional shape of a moving object (moving body), such as when performing face recognition of a person. Details of the light pattern projected by the projector P onto the target T will be described later.

[0017] The camera C is configured to be able to capture an image of the target T. The camera C may include a solid-state imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera C may also include an optical system that forms an image of the target on the imaging surface of the solid-state imaging element, a signal processing circuit that processes the output of the solid-state imaging element to obtain a luminance value for each pixel, and the like. [2-2: Configuration of the Information Processing Device 2]

[0018] 3 is a block diagram showing the configuration of an information processing device 2 according to the present disclosure. As shown in FIG. 3, the information processing device 2 includes a calculation device 21 and a storage device 22. The information processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not necessarily have to include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0019] The arithmetic device 21 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (e.g., an input device 24 described later) not shown in the drawings that is included in the information processing device 2. The arithmetic device 21 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the information processing device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the information processing device 2 are realized within the arithmetic device 21. In other words, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing operations (in other words, processes) to be performed by the information processing device 2. The arithmetic device 21 may output information to devices (not shown), such as other computers or cloud servers, that are provided outside the information processing device 2, via the communication device 23 (or other communication devices).

[0020] 3 shows an example of logical functional blocks implemented within the arithmetic device 21 to perform information processing operations. As shown in FIG. 3 , the arithmetic device 21 implements an identification unit 211, which is a specific example of a "identification means" described in the appendix below, a classification unit 212, a generation unit 213, which is a specific example of a "generation means" described in the appendix below, an output unit 214, which is a specific example of an "output means" described in the appendix below, a light pattern control unit 215, and a camera control unit 216. However, at least one of the classification unit 212, the generation unit 213, the output unit 214, the light pattern control unit 215, and the camera control unit 216 does not have to be implemented within the arithmetic device 21. Details of the operations of the identification unit 211, the classification unit 212, the generation unit 213, the output unit 214, the light pattern control unit 215, and the camera control unit 216 will be described later with reference to FIG. 4.

[0021] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the information processing device 2 stores long-term. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0022] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The communication device 23 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus).

[0023] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0024] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-3: Information Processing Operation Performed by the Information Processing Device 2]

[0025] The flow of information processing operations performed by the information processing device 2 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of information processing operations performed by the information processing device 2. [2-3-1: Identifying the region of the target T in the image]

[0026] As shown in FIG. 4 , the projector P projects a light pattern onto the target T (step S20). The light pattern control unit 215 controls the projector P to project the light pattern into a predetermined space. The target T exists in the predetermined space. The light pattern projected by the projector P may be any light pattern. The light pattern control unit 215 may, for example, control the projector P to project light of a single brightness value (i.e., light with no shading). The light pattern control unit 215 may also control the projector P to project a light pattern suitable for measuring a three-dimensional shape. Examples of light patterns suitable for measuring a three-dimensional shape include a striped pattern that expresses a black-and-white gradation with a sine wave, a pattern with multiple dots, a grid pattern, a random dot pattern, and a gray code pattern. The light pattern control unit 215 may also control the projector P to project, for example, spot light, slit light, etc.

[0027] The light pattern control unit 215 may perform control to project the same multiple light patterns as the multiple light patterns projected onto the target T in step S24, which will be described later. Alternatively, the light pattern control unit 215 may perform control to project one or more light patterns from the multiple light patterns projected onto the target T in step S24, which will be described later. Alternatively, the light pattern control unit 215 may project a light pattern different from the multiple light patterns projected onto the target T in step S24, which will be described later.

[0028] The camera C captures an image of the target T onto which the light pattern is projected, and generates a target image (step S21). The camera control unit 216 causes the camera C to capture an image of the target T onto which the light pattern is projected by the projector P. The camera control unit 216 acquires the target image captured and generated by the camera C.

[0029] The classification unit 212 classifies the target image into a plurality of partial images (step S22). The classification unit 212 may classify the target image into a plurality of partial images using a trained classification model. The classification model may be a model that identifies to which of a plurality of classes each pixel included in the image belongs. In other words, the classification model may be a model that classifies each pixel into a class. In this case, the classification unit 212 may classify the target image into a plurality of partial images, with regions of pixels classified into the same class being treated as partial images. The plurality of classes may be provided in relation to regions of the head and neck. The plurality of classes may be provided in relation to brightness characteristics within the image.

[0030] Although the classification unit 212 uses a model that classifies each pixel into a class, the classification unit 212 may classify the target image into a plurality of partial images using a mechanism other than a model that classifies each pixel into a class. The classification unit 212 can classify the target image into a plurality of partial images using any method.

[0031] The multiple classes may include at least one of a portion including a person's nose, a portion including a person's neck, a portion including a person's chin, a portion including a person's cheeks, and a portion including a person's ears. In this case, the multiple partial images include at least one of a partial image including a person's nose, a partial image including a person's neck, a partial image including a person's chin, a partial image including a person's cheeks, and a partial image including a person's ears. Furthermore, a person's chin may be further classified based on whether or not it has beard.

[0032] The identification unit 211 identifies a region of the target T (referred to as a "target region") for each of a plurality of partial images included in the target image based on luminance information regarding the luminance of the partial image (step S23). The identification unit 211 identifies the target region based on a comparison between the luminance values ​​of pixels included in the partial image and a threshold. The identification unit 211 may identify pixels included in the partial image whose luminance values ​​exceed the threshold as pixels of the target region.

[0033] The threshold value may be different for each partial image. The threshold value may be predetermined. The threshold value may be set based on a representative value of the luminance values ​​of each pixel included in the partial image. The representative value may be the most frequent value of the luminance values ​​of each pixel. The representative value may be the average value of the luminance values ​​of each pixel.

[0034] When the camera control unit 216 acquires a plurality of target images corresponding to each of the plurality of light patterns, the identification unit 211 identifies the region of the target T based on each of the plurality of target images corresponding to each of the plurality of light patterns. The target images identify the region of the target T within the shooting range of the camera C. [2-3-2: Measurement of the three-dimensional shape of the target T]

[0035] The camera control unit 216 instructs the light pattern control unit 215 to cause the projector P to project a light pattern. The light pattern control unit 215 controls the operation of the projector P so that the projector P projects the light pattern. The camera control unit 216 controls the operation of the camera C and controls the timing of capturing images by the camera C. The camera control unit 216 acquires each of the images of the target T onto which each of the multiple light patterns is projected (referred to as a "pattern projection image") (step S24).

[0036] The light pattern control unit 215 may generate a first light pattern having periodicity. The first light pattern is a short-period light pattern with a relatively short period. The first light pattern may be a sinusoidal grating-like light pattern (hereinafter referred to as a sinusoidal pattern) in which the brightness varies sinusoidally. The information processing device 2 may measure the three-dimensional shape of the target T using a sinusoidal grating phase shift method. The sinusoidal grating phase shift method is a method in which a sinusoidal pattern is projected onto the target T while gradually shifting the phase, and the three-dimensional shape of the target T is measured based on an image of the target T onto which the sinusoidal pattern is projected.

[0037] The light pattern control unit 215 may generate a plurality of sinusoidal wave patterns with different phases. For example, the light pattern control unit 215 may generate four light patterns with phases shifted by a quarter wavelength in the direction in which the sinusoidal wave pattern is repeated.

[0038] The light pattern control unit 215 may generate a second light pattern having a longer periodicity than the first light pattern. The light pattern control unit 215 may generate, as the second light pattern, a light pattern including one pattern in the captured image. The second light pattern may be a luminance gradient pattern in which the luminance value changes linearly at a constant rate. The camera control unit 216 may acquire four pattern projection images onto which a sine wave pattern is projected and one pattern projection image onto which a luminance gradient pattern is projected.

[0039] The generating unit 213 measures the three-dimensional shape of the target area based on each of the plurality of pattern projected images, and generates three-dimensional shape data (step S25).

[0040] The output unit 214 outputs the three-dimensional shape data of the target area (step S26). The output unit 214 may control a display as the output device 25 to display the three-dimensional shape. The output unit 214 may output the three-dimensional shape data to a device (not shown), such as another computer or a cloud server, that is provided outside the information processing device 2. [2-4: Technical Effects of the Information Processing System S2]

[0041] The information processing system S2 according to the present disclosure identifies an object for each of multiple partial images included in a target image and accurately identifies the object, thereby enabling accurate measurement of only the three-dimensional shape of the object. The information processing system S2 identifies an area for each partial image using a different threshold for each partial image, making it possible to identify both objects included in partial images that are brighter than other parts and objects included in partial images that are darker than other parts. For example, by using different thresholds to identify objects included in partial images including a relatively bright cheek portion of the head and neck region and to identify objects included in partial images including a relatively dark neck portion of the head and neck region, the entire head and neck region can be accurately identified.

[0042] In order to distinguish between the head and neck of the subject being photographed and other areas, objects around the subject may be removed, or a black wall or board may be placed behind the subject to prevent the projected pattern light from being reflected on objects other than the subject. Without these limitations on the photographing environment, the pattern light projected on areas other than the subject may cause the calculation of a three-dimensional shape to include areas corresponding to objects other than the subject, resulting in three-dimensional data that includes non-subject objects as noise. However, if the above-mentioned limitations on the photographing environment are imposed, a large space must be provided for installing the device to prevent unnecessary objects from being reflected around the subject, and the installation of a wall or board behind the subject may result in an increase in the size of the device itself. Furthermore, if a wall or board is placed behind the subject, inconvenience may occur due to the subject touching these. The information processing system S2 according to the present disclosure can accurately identify the target area, eliminating the need to remove objects around the subject and eliminating the need to install a board behind the subject. [3: Third Embodiment]

[0043] A third embodiment of an information processing system, an information processing method, and a recording medium will be described below. Hereinafter, the third embodiment of an information processing system, an information processing method, and a recording medium will be described using the information processing system according to the present disclosure. The information processing system according to the present disclosure may include a projector P, a camera C, and an information processing device 3. [3-1: Configuration of the information processing device 3]

[0044] The configuration of the information processing device 3 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 3.

[0045] As shown in FIG. 5, the information processing device 3 includes a calculation device 21 and a storage device 22, similar to the information processing device 2. Furthermore, the information processing device 3 may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2. However, the information processing device 3 does not have to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 3 differs from the information processing device 2 in that a recognition unit 317 is further implemented in the calculation device 21. Other features of the information processing device 3 may be the same as other features of the information processing device 2. Therefore, hereinafter, differences from the embodiments already described will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [3-2: Information Processing Operation Performed by Information Processing Device 3]

[0046] The flow of the information processing operation performed by the information processing device 4 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of the information processing operation performed by the information processing device 3.

[0047] 6, the projector P projects a light pattern onto the target T (step S20). The camera C captures the target T onto which the light pattern is projected, and generates a target image (step S21).

[0048] The recognition unit 317 recognizes the range of the target T from the target image (step S30). The recognition unit 317 may detect an area likely to be the head and neck and recognize it as the target T. The recognition unit 317 may also recognize an area where a strong light pattern is projected as the target T. The recognition unit 317 may recognize the range of the target T by determining the brightness value for each pixel. In this case, for example, a threshold for threshold determination may be set higher for the upper portion of the target image (e.g., two-thirds of the vertical area from the top of the target image) than for the lower portion of the target image (e.g., one-third of the vertical area from the bottom of the target image). Alternatively, a threshold for threshold determination may be set higher for the middle portion of the target image (e.g., one-third of the vertical area from the center of the target image and one-third of the vertical area from the bottom of the target image) than for the edge portions of the target image (e.g., one-sixth of the vertical area from the top of the target image and one-sixth of the vertical area from the bottom of the target image). The classification unit 312 classifies the range of the target T into multiple partial images (step S31).

[0049] The identification unit 211 identifies a target region for each of a plurality of partial images included in the target image based on brightness information related to the brightness of the partial image (step S23). The camera control unit 616 acquires each of the pattern projection images onto which each of the plurality of light patterns is projected (step S24). The generation unit 613 generates three-dimensional shape data of the target region based on each of the plurality of pattern projection images (step S25). The output unit 214 outputs the three-dimensional shape data of the target region (step S26). [3-3: Technical Effects of the Information Processing System]

[0050] The operation of recognizing the range of the target T can be performed with a smaller amount of calculation compared to the operation of classifying partial images. Therefore, when the range of the target T is recognized and the range of the target T is classified into a plurality of partial images, the processing load on the information processing device 3 can be reduced compared to when the target image is directly classified into a plurality of partial images. [4: Fourth Embodiment]

[0051] An information processing system, an information processing method, and a recording medium according to a fourth embodiment of the present disclosure will be described below. [4-1: Configuration of Information Processing System S4]

[0052] Fig. 7 is a schematic diagram showing the configuration of an information processing system S4. As shown in Fig. 7, the information processing system S4 according to the present disclosure includes a projection photography apparatus 100 and an information processing apparatus 4. Fig. 7 illustrates an example in which the object T whose three-dimensional shape is to be measured is the head and neck of a person, mainly including the face of the person.

[0053] The projection and photography apparatus 100 projects and photographs a target T. The projection and photography apparatus 100 includes a first projector P1 as a projection means, a second projector P2 as another projection means, a first camera C1, a second camera C2, and a third camera C3.

[0054] The first projector P1 projects a light pattern onto the target T from a first side of the target T. Fig. 5 illustrates a case where the first side is the left side of the target T. The first camera C1 photographs the target T onto which the light pattern is projected by the first projector P1 from the first side.

[0055] The second projector P2 projects a light pattern onto the target T from the second side. Fig. 5 illustrates a case where the second side is the right side of the target T. That is, the second projector P2 and the first projector P1 project light patterns into a predetermined space from different directions. In other words, the projection shooting device 100 includes the second projector P2 that projects a light pattern onto the target T from a direction different from that of the first projector P1.

[0056] The second camera C2 photographs the target T onto which the light pattern is projected by the second projector P2 from the second side. The second camera C2 may also photograph the target onto which the light patterns are projected by the second projector P2 and the first projector P1. The first camera C1 may also photograph the target onto which the light patterns are projected by the first projector P1 and the second projector P2. The third camera C3 photographs the target T from the front of the target T.

[0057] The first projector P1 and the second projector P2 may have the same mechanism as the above-described projector P. The first camera C1, the second camera C2, and the third camera C3 may have the same mechanism as the above-described camera C. [4-2: Configuration of the information processing device 4]

[0058] The configuration of the information processing device 4 will be described with reference to FIG. 8. FIG. 8 is a block diagram showing the configuration of the information processing device 4. As shown in FIG. 8, the information processing device 4 may have the same configuration as the information processing device 2. [4-3: Information Processing Operation Performed by the Information Processing Device 4]

[0059] The flow of the information processing operation performed by the information processing device 4 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of the information processing operation performed by the information processing device 4. [4-3-1: Identifying the region of the target T]

[0060] 9 , the light pattern control unit 415 controls the first projector P1 and the second projector P2 to project light patterns into a predetermined space. The first projector P1 and the second projector P2 project the light patterns onto the target T (step S40). The light pattern projected by the first projector P1 and the light pattern projected by the second projector P2 may be the same light pattern. Alternatively, the light pattern projected by the first projector P1 and the light pattern projected by the second projector P2 may be different light patterns.

[0061] The camera control unit 416 acquires target images obtained by capturing images of the target T onto which the light patterns are projected by the first projector P1 and the second projector P2 (step S41). The first camera C1 and the second camera C2 capture images of the target T onto which the light patterns are projected by the first projector P1 and the second projector P2, respectively. That is, the camera control unit 416 acquires a first-side target image and a second-side target image.

[0062] The classification unit 412 classifies the target image into a plurality of partial images (step S42). The classification unit 412 classifies each of the target image on the first side and the target image on the second side into a plurality of partial images.

[0063] The identification unit 411 identifies a target region for each of a plurality of partial images included in the target image based on luminance information related to the luminance of the partial image (step S43). The identification unit 411 identifies a first-side target region and a second-side target region. [4-3-2: Measurement of the three-dimensional shape of the first side of the target T]

[0064] The camera control unit 416 instructs the light pattern control unit 415 to cause the projector P1 to project a light pattern. The light pattern control unit 415 controls the projector P1 to project a light pattern. The camera control unit 416 controls the operation of the camera C1 and controls the timing of capturing images by the camera C1. The camera control unit 416 acquires each of the first-side pattern projection images onto which each of the multiple light patterns is projected from the first side (step S44). The camera control unit 416 may acquire four pattern projection images onto which a sine wave pattern is projected from the first side, and a pattern projection image onto which a luminance gradient pattern is projected from the first side. The generation unit 413 measures the three-dimensional shape of the target area on the first side based on each of the multiple pattern projection images on the first side, and generates three-dimensional shape data for the first side (step S45). [4-3-3: Measurement of the Three-Dimensional Shape of the Second Side of the Target T]

[0065] The camera control unit 416 instructs the light pattern control unit 415 to cause the projector P2 to project a light pattern. The light pattern control unit 415 controls the projector P2 to project a light pattern. The camera control unit 416 controls the operation of the camera C2 and controls the timing of capturing images by the camera C2. The camera control unit 416 acquires each of the second-side pattern projection images onto which each of the multiple light patterns is projected from the second side (step S46). The camera control unit 416 may acquire four pattern projection images onto which a sinusoidal wave pattern is projected from the second side, and a pattern projection image onto which a luminance gradient pattern is projected from the second side. The generation unit 413 measures the three-dimensional shape of the target area on the second side based on each of the multiple second-side pattern projection images, and generates three-dimensional shape data for the second side (step S47).

[0066] The generation unit 413 combines the three-dimensional shape data on the first side and the three-dimensional shape data on the second side to generate one piece of facial shape data (step S48). The output unit 414 outputs the one piece of facial shape data (step S26). The output unit 414 may control the display serving as the output device 25 to display the one facial shape. The output unit 414 may output the one piece of facial shape data to a device (not shown), such as another computer or cloud server, provided outside the information processing device 4.

[0067] In at least one of the first to third embodiments, the information processing system may include a projection photography device 100 instead of the projector P and the camera C. In at least one of the second and third embodiments, when the information processing system includes the projection photography device 100 instead of the projector P and the camera C, the generation unit may generate facial shape data by combining the three-dimensional shape of the first side and the three-dimensional shape of the second side. [4-4: Technical Effects of Information Processing System S4]

[0068] The information processing system S4 according to the present disclosure can acquire an accurate three-dimensional shape including the three-dimensional shape of the left side of the target T, excluding the background (walls, objects, etc.) other than the target T, and the three-dimensional shape of the right side of the target T. By using this three-dimensional shape, two-dimensional images of the target T viewed from various angles can be acquired. [5: Fifth Embodiment]

[0069] An information processing system, an information processing method, and a recording medium according to a fifth embodiment will be described. Hereinafter, an information processing system, an information processing method, and a recording medium according to the fifth embodiment will be described using an information processing system S5 according to the present disclosure. [5-1: Configuration of the information processing system S5]

[0070] Fig. 10 is a schematic diagram showing the configuration of an information processing system S5. As shown in Fig. 10, the information processing system S5 according to the present disclosure further includes a first illuminator L1 and a second illuminator L2. The information processing system S5 may include the first illuminator L1 and the second illuminator L2 in addition to the projector P, the camera C, and the information processing device 5. Alternatively, the information processing system S5 may include the first illuminator L1 and the second illuminator L2 in addition to the projection shooting device 100 and the information processing device 5.

[0071] As illustrated in Fig. 10(a), the first illuminator L1 emits light from a first side of the target T. Fig. 10(a) illustrates a case where the first illuminator L1 emits light onto the target T from the left side as viewed from the target T. As illustrated in Fig. 10(a), the first illuminator L1 may be provided so that light does not reach the side of the target T opposite to the side facing the camera C. The first illuminator L1 may emit a flash from the first side of the target T.

[0072] 10(b), the second illuminator L2 emits light from the second side of the target T. FIG. 10(b) illustrates a case where the second illuminator L2 emits light onto the target T from the right side as viewed from the target T. As illustrated in FIG. 10(b), the second illuminator L2 may be provided so that light does not reach the side of the target T opposite to the side facing the camera C. The second illuminator L2 may emit a flash from the second side of the target T.

[0073] In the fifth embodiment, the camera C may generate an image of the target T captured from the front as viewed from the target T, similar to the third camera C3. [5-2: Configuration of the information processing device 5]

[0074] The configuration of the information processing device 5 will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the configuration of the information processing device 5.

[0075] As shown in FIG. 11 , the information processing device 5 includes a calculation device 21 and a storage device 22, similar to the information processing devices 2 to 4. Furthermore, the information processing device 5 may include a communication device 23, an input device 24, and an output device 25, similar to the information processing devices 2 to 4. However, the information processing device 5 does not need to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 5 differs from the information processing devices 2 to 4 in that a lighting control unit 518 is further implemented within the calculation device 21. Other features of the information processing device 5 may be the same as other features of at least one of the information processing devices 2, 3, and 4. Therefore, hereinafter, differences from the already-described embodiments will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [5-3: Information Processing Operation Performed by the Information Processing Device 5]

[0076] The information processing operation described in the fifth embodiment may be performed in addition to or instead of at least one of the information processing operations described in the second to fourth embodiments. When the information processing system S5 includes a projector P, a camera C, and an information processing device 5, the information processing operation described in the fifth embodiment may be performed in addition to or instead of the information processing operation described in the second or third embodiment. Furthermore, when the information processing system S5 includes a projection shooting apparatus 100 and an information processing device 5, the information processing operation described in the fifth embodiment may be performed in addition to or instead of the information processing operation described in the fourth embodiment. By performing the information processing operation described in the fifth embodiment, it is possible to more accurately separate the area of ​​the target T from the area outside the target T in an image.

[0077] The flow of information processing operations performed by the information processing device 5 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the information processing operations performed by the information processing device 5.

[0078] 12 , the camera control unit 516 acquires a first target image obtained by capturing an image of the target T illuminated with light from a first side (step S50). The lighting control unit 518 controls the first lighting unit L1 to illuminate a predetermined space. The lighting control unit 518 controls the first lighting unit L1 to illuminate the light in synchronization with the image capture by the camera C. The camera control unit 516 causes the camera C to capture an image of the target T illuminated with light from the first lighting unit L1.

[0079] The camera control unit 516 acquires a second target image obtained by capturing an image of the target T illuminated with light from the second side (step S51). The lighting control unit 518 controls the second lighting unit L2 to illuminate a predetermined space with light. The lighting control unit 518 controls the second lighting unit L2 to illuminate the light in synchronization with the image capture by the camera C. The camera control unit 516 causes the camera C to capture an image of the target T illuminated with light from the second lighting unit L2.

[0080] The recognition unit 517 recognizes the range of the target T from the target image (step S52). The recognition unit 517 may recognize the range of the target T based on at least one of a region of brightness equal to or greater than a predetermined value and a region of brightness less than a predetermined value included in the first target image, and at least one of a region of brightness equal to or greater than a predetermined value and a region of brightness less than a predetermined value included in the second target image.

[0081] The recognition unit 517 may generate one luminance image from the first target image and the second target image. The recognition unit 517 may generate a luminance image LI as exemplified in FIG. 13. The recognition unit 517 may recognize the range of the target T based on at least one of an area of ​​luminance equal to or greater than a predetermined value and an area of ​​luminance less than a predetermined value included in the luminance image. The classification unit 512 classifies the range of the target T into a plurality of partial images (step S53).

[0082] The identification unit 211 identifies a target region for each of a plurality of partial images included in the target image based on brightness information related to the brightness of the partial image (step S23). The camera control unit 216 acquires each of the pattern projection images onto which each of the plurality of light patterns is projected (step S24). The generation unit 213 measures the three-dimensional shape of the target region based on each of the plurality of pattern projection images and generates three-dimensional shape data (step S25). The output unit 214 outputs the three-dimensional shape data of the target region (step S26). [5-4: Technical Effects of Information Processing System S5]

[0083] The information processing system S5 according to the present disclosure can accurately identify the region of the object T in the image based on an image generated by capturing the object T illuminated with light from the first side and an image generated by capturing the object T illuminated with light from the second side. [6: Sixth Embodiment]

[0084] A sixth embodiment of an information processing system, an information processing method, and a recording medium will be described below. Hereinafter, the sixth embodiment of an information processing system, an information processing method, and a recording medium will be described using the information processing system according to the present disclosure. The information processing system according to the present disclosure may include a projector P, a camera C, and an information processing device 6. Alternatively, the information processing system according to the present disclosure may include a projection imaging device 100 and an information processing device 6. [6-1: Configuration of the information processing device 6]

[0085] As shown in Fig. 14, the information processing device 6 may have the same configuration as at least one of the information processing device 3 and the information processing device 5. The information processing device 6 differs from at least one of the information processing device 3 and the information processing device 5 in the recognition operation by the recognition unit 617. Other features of the information processing device 6 may be the same as other features of at least one of the information processing device 3 and the information processing device 5. Therefore, hereinafter, differences from the already described embodiments will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [6-2: Information Processing Operation Performed by the Information Processing Device 6]

[0086] The information processing operation described in the sixth embodiment may be performed in addition to or instead of at least one of the information processing operation described in the third embodiment and the information processing operation described in the fifth embodiment. By performing the information processing operation described in the sixth embodiment, the range of the target T can be recognized more accurately from the target image.

[0087] 15 and 16, the flow of information processing operations performed by the information processing device 6 will be described. Fig. 15 is a flowchart showing the flow of information processing operations performed by the information processing device 6. Fig. 16 is a conceptual diagram of the information processing operations performed by the information processing device 6.

[0088] As shown in Fig. 15, the camera control unit 616 acquires a plurality of pattern projection images obtained by capturing an image of a predetermined space corresponding to each of the plurality of light patterns (step S60). The light pattern control unit 615 controls the projector P (or the first projector P1 and the second projector P2, hereinafter simply referred to as "projector P") to project each of a plurality of different light patterns into the predetermined space. The camera control unit 616 controls the camera C (or the third camera C3, hereinafter simply referred to as "camera C") to capture an image of the predetermined space onto which each of the plurality of light patterns is projected and generate a pattern projection image of the predetermined space. Fig. 16(a) illustrates a case in which the projector P projects a light pattern into a predetermined space where the target T is not present, and the camera C captures the predetermined space where the target T is not present.

[0089] The generation unit 613 generates three-dimensional information of the predetermined space based on a plurality of pattern projection images of the predetermined space corresponding to each of the plurality of light patterns (step S61). That is, the generation unit 613 measures three-dimensional information of the predetermined space in a state where the target T is absent. Since the three-dimensional information of the predetermined space in a state where the target T is absent does not depend on the target T, it may be measured in advance on an occasion other than the occasion for measuring the three-dimensional shape of the target T. Figure 16(b) illustrates an example image showing three-dimensional information 3DO measured based on pattern projection images captured of the predetermined space in which the target T is absent. Note that in Figures 16(a) and (b), the target T is absent, so the target T is represented by a half-dotted line.

[0090] The camera control unit 616 acquires a plurality of pattern projection images obtained by capturing an image of the target T present in a predetermined space corresponding to each of the plurality of light patterns (step S62). The light pattern control unit 615 controls the projector P to project each of the plurality of different light patterns onto the target T. The camera control unit 616 controls the camera C to capture an image of the target T onto which each of the plurality of light patterns has been projected, and to generate a pattern projection image of the target T present in the predetermined space. Fig. 16(c) illustrates a case where the projector P projects a light pattern onto the target T present in the predetermined space, and the camera C captures an image of the target T present in the predetermined space.

[0091] The generation unit 613 generates three-dimensional information of the predetermined space including the target T based on a plurality of pattern projection images of the predetermined space including the target T corresponding to each of the plurality of light patterns (step S63). Fig. 16(d) illustrates an example of an image showing three-dimensional information 3DI measured based on the pattern projection images obtained by capturing the predetermined space including the target T.

[0092] The recognition unit 617 recognizes the range of the target T based on the three-dimensional information of the predetermined space and the three-dimensional information of the predetermined space including the target T (step S64). The recognition unit 617 may calculate difference information between the three-dimensional information 3DO of the predetermined space and the three-dimensional information 3DI of the predetermined space including the target T, and recognize the range of the target T. Fig. 16(e) illustrates an example of the range of the target T recognized based on the three-dimensional information 3DO illustrated in Fig. 16(b) and the three-dimensional information 3DI illustrated in Fig. 16(d).

[0093] The projector P projects a light pattern onto the target T (step S20). The camera C captures the target T onto which the light pattern is projected to generate a target image (step S21). The classification unit 612 classifies the range of the target T recognized by the recognition unit 617 in the target image into multiple partial images (step S65).

[0094] The identification unit 211 identifies a target region for each of a plurality of partial images included in the target image based on brightness information related to the brightness of the partial image (step S23). The camera control unit 616 acquires each of the pattern projection images onto which each of the plurality of light patterns is projected (step S24). The generation unit 613 generates three-dimensional shape data of the target region based on each of the plurality of pattern projection images (step S25). The output unit 214 outputs the three-dimensional shape data of the target region (step S26). [6-3: Technical Effects of the Information Processing System]

[0095] The information processing system according to the present disclosure measures three-dimensional information of a predetermined space and three-dimensional information of a predetermined space including the target T, and therefore can accurately recognize the range of the target T in the target image.

[0096] The following supplementary notes are further disclosed with respect to the above-described embodiment. [Supplementary Note 1] An information processing system comprising: a projection unit that projects a light pattern onto a target; an imaging unit that captures an image of the target onto which the light pattern is projected to generate a target image; an identification unit that identifies a region of the target for each of a plurality of partial images included in the target image based on luminance information regarding the luminance of the partial image; and a measurement unit that measures a three-dimensional shape of the target based on the region of the target. [Supplementary Note 2] The information processing system according to claim 1, further comprising: a projection unit that projects a plurality of light patterns onto the target; a imaging unit that captures an image of the target onto which each of the plurality of light patterns is projected to generate a plurality of target images corresponding to each of the plurality of light patterns; a generation unit that measures the three-dimensional shape of the region of the target based on each of the plurality of target images and generates three-dimensional shape data; and an output unit that outputs the three-dimensional shape data. [Supplementary Note 3] The information processing system according to claim 1 or 2, wherein the identification means identifies the region of the target based on brightness values ​​of pixels included in the partial images and a threshold, and the threshold differs for each partial image. [Supplementary Note 4] The information processing system according to claim 1 or 2, wherein the target is a human head and neck region, primarily consisting of a face, and the plurality of partial images include at least one of a partial image including a nose portion of the target, a partial image including a neck portion of the target, a partial image including a chin portion of the target, a partial image including a cheek portion of the target, and a partial image including an ear portion of the target. [Supplementary Note 5] The information processing system according to claim 2, further comprising: a recognition means for recognizing the range of the target from the target image; and a classification means for classifying the range of the target into the plurality of partial images. [Supplementary Note 6] The information processing system according to claim 1 or 2, further comprising: another projection means for projecting a light pattern onto the target from a direction different from that of the projection means; and the imaging means for generating the target image by imaging the target onto which the light patterns are projected by the projection means and the other projection means.[Supplementary Note 7] The information processing system according to claim 5, wherein the recognition means recognizes the range of the object based on at least one of an area of ​​brightness equal to or greater than a predetermined value and an area of ​​brightness less than a predetermined value contained in an image generated by photographing the object illuminated with light from a first side, and at least one of an area of ​​brightness equal to or greater than a predetermined value and an area of ​​brightness less than a predetermined value contained in an image generated by photographing the object illuminated with light from a second side different from the first side. [Supplementary Note 8] The information processing system according to claim 5, wherein the imaging means generates an image of the predetermined space by imaging the predetermined space onto which the light pattern is projected by the projection means, and the target image by imaging the target present in the predetermined space onto which the light pattern is projected by the projection means, the generation means generates three-dimensional information of the predetermined space based on a plurality of images of the predetermined space corresponding to each of the plurality of light patterns, and three-dimensional information of the predetermined space including the target based on a plurality of target images corresponding to each of the plurality of light patterns, and the recognition means recognizes an area of ​​the target based on the three-dimensional information of the predetermined space and the three-dimensional information of the predetermined space including the target. [Supplementary Note 9] An information processing method comprising: projecting a light pattern onto a target; capturing an image of the target onto which the light pattern is projected to generate a target image; identifying a region of the target for each of a plurality of partial images included in the target image based on luminance information regarding the luminance of the partial images; and measuring the three-dimensional shape of the target. [Supplementary Note 10] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method of: projecting a light pattern onto an object; capturing an image of the object onto which the light pattern is projected to generate an object image; identifying an area of ​​the object for each of a plurality of partial images included in the object image based on luminance information relating to the luminance of the partial image; and measuring a three-dimensional shape of the object.

[0097] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0098] S1, S2, S4, S5 Information processing system 101 Projection unit 102 Photography unit 11, 211, 411 Identification unit 13 Measurement unit 2, 3, 4, 5, 6 Information processing device P Projector C Camera 212, 312, 412, 512, 612 Classification unit 213, 413, 613 Generation unit 214 Output unit 215, 315, 415, 515, 615 Light pattern control unit 216, 316, 416, 516, 616 Camera control unit 317, 517, 617 Recognition unit 100 Projection photography device P1 First projector C1 First camera P2 Second projector C2 Second camera C3 Third camera 518 Lighting control unit L1 First lighting L2 Second lighting

Claims

1. An information processing system comprising: a projection means for projecting a light pattern onto an object; an imaging means for imaging the object onto which the light pattern is projected to generate an object image; an identification means for identifying an area of ​​the object based on luminance information regarding the luminance of each of a plurality of partial images included in the object image; and a measurement means for measuring a three-dimensional shape of the object based on the area of ​​the object.

2. The information processing system of claim 1, further comprising: a projection means for projecting a plurality of light patterns onto the target; a photographing means for photographing the target onto which each of the plurality of light patterns is projected, and generating a plurality of target images corresponding to each of the plurality of light patterns; a generation means for measuring a three-dimensional shape of an area of ​​the target based on each of the plurality of target images, and generating three-dimensional shape data; and an output means for outputting the three-dimensional shape data.

3. An information processing system according to claim 1 or 2, wherein the identification means identifies the target area based on the brightness values ​​of pixels contained in the partial image and a threshold value, the threshold value being different for each of the partial images.

4. An information processing system as described in claim 1 or 2, wherein the object is a person's head and neck, primarily consisting of the face, and the plurality of partial images include at least one of a partial image including a nose portion of the object, a partial image including a neck portion of the object, a partial image including a chin portion of the object, a partial image including a cheek portion of the object, and a partial image including an ear portion of the object.

5. An information processing system according to claim 2, further comprising: a recognition means for recognizing an area of ​​the object from the object image; and a classification means for classifying the area of ​​the object into the plurality of partial images.

6. An information processing system as described in claim 1 or 2, further comprising another projection means for projecting a light pattern onto the target from a direction different from that of the projection means, and wherein the imaging means images the target onto which the light pattern is projected by the projection means and the other projection means to generate the target image.

7. The information processing system of claim 5, wherein the recognition means recognizes the range of the object based on at least one of an area of ​​brightness equal to or greater than a predetermined value and an area of ​​brightness less than a predetermined value contained in an image generated by photographing the object illuminated with light from a first side, and at least one of an area of ​​brightness equal to or greater than a predetermined value and an area of ​​brightness less than a predetermined value contained in an image generated by photographing the object illuminated with light from a second side different from the first side.

8. The information processing system described in claim 5, wherein the photographing means generates an image of the specified space photographed of the specified space onto which the light pattern is projected by the projection means, and an image of the target photographed of the target present in the specified space onto which the light pattern is projected by the projection means; the generation means generates three-dimensional information of the specified space based on a plurality of images of the specified space corresponding to each of the plurality of light patterns, and three-dimensional information of the specified space including the target based on a plurality of images of the specified space corresponding to each of the plurality of light patterns; and the recognition means recognizes the range of the target based on the three-dimensional information of the specified space and the three-dimensional information of the specified space including the target.

9. An information processing method comprising: projecting a light pattern onto an object; photographing the object onto which the light pattern is projected to generate an image of the object; identifying an area of ​​the object for each of a plurality of partial images included in the object image based on luminance information regarding the luminance of the partial image; and measuring the three-dimensional shape of the object.

10. A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method of projecting a light pattern onto an object, photographing the object onto which the light pattern is projected to generate an image of the object, identifying an area of ​​the object for each of a plurality of partial images included in the object image based on luminance information relating to the luminance of the partial images, and measuring the three-dimensional shape of the object.

Images