Identification method, projection method, identification system, information processing device, and program

Abstract

To accurately identify an area corresponding to an object included in a subject even when the color of the object and the color of a background object other than the object have a small difference therebetween.SOLUTION: There is provided an information processing apparatus 30 having a first acquisition unit 331, a second acquisition unit 332, a calculation unit 333, and an identification unit 334. The first acquisition unit 331 acquires, on the basis of a first image obtained by picking up an image of a subject SC including an object SC1 with an imaging apparatus 20, first information on reflected light from the subject SC for every pixel. The second acquisition unit 332 acquires second information according to the distance from the subject SC to the imaging apparatus 20 for every pixel of the first image. The calculation unit 333 executes an operation with the first information and the second information as variables, and calculates an index value being a result of the operation for every pixel. The identification unit 334 determines, for every pixel of the first image, whether the pixel belongs to an area corresponding to the object SC1 on the basis of the index value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an identification method, a projection method, an identification system, an information processing device, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for detecting an area corresponding to an object from a captured image of a subject including the object. In the technique disclosed in Patent Document 1, an area corresponding to an object and an area corresponding to a background are roughly detected using a depth map indicating a distribution of depth values, which are distances between an imaging device and each part of the subject. In the technique disclosed in Patent Document 1, for an area for which a depth value could not be obtained due to the influence of noise, occlusion, etc., it is determined whether the area corresponds to an object or to the background based on color. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-091298 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, since depth values ​​are not used in color-based judgment, there is a risk of misjudgment if the difference between the color of the object in the subject and the color of the background is small. Similarly, there is a risk of misjudgment if the color of a specific wavelength range cannot be captured with high accuracy due to the characteristics or individual differences of the imaging device. [Means for solving the problem]

[0005] In order to solve the above problem, the identification method disclosed herein acquires, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image, acquires second information corresponding to the distance from the first subject to the first camera for each first pixel, performs a calculation using the first information and the second information as variables, calculates an index value resulting from the calculation for each first pixel, and identifies the first region by determining, based on the index value, whether or not each first pixel belongs to a first region of the first image corresponding to the object.

[0006] In addition, in order to solve the above problem, the projection method disclosed herein acquires, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image, acquires second information corresponding to a distance from the first subject to the first camera for each of the first pixels, performs a calculation using the first information and the second information as variables, calculates an index value resulting from the calculation for each of the first pixels, and determines, based on the index value, whether or not each of the first pixels belongs to a first region of the first image corresponding to the object, thereby identifying the first region, divides a projection image to be projected from a projector onto the first subject into a second region corresponding to the first region and a third region other than the second region, fills the second region with a first color, and has the projector project the projection image in which the third region is filled with a second color.

[0007] In order to solve the above problems, the specific system of the present disclosure includes a first camera and an information processing device. identificationIn the system, the information processing device performs the following operations based on a first image obtained by capturing an image of a first subject including an object with the first camera: acquire first information regarding reflected light from the first subject for each first pixel constituting the first image; acquire second information corresponding to the distance from the first subject to the first camera for each first pixel; perform an operation using the first information and the second information as variables and calculate an index value for each first pixel that is a result of the operation; and identify the first region by determining, based on the index value, whether or not each first pixel belongs to a first region of the first image corresponding to the object.

[0008] In addition, in order to solve the above problem, the information processing device of the present disclosure performs the following operations: based on a first image obtained by capturing an image of a first subject including an object with a first camera, acquire first information regarding reflected light from the first subject for each first pixel constituting the first image; acquire second information corresponding to the distance from the first subject to the first camera for each first pixel; perform an operation using the first information and the second information as variables, calculate an index value for each first pixel that is a result of the operation; and identify the first region by determining, based on the index value, for each first pixel, whether or not the first pixel belongs to a first region of the first image corresponding to the object.

[0009] In addition, in order to solve the above problem, the program disclosed herein causes a computer to perform the following operations: based on a first image obtained by capturing an image of a first subject including an object with a first camera, acquire first information regarding reflected light from the first subject for each first pixel constituting the first image; acquire second information corresponding to the distance from the first subject to the first camera for each first pixel; perform an operation using the first information and the second information as variables, calculate an index value for each first pixel that is a result of the operation; and identify the first region by determining, based on the index value, whether or not each first pixel belongs to a first region of the first image corresponding to the object. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example configuration of a projection system 1 including an information processing device 30 that executes a projection method according to an embodiment of the present disclosure. [Diagram 2] FIG. 13 is a diagram showing an example of a user setting of pixels representing a foreground region and pixels representing a background region. [Diagram 3] 13 is a flowchart showing the flow of a projection method executed by a processing device 330 of the information processing device 30 in accordance with a program 321. [Figure 4] FIG. 11 is a diagram for explaining the effect of the present embodiment. [Diagram 5] 11A and 11B are diagrams for explaining problems that occur when a foreground region is identified based only on reflected light information. [Figure 6] 11A and 11B are diagrams for explaining problems that occur when a foreground region is identified based only on distance information; [Figure 7] FIG. 11 is a diagram showing an example of a histogram indicated by reflected light information. [Figure 8] FIG. 11 is a diagram showing an example of a histogram indicated by reflected light information. [Figure 9] FIG. 11 is a diagram showing an example of a histogram indicated by reflected light information. [Figure 10] FIG. 11 is a diagram illustrating an example of a histogram indicated by distance information. [Figure 11] FIG. 11 is a diagram illustrating an example of a histogram indicated by distance information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Various technically preferable limitations are attached to the embodiments described below. However, the embodiments of the present disclosure are not limited to the forms described below.

[0012] 1. Embodiment FIG. 1 is a schematic diagram showing an example of a projection system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the projection system 1 includes a projector 10, an imaging device 20, and an information processing device 30. The projection system 1 shown in FIG. 1 is a system for projecting an image onto an object SC1 having a three-dimensional shape. In the projection system 1, an image of a subject SC including the object SC1 and a background object SC2 is captured by the imaging device 20. The subject SC is an example of a first subject in the present disclosure. The imaging device 20 is an example of a first camera in the present disclosure. The image of the subject SC captured by the imaging device 20 is an example of a first image in the present disclosure.

[0013] In the projection system 1, an area corresponding to an object SC1 in an image of the subject SC captured by the imaging device 20 is identified by the information processing device 30. Then, the information processing device 30 controls the projector 10 to project onto the subject SC a projection image in which the area corresponding to the object SC1 is filled with a first color and the area other than the area corresponding to the object SC1, i.e., the area corresponding to a background object SC2, is filled with a second color different from the first color.

[0014] The projector 10 includes a liquid crystal light valve, a projection lens, and a liquid crystal driver for projecting a projection image. The projector 10 also includes an extra-high pressure mercury lamp or a metal halide lamp as a light source. The projector 10 is communicably connected to the information processing device 30, for example, by a cable. The projector 10 acquires image data representing an image from the information processing device 30 by communication via the cable. The projector 10 projects an image indicated by the acquired image data onto a subject SC. In this embodiment, the communication between the projector 10 and the information processing device 30 is wired communication conforming to a standard such as Ethernet or USB (Universal Serial Bus). However, the communication between the projector 10 and the information processing device 30 may be wireless communication conforming to a standard such as Wi-Fi. Note that WI-FI and Ethernet are registered trademarks.

[0015] The imaging device 20 is, for example, a camera equipped with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) that is an imaging element that converts collected light into an electric signal. In the following, for the sake of simplicity of explanation, it is assumed that the imaging device 20 captures a still image. Note that the imaging device 20 may capture a moving image instead of capturing a still image. The imaging device 20 captures an image of a subject SC. The imaging device 20 is connected to the information processing device 30 by, for example, a cable in a manner similar to the projector 10 so as to be able to communicate with the information processing device 30. The imaging device 20 transmits image data representing the captured image to the information processing device 30. In this embodiment, the communication between the imaging device 20 and the information processing device 30 is, for example, wired communication conforming to a standard such as Ethernet or USB, but may be wireless communication conforming to a standard such as Wi-Fi.

[0016] The information processing device 30 is, for example, a personal computer. As shown in Fig. 1, the information processing device 30 has a communication device 300, a touch panel 310, a storage device 320, and a processing device 330. The communication device 300 is connected to the projector 10 via a cable. The communication device 300 is also connected to the imaging device 20 via a cable. The communication device 300 receives image data transmitted from the imaging device 20. The communication device 300 also transmits image data representing an image to be projected on a subject SC to the projector 10 under the control of the processing device 330.

[0017] The touch panel 310 is a device that integrates a display device that displays images and an input device into which a user inputs information. The input device is, for example, a transparent sheet-like contact sensor. The input device is provided so as to cover the display surface of the display device. The input device detects a touch position using electrostatic capacitance determined by an object that touches the input device and the input device. The input device outputs data indicating the detected touch position to the processing device 330. As a result, the content of a user's operation on the touch panel 310 is transmitted to the processing device 330.

[0018] The storage device 320 is a recording medium readable by the processing device 330. The storage device 320 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), or an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM).

[0019] A program 321 to be executed by the processing device 330 is stored in advance in the non-volatile memory of the storage device 320. The volatile memory of the storage device 320 is used by the processing device 330 as a work area when executing the program 321. The program 321 may also be called an "application program," "application software," or an "app." The program 321 is obtained, for example, from a server or the like (not shown) via the communication device 300, and then stored in the storage device 320.

[0020] The processing device 330 includes a processor such as a CPU (Central Processing Unit), that is, a computer. The processing device 330 may be configured with a single computer or multiple computers. The processing device 330 reads the program 321 from a non-volatile memory to a volatile memory when an operation to instruct the start of execution of the program 321 is performed on an input device (not shown). The processing device 330 executes the program 321 read to the volatile memory. The processing device 330 operating according to the program 321 functions as a first acquisition unit 331, a second acquisition unit 332, a calculation unit 333, a determination unit 334, and a projection control unit 335 shown in FIG. 1. The first acquisition unit 331, the second acquisition unit 332, the calculation unit 333, the determination unit 334, and the projection control unit 335 shown in FIG. 1 are software modules realized by operating the processing device 330 according to the program 321. The functions of the first acquisition unit 331, the second acquisition unit 332, the calculation unit 333, the identification unit 334, and the projection control unit 335 are as follows.

[0021] The first acquisition unit 331 acquires reflected light information regarding reflected light from the subject SC based on the captured image of the subject SC by the imaging device 20, i.e., the first image, for each pixel constituting the first image. The pixels constituting the first image are an example of the first pixel in the present disclosure. The reflected light information in this embodiment is luminance information indicating the luminance of reflected light from the subject SC. In this embodiment, the captured image by the imaging device 20 is a luminance image in which pixel values ​​indicate the luminance of reflected light from the subject. Therefore, in this embodiment, the image data output from the imaging device 20 becomes the reflected light information. Note that the reflected light information may be gradation information indicating the gradation of each color of R, G, and B in the reflected light from the subject. In addition, the reflected light information may be information indicating both the luminance and gradation of the reflected light. The reflected light information is an example of the first information in the present disclosure.

[0022] The second acquisition unit 332 acquires distance information corresponding to the distance from each part of the subject SC to the imaging device 20 for each first pixel. In this embodiment, the second acquisition unit 332 causes the projector 10 to project a pattern image for acquiring distance information onto the subject SC. Specific examples of the pattern image projected onto the subject SC from the projector 10 include an image of a gray code pattern, an image of a binary code pattern, and an image of a phase shift pattern. Next, the second acquisition unit 332 causes the imaging device 20 to capture the subject SC in a state in which the pattern image is projected. The captured image of the subject SC by the imaging device 20 in a state in which the pattern image is projected is an example of a second image in this disclosure. The multiple pixels constituting the second image are an example of third pixels that correspond one-to-one to the multiple first pixels constituting the first image. Then, the second acquisition unit 332 acquires distance information for each first pixel based on the pattern image and the captured image of the subject SC in a state in which the pattern image is projected, using the principle of triangulation. The distance information is an example of second information in this disclosure.

[0023] In this embodiment, information indicating the respective positions of the projector 10 and the imaging device 20, i.e., external parameters of the projector 10 and the imaging device 20, are stored in advance in the storage device 320. Information indicating the focus and lens distortion of the projection lens of the projector 10, i.e., internal parameters of the projector 10, are also stored in advance in the storage device 320. Information indicating the focus and lens distortion of the imaging lens of the imaging device 20, i.e., internal parameters of the imaging device 20, are also stored in advance in the storage device 320. These external parameters and internal parameters are used when performing triangulation.

[0024] The calculation unit 333 first allows the user to set a pixel representing the foreground region and a pixel representing the background region in the first image. The foreground region refers to a region corresponding to the object SC1 in the first image. The pixel representing the foreground region is at least one pixel among the pixels belonging to the foreground region. The foreground region is an example of a first region in the present disclosure. The pixel representing the foreground region in the first image is an example of a second pixel in the present disclosure. The background region refers to a region corresponding to the background object SC2 in the first image. The pixel representing the background region is at least one pixel among the pixels belonging to the background region.

[0025] In this embodiment, the calculation unit 333 displays the first image on the display device of the touch panel 310. The user performs an operation to designate a pixel representing the foreground region and an operation to designate a pixel representing the background region on the touch panel 310 in a state in which the display device is displaying the first image, thereby setting a pixel representing the foreground region and a pixel representing the background region. A specific example of the operation to designate a pixel representing the foreground region is an operation to draw a line or a point in the foreground region. In a mode in which a pixel representing the foreground region is designated by an operation to draw a line or a point, a pixel on the line or point drawn in the foreground region becomes a pixel representing the foreground region. The same applies to the operation to designate a pixel representing the background region. FIG. 2 illustrates an example in which a pixel representing the foreground is designated by an operation to draw a line indicated by a dotted line on the first image G1, and a pixel representing the background region is designated by an operation to draw a line indicated by a dashed line.

[0026] Next, the calculation unit 333 performs a calculation using the reflected light information and the distance information as variables for each pixel constituting the first image, thereby calculating an index value as a result of the calculation for each pixel. In this embodiment, the calculation unit 333 calculates a first index value and a second index value for each pixel constituting the first image. Hereinafter, a pixel that is the target of the calculation of the index value is referred to as a target pixel.

[0027] The first index value is an index value calculated based on the luminance indicated by the reflected light information of the pixel representing the foreground region and the distance indicated by the distance information of the pixel representing the foreground region. Hereinafter, the luminance indicated by the reflected light information of the pixel representing the foreground region is referred to as the first luminance reference. Also, the distance indicated by the distance information of the pixel representing the foreground region is referred to as the first distance reference. When there is one pixel representing the foreground region, the luminance indicated by the reflected light information of that pixel may be used as the first luminance reference. When multiple pixels representing the foreground region are set, the average value of the luminance indicated by the reflected light information of each of the multiple pixels may be used as the first luminance reference. The same applies to the first distance reference.

[0028] The second index value is an index value calculated based on the luminance indicated by the reflected light information of the pixel representing the background region and the distance indicated by the distance information of the pixel representing the background region. Hereinafter, the luminance indicated by the reflected light information of the pixel representing the background region is referred to as the second luminance standard. Also, the distance indicated by the distance information of the pixel representing the background region is referred to as the second distance standard. The second luminance standard and the second distance standard may be calculated in the same manner as the first luminance standard and the first distance standard.

[0029] In this embodiment, the calculation for calculating the first index value is the sum of the result of multiplying the difference between the luminance indicated by the reflected light information of the target pixel and the first luminance reference by the first coefficient α and the result of multiplying the difference between the distance indicated by the distance information of the target pixel and the first distance reference by the second coefficient β. The first coefficient α is zero or a positive value. The second coefficient β is also zero or a positive value. However, the first coefficient α and the second coefficient β are not both set to zero. In this embodiment, the first coefficient α and the second coefficient β are fixed values ​​preset in the program 321. For example, it is assumed that the difference between the luminance indicated by the reflected light information of the target pixel and the first luminance reference is expressed by the square root of d11. It is also assumed that the difference between the distance indicated by the distance information of the target pixel and the first distance reference is expressed by the square root of d12. For this target pixel, the calculation unit 333 calculates the first index value J1 according to the following formula (1). When the target pixel is a pixel that represents the foreground region, d11 and d12 are both zero or a value close to zero, and therefore the first index value J1 is 0 or a value close to zero.

number

[0030] In this embodiment, the calculation for calculating the second index value is the sum of the result of multiplying the difference between the luminance indicated by the luminance information of the target pixel and the second luminance standard by the first coefficient α and the result of multiplying the difference between the distance indicated by the distance information of the target pixel and the second distance standard by the second coefficient β. For example, it is assumed that the difference between the luminance indicated by the reflected light information of the target pixel and the second luminance standard is expressed by the square root of d21. It is also assumed that the difference between the distance indicated by the distance information of the target pixel and the second distance standard is expressed by the square root of d22. For this target pixel, the calculation unit 333 calculates the second index value J2 according to the following formula (2). When the target pixel is a pixel representing the background region, both d21 and d22 are zero or near zero, so the second index value J2 is 0 or near zero.

number

[0031] The identification unit 334 identifies the foreground region by determining whether each of the pixels constituting the captured image of the subject SC belongs to the foreground region based on the index value calculated by the calculation unit 333. Specifically, the identification unit 334 determines that a pixel whose first index value J1 is 0 or a value close to 0 belongs to the foreground region. The identification unit 334 also determines that a pixel whose second index value J2 is 0 or a value close to 0 belongs to the background region. The identification unit 334 may also determine that a pixel whose first index value J1 is smaller than the second index value J2 belongs to the foreground region, and determine that a pixel whose second index value J2 is smaller than the first index value J1 belongs to the background region. In this way, by identifying pixels that belong to the foreground region among the pixels constituting the captured image of the subject SC, segmentation is realized to divide the first image into a foreground region and a background region.

[0032] The projection control unit 335 divides the projection image projected from the projector 10 onto the subject SC into a second region corresponding to the foreground region identified by the identification unit 334, and a third region other than the second region. Then, the projection control unit 335 provides the projector 10 with image data representing a projection image in which the second region is filled with a first color and the third region is filled with a second color, thereby causing the projector 10 to project onto the subject SC a projection image in which the second region is filled with a first color and the third region is filled with a second color. The projection control unit 335 may also cause the projector 10 to project onto the subject SC a projection image in which a first content image is drawn in the second region and a second content image different from the first content image is drawn in the third region. The content image is, for example, an image showing a photograph or a picture. The projection image that the projection control unit 335 causes the projector 10 to project is not limited to the above-mentioned form, and may be any form in which the second area and the third area are different.

[0033] Moreover, the processing device 330 operating according to the program 321 executes the projection method of the present disclosure. Fig. 3 is a flowchart showing the flow of the projection method of the present disclosure. As shown in Fig. 3, the projection method of the present disclosure includes a first acquisition process SA110, a second acquisition process SA120, a calculation process SA130, a determination process SA140, and a projection control process SA150.

[0034] In the first acquisition process SA110, the processing device 330 functions as a first acquisition unit 331. In the first acquisition process SA110, the processing device 330 acquires reflected light information regarding reflected light from the subject SC for each pixel constituting the captured image based on the captured image of the subject SC.

[0035] In the second acquisition process SA120, the processing device 330 functions as a second acquisition unit 332. In the second acquisition process SA120, the processing device 330 acquires distance information corresponding to the distance from each part of the subject SC to the imaging device 20 for each pixel constituting the captured image of the subject SC.

[0036] In calculation process SA130, processing device 330 functions as calculation unit 333. In calculation process SA130, processing device 330 allows a user to set a pixel representing a foreground region and a pixel representing a background region, and calculates a first index value and a second index value for each pixel constituting the captured image of subject SC.

[0037] In the identification process SA140, the processing device 330 functions as an identification unit 334. In the identification process SA140, the processing device 330 determines whether or not each pixel constituting the captured image of the subject SC belongs to the foreground region based on the first index value and the second index value calculated in the calculation process SA130, thereby identifying the foreground region in the captured image.

[0038] In the projection control process SA150, the processing device 330 functions as a projection control unit 335. In the projection control process SA150, the processing device 330 divides the projection image projected from the projector 10 onto the subject SC into a second region corresponding to the foreground region and a third region other than the second region, and causes the projector 10 to project a projection image in which the second region is filled with a first color and the third region is filled with a second color.

[0039] In this embodiment, the foreground region is identified using an index value calculated based on the luminance of reflected light from the subject SC and the distance from the subject SC to the imaging device 20. Fig. 4 is a diagram showing an example of a foreground region AC1 and a background region AC2 identified in this embodiment when the first coefficient α is 1 and the second coefficient β is 0.1. Region A1 in Fig. 4 is a region where the difference between the distance from the object SC1 to the imaging device 20 and the distance from the background object SC2 to the imaging device 20 is small. Region A2 in Fig. 4 is a region where the difference between the luminance of reflected light from the object SC1 and the luminance of reflected light from the background object SC2 is small.

[0040] In a mode in which the foreground region is identified based only on distance information, erroneous determination may occur for pixels belonging to region A1. Therefore, in a mode in which the foreground region is identified based only on distance information, as shown in Fig. 5, the boundary between foreground region AC1 and background region AC2 in region A1 may become shaky. In contrast, according to this embodiment, whether a pixel belongs to the foreground region is determined based on an index value calculated from reflected light information and distance information, so that erroneous determination of pixels belonging to region A1 can be avoided.

[0041] On the other hand, in a mode in which the foreground region is identified based only on reflected light information, erroneous determination of pixels belonging to region A2 may occur. Therefore, in a mode in which the foreground region is identified based only on reflected light information, as shown in Fig. 6, the boundary between foreground region AC1 and background region AC2 in region A2 may become shaky. In contrast, according to this embodiment, whether a pixel belongs to the foreground region is determined based on an index value calculated from reflected light information and distance information, so that erroneous determination of pixels belonging to region A2 can be avoided.

[0042] As described above, according to this embodiment, even when there is no difference in color between the object SC1 and the background object SC2, or when the color in a specific wavelength range cannot be captured with high accuracy due to the characteristics of the camera, etc., it is possible to accurately identify the foreground area.

[0043] 2. Variations The above embodiment may be modified as follows. (1) The second acquisition unit 332 in the above embodiment acquires the second information according to the distance from each part of the subject SC to the imaging device 20 for each pixel constituting the captured image of the imaging device 20 by the principle of triangulation using the projection image projected from the projector 10 onto the subject SC and the captured image of the subject SC by the imaging device 20. However, the second information may be acquired for each pixel constituting the captured image of the imaging device 20 by the principle of triangulation using the captured image of the subject SC by the imaging device 20 in a state in which a pattern image is projected from the projector 10 and the captured image of the subject SC by a second camera disposed at a position different from the imaging device 20. The captured image obtained by capturing the subject SC in a state in which a pattern image is projected by the second camera is an example of the third image in the present disclosure. The multiple pixels constituting the third image are an example of fourth pixels that correspond one-to-one to the multiple third pixels constituting the second image. In addition, a distance camera may be provided separately from the imaging device 20, and the second information may be acquired from a distance image obtained by capturing an image of the subject SC by this distance camera. The range image obtained by capturing an image of the subject SC with the range camera is an example of a fourth image in this disclosure.

[0044] (2) In the above embodiment, the pixel representing the foreground region and the pixel representing the background region are set by the user. However, either one of the pixel representing the foreground region and the pixel representing the background region may be set by the user. In an embodiment in which only the pixel representing the foreground region is set by the user, the foreground region is identified using only the first index value. Specifically, a pixel having a first index value less than a predetermined threshold is determined to be a pixel belonging to the foreground region. In an embodiment in which only the pixel representing the background region is set by the user, the foreground region is identified using only the second index value. Specifically, a pixel having a second index value equal to or greater than a predetermined threshold is determined to be a pixel belonging to the foreground region. In addition, at least one of the pixel representing the foreground region and the pixel representing the background region in the captured image of the subject SC may be automatically set based on distance information of each pixel. Since the distance from the object SC1 to the imaging device 20 is generally different from the distance from the background object SC2 to the imaging device 20, the contour of the foreground region may be roughly extracted based on the distance information of each pixel. Therefore, the pixel on the line obtained by contracting the contour extracted based on the distance information may be determined to be the pixel representing the foreground region. Similarly, the pixels on the line resulting from the expansion of the contour may be regarded as pixels representative of the background region.

[0045] (3) In the above embodiment, the first index value J1 is calculated according to the formula (1), but the first index value J1 may be calculated according to the following formula (3) or formula (4). The same applies to the second index value. In short, the calculation in the calculation unit 333 may be a calculation that calculates an index value using the first information and the second information as variables.

number

number

[0046] (4) In the above embodiment, the calculation unit 333 calculates the index value using the first coefficient and the second coefficient, which are preset fixed values. However, the calculation unit 333 may display a screen on the touch panel 310 prompting the user to set the first coefficient and the second coefficient prior to the calculation of the index value, and may receive an operation from the user to set the first coefficient and the second coefficient. Then, the calculation unit 333 may calculate the index value according to the first coefficient and the second coefficient set by the user. According to this aspect, it is possible to allow the user to flexibly set whether to give priority to a judgment based on color difference or a judgment based on distance difference depending on the situation.

[0047] Furthermore, prior to the calculation of the index value, the calculation unit 333 may set the first coefficient according to a histogram of the occurrence frequency of the luminance or gradation in the first image indicated by the reflected light information, and set the second coefficient according to a histogram of the occurrence frequency of the distance indicated by the distance information. According to this aspect, for example, when the first image is an image of each frame constituting a moving image of the object SC1, it becomes possible to automatically set a suitable first coefficient and a suitable second coefficient for each frame of the moving image.

[0048] When the difference between the color of the object SC1 and the color of the background object SC2 is small, as shown in FIG. 7, only one peak appears in the histogram of the occurrence frequency of SSD calculated based on the reflected light information. In FIG. 7, the contour of the object SC1 is drawn with a dotted line. Similarly, in FIG. 8 and FIG. 9, the contour of the object SC1 is drawn with a dotted line. In contrast, when the difference between the color of the object SC1 and the color of the background object SC2 is large, as shown in FIG. 8, two peaks appear in the histogram of the occurrence frequency of SSD. When the boundary between the color of the object SC1 and the color of the background object SC2 is not clear, such as when the boundary between the two objects is a gradation, the boundary between the two peaks also becomes gentle, as shown in FIG. 9. When only one peak appears in the histogram of the occurrence frequency of SSD as shown in FIG. 7, the first coefficient is set to zero or a value close to zero, thereby avoiding the occurrence of erroneous judgment due to the small difference between the color of the object SC1 and the color of the background object SC2.

[0049] As shown in Fig. 10, when the difference between the distance X from the object SC1 to the imaging device 20 and the distance Y from the background object SC2 to the imaging device 20 is small, only one peak appears near the distance difference = 0 in the histogram of the occurrence frequency of the distance difference calculated based on the distance information. In contrast, as shown in Fig. 11, when the difference between the distance X from the object SC1 to the imaging device 20 and the distance Y from the background object SC2 to the imaging device 20 is large, two peaks appear in the histogram of the occurrence frequency of the distance difference. When only one peak appears in the histogram of the occurrence frequency of the distance difference as shown in Fig. 10, the second coefficient is set to zero or a value close to zero, thereby avoiding the occurrence of an erroneous determination caused by the small difference between the distance X from the object SC1 to the imaging device 20 and the distance Y from the background object SC2 to the imaging device 20.

[0050] Furthermore, the calculation unit 333 may set the first coefficient and the second coefficient based on the difference between the first luminance standard and the second luminance standard and the difference between the first distance standard and the second distance standard prior to the calculation of the index value. More specifically, when the difference value between the first luminance standard and the second luminance standard is greater than the difference value between the first distance standard and the second distance standard, the calculation unit 333 sets the first coefficient to a value greater than the second coefficient. On the other hand, when the difference value between the first luminance standard and the second luminance standard is smaller than the difference value between the first distance standard and the second distance standard, the calculation unit 333 sets the first coefficient to a value smaller than the second coefficient. Furthermore, the calculation unit 333 may set the first coefficient according to the difference value between the first luminance standard and the second luminance standard, and set the second coefficient based on the difference between the first distance standard and the second distance standard. More specifically, the calculation unit 333 sets the first coefficient to a value greater as the difference value between the first luminance standard and the second luminance standard is greater. Furthermore, the calculation unit 333 sets the second coefficient so that the larger the difference between the first distance criterion and the second distance criterion, the larger the value of the second coefficient becomes. According to this aspect, it is possible to automatically set more suitable first and second coefficients.

[0051] (5) In the above embodiment, the first acquisition unit 331, the second acquisition unit 332, the calculation unit 333, the determination unit 334, and the projection control unit 335 are software modules. However, some or all of the first acquisition unit 331, the second acquisition unit 332, the calculation unit 333, the determination unit 334, and the projection control unit 335 may be hardware. Examples of this hardware include a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). Even if some or all of the first acquisition unit 331, the second acquisition unit 332, the calculation unit 333, the determination unit 334, and the projection control unit 335 are hardware, the same effects as those of the above embodiment can be achieved.

[0052] (6) In the above embodiment, the projection method including the first acquisition process SA110, the second acquisition process SA120, the calculation process SA130, the identification process SA140, and the projection control process SA150 has been described. However, a method in which the projection control process SA150 is omitted from this projection method, that is, an identification method for identifying a foreground area corresponding to an object SC1 included in a captured image of the subject SC, may be provided. Also, an information processing device including the first acquisition unit 331, the second acquisition unit 332, the calculation unit 333, and the identification unit 334 may be provided by omitting the projection control unit 335 from the information processing device 30. By combining this information processing device with the imaging device 20, a identification system for identifying a foreground area corresponding to an object SC1 included in the subject SC in a captured image of the subject SC by the imaging device 20 is configured. Also, in the above embodiment, the imaging device 20 is provided separately from the information processing device 30, but the information processing device 30 may include the imaging device 20. Specific examples of the information processing device 30 including the imaging device 20 include a smartphone or a tablet terminal. Furthermore, in the above embodiment, the information processing device 30 includes a display device, but a display device such as a liquid crystal display may be provided separately from the information processing device 30. Furthermore, the projector 10 may be used as a display device.

[0053] (7) In the above embodiment, the program 321 is stored in the storage device 320. However, the program 321 may be manufactured or distributed as a standalone program. Specific methods for distributing the program 321 include distributing the program 321 by writing it to a computer-readable recording medium such as a flash ROM (Read Only Memory) or the like, or distributing it by downloading it via a telecommunications line such as the Internet.

[0054] 3. Aspects grasped from at least one of the embodiments and each modified example The present disclosure is not limited to the above-mentioned embodiments and modifications, and can be realized in various aspects without departing from the spirit of the present disclosure. For example, the present disclosure can also be realized in the following aspects. The technical features in the above-mentioned embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined in order to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0055] In order to solve the above-mentioned problems, the identification method of the present disclosure includes the following first acquisition process SA110, second acquisition process SA120, calculation process SA130, and identification process SA140. In the first acquisition process SA110, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject is acquired for each first pixel constituting the first image. In the second acquisition process SA120, second information according to the distance from the first subject to the first camera is acquired for each first pixel. In the calculation process SA130, a calculation is performed using the first information and the second information as variables, and an index value, which is a result of the calculation, is calculated for each first pixel. In the identification process SA140, the first region is identified by determining for each first pixel whether or not the first pixel belongs to a first region of the first image corresponding to the object based on the index value. According to the present identification method, even when there is a small difference between the color of an object included in the subject and the color of background objects other than the object, or when the color of a specific wavelength range cannot be captured with high accuracy due to the characteristics of the camera, etc., it is possible to accurately identify the area corresponding to the object.

[0056] In a more preferred embodiment of the identification method, the first information may be at least one of luminance information indicating the luminance of the reflected light and gradation information indicating the gradation of each color of the reflected light. According to the identification method of this embodiment, it is possible to accurately identify an area corresponding to an object by performing a single determination for each pixel of the first image based on an index value calculated from the luminance or gradation of the reflected light from the subject and the distance from the subject to the first camera.

[0057] In another preferred embodiment of the identification method, in the calculation process SA130, the second pixel belonging to the first region may be set based on a user's instruction or the second information, prior to the execution of the calculation to calculate the index value. The calculation performed in the calculation process SA130 may be the sum of a result of multiplying the difference between the first information about the first pixel and the first information about the second pixel by a first coefficient and a result of multiplying the difference between the second information about the first pixel and the second information about the second pixel by a second coefficient. According to the identification method of this embodiment, it is possible to identify the region corresponding to the object based on the sum of a result of multiplying the difference between the first information about the first pixel and the first information about the second pixel by a first coefficient and a result of multiplying the difference between the second information about the first pixel and the second information about the second pixel by a second coefficient.

[0058] In a further preferred embodiment of the method, the second information may be generated based on a pattern image projected from the projector 10 onto the subject SC, and a second image obtained by imaging the subject SC with the image projected by the image capture device 20. The second image is preferably composed of a plurality of third pixels that correspond one-to-one to a plurality of first pixels that constitute the first image. In another preferred embodiment of the method, the second information may be generated based on a second image obtained by imaging the subject SC with the image capture device 20 with the pattern image projected from the projector 10, and a third image obtained by imaging the subject SC with the second camera. The second image is preferably composed of a plurality of third pixels that correspond one-to-one to a plurality of first pixels that constitute the first image. The third image is preferably composed of a plurality of fourth pixels that correspond one-to-one to a plurality of third pixels that constitute the second image. In the preferred embodiment of the method, the second information may be generated based on a fourth image obtained by imaging the first subject with a distance camera that captures a distance image whose pixel value represents the distance to the subject.

[0059] In another preferred embodiment of the method, the first coefficient and the second coefficient may be preset. According to this embodiment, it becomes possible to calculate an index value for determining for each pixel whether or not the pixel belongs to a region corresponding to an object, based on the preset first coefficient and second coefficient.

[0060] In another more preferred embodiment of the identification method, prior to calculation of the index value, settings of the first coefficient and the second coefficient may be accepted, and the index value may be calculated according to the accepted settings of the first coefficient and the second coefficient. According to the identification method of this embodiment, it is possible to calculate an index value for determining for each pixel whether or not the pixel belongs to a region corresponding to an object, based on the first coefficient and the second coefficient set by a user.

[0061] In another more preferred embodiment of the identification method, prior to calculating the index value, the first coefficient may be set according to a histogram of the frequency of occurrence of luminance or gradation in the first image indicated by the first information, and the second coefficient may be set according to a histogram of the frequency of occurrence of distance indicated by the second information. According to the identification method of this embodiment, it becomes possible to calculate an index value for determining for each pixel whether or not the pixel belongs to a region corresponding to an object, based on the first coefficient set according to the histogram of the frequency of occurrence of luminance or gradation in the first image, and the second coefficient set according to the histogram of the frequency of occurrence of distance indicated by the second information.

[0062] In order to solve the above problem, the projection method of the present disclosure includes the above-mentioned first acquisition process SA110, second acquisition process SA120, calculation process SA130, and specification process SA140, and further includes the following projection control process SA150. In the projection control process SA150, a projection image projected from a projector onto the first object is divided into a second area corresponding to the first area and a third area other than the second area, and the second area is filled with a first color and the third area is filled with a second color, and the projector projects the projection image. According to the projection method of this aspect, even if the difference between the color of an object included in the object and the color of a background object other than the object is small, or even if a color in a specific wavelength range cannot be captured with high accuracy due to the characteristics of the camera, etc., it is possible to accurately specify the area corresponding to the object. It is possible to project onto the first object a projection image in which the area corresponding to the object is accurately filled with the first color and the area corresponding to the background object other than the object is accurately filled with the second color.

[0063] In order to solve the above problem, the identification system of the present disclosure includes a first camera and an information processing device. This information processing device executes the above-mentioned first acquisition process SA110, second acquisition process SA120, calculation process SA130, and identification process SA140. According to the identification system of this embodiment, even if the difference between the color of an object included in the subject and the color of a background object other than the object is small, or even if the color of a specific wavelength range cannot be captured with high accuracy due to the characteristics of the camera, it is possible to accurately identify the area corresponding to the object.

[0064] In order to solve the above problem, the information processing device of the present disclosure executes the first acquisition process SA110, the second acquisition process SA120, the calculation process SA130, and the determination process SA140. information The processing device makes it possible to accurately identify the area corresponding to the object even when there is a small difference in color between an object included in the subject and the color of background objects other than the object, or when the camera characteristics or the like make it impossible to capture colors in a specific wavelength range with high accuracy.

[0065] In order to solve the above problem, the program of the present disclosure causes a computer to execute the above-mentioned first acquisition process SA110, second acquisition process SA120, calculation process SA130, and identification process SA140. According to the program of this aspect, even if the difference between the color of an object included in the subject and the color of a background object other than the object is small, or even if the color of a specific wavelength range cannot be captured with high accuracy due to the characteristics of the camera, etc., it is possible to accurately identify the area corresponding to the object. [Explanation of symbols]

[0066] 1...projection system, 10...projector, 20...imaging device, 30...information processing device, 300...communication device, 310...touch panel, 320...storage device, 330...processing device, 331...first acquisition unit, 332...second acquisition unit, 333...calculation unit, 334...identification unit, 335...projection control unit, 321...program, SC1...object, SC2...background object, SC...subject.

Claims

1. acquiring, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image; acquiring second information corresponding to a distance from the first object to the first camera for each of the first pixels; executing an operation using the first information and the second information as variables, and calculating an index value as a result of the operation for each of the first pixels; determining, for each of the first pixels, whether or not the pixel belongs to a first region of the first image corresponding to the object based on the index value, thereby identifying the first region; Specific method.

2. The method according to claim 1 , wherein the first information is at least one of luminance information indicating a luminance of the reflected light and gradation information indicating a gradation for each color of the reflected light.

3. Prior to the execution of the calculation, a second pixel belonging to the first region is set based on a user instruction or the second information; 3. The method of claim 1, wherein the calculation is a sum of a result of multiplying a difference between the first information about the first pixel and the first information about the second pixel by a first coefficient and a result of multiplying a difference between the second information about the first pixel and the second information about the second pixel by a second coefficient.

4. The second information is A pattern image projected from a projector onto the first object; a second image obtained by imaging the first subject with the first camera while the pattern image is being projected, the second image being configured of a plurality of third pixels that correspond one-to-one to a plurality of the first pixels that configure the first image; The method according to claim 1 , wherein the identification is performed based on:

5. The second information is a second image obtained by capturing an image of the first subject with the first camera while a pattern image is being projected from a projector, the second image being configured of a plurality of third pixels that correspond one-to-one to a plurality of the first pixels that configure the first image; a third image obtained by imaging the first subject with a second camera while the pattern image is being projected, the third image being configured of a plurality of fourth pixels that correspond one-to-one to the plurality of third pixels that configure the second image; The method according to claim 1 , wherein the identification is performed based on:

6. the second information is generated based on a fourth image obtained by capturing an image of the first subject as the subject using a distance camera that captures a distance image in which pixel values ​​represent a distance to the subject; The method according to any one of claims 1 to 3.

7. The method according to claim 3 , wherein the first coefficient and the second coefficient are preset.

8. The method according to claim 3 , further comprising the steps of: receiving settings of the first coefficient and the second coefficient prior to the execution of the operation; and calculating the index value according to the received settings of the first coefficient and the second coefficient.

9. The method of claim 3, wherein prior to performing the calculation, the first coefficient is set according to a histogram of occurrence frequency of brightness or gradation in the first image indicated by the first information, and the second coefficient is set according to a histogram of occurrence frequency of distance indicated by the second information.

10. acquiring, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image; acquiring second information corresponding to a distance from the first object to the first camera for each of the first pixels; executing a calculation using the first information and the second information as variables, and calculating an index value as a result of the calculation for each of the first pixels; determining, for each of the first pixels, whether or not the pixel belongs to a first region of the first image corresponding to the object based on the index value, thereby identifying the first region; a projection image to be projected from a projector onto the first subject is divided into a second area corresponding to the first area and a third area other than the second area, the second area is filled with a first color, and the third area is filled with a second color, and the projection image is projected by the projector; Projection method.

11. A first camera; An information processing device, The information processing device includes: acquiring, based on a first image obtained by capturing an image of a first subject including an object with the first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image; acquiring second information corresponding to a distance from the first object to the first camera for each of the first pixels; executing an operation using the first information and the second information as variables, and calculating an index value as a result of the operation for each of the first pixels; and determining, for each of the first pixels, whether or not the pixel belongs to a first region of the first image corresponding to the object based on the index value, thereby identifying the first region; The specific system on which you run.

12. acquiring, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image; acquiring second information corresponding to a distance from the first object to the first camera for each of the first pixels; executing an operation using the first information and the second information as variables, and calculating an index value as a result of the operation for each of the first pixels; and determining, for each of the first pixels, whether or not the pixel belongs to a first region of the first image corresponding to the object based on the index value, thereby identifying the first region; An information processing device that executes the above.

13. On the computer, acquiring, based on a first image obtained by capturing an image of a first subject including an object with a first camera, first information regarding reflected light from the first subject for each first pixel constituting the first image; acquiring second information corresponding to a distance from the first object to the first camera for each of the first pixels; executing an operation using the first information and the second information as variables, and calculating an index value as a result of the operation for each of the first pixels; and determining, for each of the first pixels, whether or not the pixel belongs to a first region of the first image corresponding to the object based on the index value, thereby identifying the first region; A program that executes the following.

Images