Display method and program

Abstract

To provide a display method and a program that facilitate an adjustment to a mask region.SOLUTION: A processing apparatus that receives an input of a threshold generates a temporary masked region 540 indicating a masked region, which is a region in a captured image where an input image 542 is displayed or a non-masked region, which is a region in the captured image where the input image is not displayed, by using the threshold. The processing apparatus causes a temporary projection image 54 in which the temporary masked image 540 is superposed on the input image 542 to be displayed. The threshold is a value used for determining whether each of pixels of the captured image is a pixel constituting the masked region, and is a value indicating a probability in which each of the pixels of the captured image is a pixel constituting the masked region, for example.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display method and a program. [Background technology]

[0002] For example, Patent Document 1 below discloses a subject detection method for detecting a subject from an input image. The subject detection method of Patent Document 1 acquires feature amounts from the input image, generates a probability distribution image indicating the probability of the subject's existence based on the feature amounts, and detects the subject by comparing the probability distribution image with a pre-stored threshold value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-99070 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, since a pre-stored threshold is used for detecting a subject, there is a possibility that the area that the user really wants to detect may not be detected. Therefore, with the conventional technology, it is not possible to easily adjust the area according to the user's sensibility. [Means for solving the problem]

[0005] A display method according to one embodiment of the present invention includes accepting input of a threshold value, generating a second image using the threshold value to indicate a masked area in a first image in which an input image is displayed, or a non-masked area in the first image in which the input image is not displayed, and displaying a projection image in which the second image is superimposed on the input image.

[0006] A program according to one embodiment of the present invention causes a computer to accept input of a threshold value, generate a second image using the threshold value to indicate a masked area, which is an area of ​​a first image where an input image is displayed, or a non-masked area, which is an area of ​​the first image where the input image is not displayed, and display a projection image in which the second image is superimposed on the input image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a projection system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the processing device. [Diagram 3] 11 is a diagram showing an example of a manner in which a region selection receiving unit receives a region selection. FIG. [Figure 4] 11 is a diagram showing an example of a manner in which a region selection receiving unit receives a region selection. FIG. [Diagram 5] 13 is a diagram showing an example of a probability distribution image generated by a probability calculation unit; FIG. [Figure 6] 11 is a diagram showing an example of a manner in which a threshold value input is received by a threshold value receiving unit; FIG. [Figure 7] 13 is a diagram illustrating an example of a display mode of an image to be projected by a display control unit. [Figure 8] 13 is a diagram illustrating an example of a display mode of an image to be projected by a display control unit. [Figure 9] 10 is a flowchart showing the flow of a display method executed by a processing device of the information processing device in accordance with a program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Note that the dimensions or scale of each part in the drawings may differ from the actual dimensions, and some parts are shown diagrammatically to facilitate understanding. Furthermore, the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited thereto.

[0009] 1. First embodiment FIG. 1 is a schematic diagram showing an example of a projection system 1 according to an embodiment. As shown in FIG. 1, the projection system 1 includes a projector 10, an imaging device 20, and an information processing device 30. The projection system 1 shown in FIG. 1 is a system for projecting an image onto an object 40 having a three-dimensional shape. In this embodiment, the object 40 is formed by cutting a plate-like member having a predetermined thickness into the shape of the letter "E". The object 40 has an adhesive surface (not shown) that contacts a wall surface 42, a surface 400 that is the surface opposite to the adhesive surface, and a side surface 402 that connects the adhesive surface and the surface 400. In this embodiment, the surface 400 of the object 40 is set as a projection area of ​​an image. The projection area of ​​an image from the projector 10 corresponds to a mask area described later.

[0010] The projector 10 includes a liquid crystal light valve, a projection lens, and a liquid crystal driver for projecting projection images 56 and 58, which will be described later. The projector 10 also includes an extra-high pressure mercury lamp or a metal halide lamp as a light source. The projector 10 is connected to the information processing device 30 via, for example, a cable so as to be able to communicate with the information processing device 30. The projector 10 acquires projection image data from the information processing device 30 by communication via the cable. The projector 10 projects the projection images 56 and 58 indicated by the acquired projection image data onto a masked area. In the following, the masked area is an area where input images 562 and 582, which will be described later, are displayed, and an area other than the masked area is an area where the input images 562 and 582 are not displayed, that is, an area that is not masked. In this embodiment, the communication between the projector 10 and the information processing device 30 is, for example, wired communication conforming to a standard such as Ethernet or USB (Universal Serial Bus). However, the communication between the projector 10 and the information processing device 30 may be wireless communication conforming to a standard such as Wi-Fi. Note that Wi-Fi and Ethernet are registered trademarks.

[0011] The imaging device 20 is, for example, a camera equipped with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), which is an imaging element that converts collected light into an electric signal. In the following, for the sake of simplicity of explanation, it is assumed that the imaging device 20 captures a still image. Note that the imaging device 20 may capture a moving image instead of capturing a still image. The imaging device 20 captures an image of an imaging area 44 including an object 40. In this embodiment, the imaging area 44 includes the object 40 and a wall surface 42. Note that, in this embodiment, it is assumed that projection from the projector 10 is not performed when capturing an image with the imaging device 20. That is, the captured image 50 captured by the imaging device 20 is an image captured in the imaging area 44 in a state where projection from the projector 10 is not performed. The captured image 50 is exemplified in FIG. 3. The imaging device 20 is connected to the information processing device 30 by, for example, a cable so as to be able to communicate with the information processing device 30, similar to the projector 10. The imaging device 20 transmits captured image data representing a captured image to the information processing device 30. In this embodiment, communication between the imaging device 20 and the information processing device 30 is, for example, wired communication conforming to a standard such as Ethernet or USB, but may be wireless communication conforming to a standard such as Wi-Fi. In this embodiment, the imaging device 20 is separate from the information processing device 30 and the projector 10, but the imaging device 20 may be mounted on either the information processing device 30 or the projector 10.

[0012] The information processing device 30 is an example of an electronic device, such as a personal computer. As shown in FIG. 1, the information processing device 30 has a communication device 300, a touch panel 310, a storage device 320, and a processing device 330. The communication device 300 is connected to the projector 10 via a cable. The communication device 300 is also connected to the imaging device 20 via a cable. The communication device 300 receives captured image data transmitted from the imaging device 20. The communication device 300 also transmits projection image data representing an image to be projected on the object 40 to the projector 10 under the control of the processing device 330.

[0013] The touch panel 310 is a device in which a display device that displays various information and an input device to which information is input by a user are integrated. The input device is, for example, a transparent sheet-like contact sensor. The input device is provided so as to cover the display surface of the display device. The input device detects a touch position using electrostatic capacitance determined by an object that contacts the input device and the input device. The input device outputs data indicating the detected touch position to the processing device 330. As a result, the contents of the user's operation on the touch panel 310 are transmitted to the processing device 330. In this embodiment, the touch panel 310 serves as both the display device and the input device, but the display device and the input device may be provided separately. Specifically, for example, the information processing device 30 may be provided with a display as a display device and a keyboard and a mouse as input devices.

[0014] The storage device 320 is a recording medium that can be read by the processing device 330. The storage device 320 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Ra n dom Access Memory).

[0015] A program 322 to be executed by the processing device 330 is stored in advance in the non-volatile memory of the storage device 320. The volatile memory of the storage device 320 is used by the processing device 330 as a work area when executing the program 322. The program 322 may also be referred to as an "application program," "application software," or an "app." The program 322 is obtained, for example, from a server or the like (not shown) via the communication device 300, and then stored in the storage device 320.

[0016] The processing device 330 includes a processor such as a CPU (Central Processing Unit), that is, a computer. The processing device 330 may be configured with a single computer or multiple computers. The processing device 330 reads the program 322 from the non-volatile memory to the volatile memory when an operation to instruct the start of execution of the program 322 is performed on the input device. The processing device 330 executes the program 322 read into the volatile memory.

[0017] Fig. 2 is a block diagram showing a functional configuration of the processing device 330. The processing device 330 operating according to the program 322 functions as a captured image acquisition unit 331, a region selection acceptance unit 332, a probability calculation unit 333, a threshold acceptance unit 334, a mask image generation unit 335, and a display control unit 336 shown in Fig. 2. The captured image acquisition unit 331, the region selection acceptance unit 332, the probability calculation unit 333, the threshold acceptance unit 334, the mask image generation unit 335, and the display control unit 336 shown in Fig. 2 are software modules realized by operating the processing device 330 according to the program 322.

[0018] The captured image acquisition unit 331 acquires a captured image 50 obtained by capturing an image of an imaging region 44 including an object 40 from the imaging device 20. The captured image 50 corresponds to a first image. In this embodiment, the captured image acquisition unit 331 receives captured image data from the imaging device 20 via the communication device 300. The captured image 50 may be the image itself captured by the imaging device 20, or may be an image obtained by converting the image captured by the imaging device 20 into an image captured from a viewpoint of the projector 10 using a structured light projection method or the like.

[0019] The area selection receiving unit 332 displays the captured image 50 acquired by the captured image acquisition unit 331 on the touch panel 310, and receives input of information for selecting a mask area using the captured image 50. In this embodiment, the area selection receiving unit 332 receives drawing input from the user on the captured image 50. This drawing input specifies a part of a mask area or a part of a non-mask area on the captured image 50.

[0020] 3 and 4 are diagrams showing an example of a state in which a region selection is received by the region selection receiving unit 332. In Fig. 3, a captured image 50 is displayed on the touch panel 310. The captured image 50 is an image obtained by capturing an image of the imaging region 44 including the object 40 shown in Fig. 1, and includes a first region 500 corresponding to the surface 400 of the object 40, a second region 502 corresponding to the side surface 402 of the object 40, and a third region 504 corresponding to the wall surface 42.

[0021] In addition, on the touch panel 310, a drawing button 60, a cancel button 62, and a completion button 64 are displayed together with the captured image 50. When the user touches the drawing button 60, the area selection receiving unit 332 displays an area type designation section 66 for designating whether the current drawing designates a masked area or a non-masked area, as shown in FIG. 4. The area type designation section 66 includes a masked area designation section 660 and a non-masked area designation section 662 as selection items. The user selects the masked area designation section 660 when designating a masked area, and selects the non-masked area designation section 662 when designating a non-masked area. In FIG. 4, the masked area designation section 660 is selected. When the user touches the captured image 50 in this state, the area selection receiving unit 332 displays the locus as if it were drawn with a pen. The area selection receiving unit 332 accepts information of any pixel designated by drawing input, regarding the pixel in the captured image 50 where the locus is displayed as a pixel included in the masked area. Similarly, when the user touches the captured image 50 while the non-mask region designation section 662 is selected, the region selection receiving section 332 displays the locus as if it were drawn with a pen. In this case, the region selection receiving section 332 accepts the pixel in the captured image 50 where the locus is displayed as a pixel included in the non-mask region.

[0022] In this embodiment, the surface 400 of the object 40 is the mask region, so when the user selects the mask region designation section 660, the user draws on the first region 500 of the captured image 50. At this time, the user may draw by tracing the entire first region 500 as in the locus L1, or may draw by touching only one point or multiple points of the first region 500 as in the locus L2. Also, drawing of a line as in the locus L1 and drawing of points as in the locus L2 may be combined. Also, when the user selects a non-mask region, the user selects the non-mask region designation section 662 in the region type designation section 66, and draws on at least one of the second region 502 and the third region 504 of the captured image 50. The user may designate at least one pixel of the captured image 50 and designate whether it is included in the mask region or the non-mask region.

[0023] Note that the user may be able to designate both the masked region designation portion 660 and the non-masked region designation portion 662 by designating and drawing the masked region designation portion 660, and then designating and further drawing the non-masked region designation portion 662. In this case, if the color of the trail when the masked region designation portion 660 is selected is different from the color of the trail when the non-masked region designation portion 662 is selected, the user will be able to easily recognize the region type of the portion that he or she has selected, improving operability.

[0024] The cancel button 62 is used to cancel the drawing input. For example, if the user wants to cancel the drawing because the trajectory goes beyond the first area 500, the user touches the cancel button 62 and then traces the part of the trajectory that the user wants to cancel. The area selection receiving unit 332 then erases that part of the trajectory and cancels the designation of the corresponding pixels as a masked area or a non-masked area. If the traced part is the entire drawing input, the entire input is cancelled, and if the traced part is only a part of the drawing input, only that part is cancelled.

[0025] A slider bar 600 is displayed below the drawing button 60. The slider bar 600 is an interface for adjusting the width of a path when touching the drawing button 60 to perform drawing. The slider bar 600 includes a bar 602 extending in a predetermined direction and a slider 604 movable on the bar 602. In the example of FIG. 4, the bar 602 extends in the left-right direction of the screen of the touch panel 310. The center position of the bar 602 in the left-right direction is the reference position of the slider 604, and the width of the path at this time is the reference width. When the user moves the slider 604 to the right from the reference position, the area selection receiving unit 332 makes the width of the path larger than the reference width. As a result, even if the same drawing operation is performed as when the slider 604 is at the reference position, more pixels are specified as the masked area or the non-masked area. Also, when the user moves the slider 604 to the left from the reference position, the area selection receiving unit 332 makes the width of the path smaller than the reference width. As a result, even if the same drawing operation is performed as when the slider 604 is in the reference position, the number of pixels designated as masked or unmasked areas will be reduced.

[0026] Similarly, a slider bar 620 is displayed below the cancel button 62. The slider bar 620 is an interface for adjusting the width of the cancellation area when the cancel button 62 is touched to cancel the drawing. The slider bar 620 includes a bar 622 extending in a predetermined direction and a slider 624 movable on the bar 622. In the example of FIG. 4, the bar 622 extends in the left-right direction of the screen of the touch panel 310. The center position of the bar 622 in the left-right direction is the reference position of the slider 624, and the width of the cancellation area at this time is the reference width. When the user moves the slider 624 to the right from the reference position, the area selection receiving unit 332 makes the width of the cancellation area larger than the reference width. As a result, even if the same cancellation operation is performed as when the slider 624 is at the reference position, more pixels of the trajectory are canceled. Also, when the user moves the slider 624 to the left from the reference position, the area selection receiving unit 332 makes the width of the cancellation area smaller than the reference width. As a result, even if the same drawing operation is performed as when the slider 604 is in the reference position, fewer pixels of the locus are cancelled.

[0027] When the user finishes drawing in the masked area or non-masked area, he / she presses the done button 64. The area selection receiving unit 332 receives the pixels selected by the user as the masked area or non-masked area, based on the drawing content at the time the done button 64 is pressed.

[0028] That is, the area selection receiving unit 332 displays the captured image 50 and receives input of information for selecting a mask area using the captured image 50. Receiving input of information for selecting a mask area by the area selection receiving unit 332 includes designating that any pixel of the captured image 50 is a pixel that constitutes a mask area or a pixel that constitutes a non-mask area. Furthermore, receiving input of information for selecting a mask area by the area selection receiving unit 332 includes receiving drawing input for the captured image 50. Furthermore, receiving input of information for selecting a mask area by the area selection receiving unit 332 includes canceling at least a part of the drawing input using the cancel button 62.

[0029] In this embodiment, the area selection receiving unit 332 receives the selection of the masked area or the non-masked area by receiving drawing on the masked area or the non-masked area. Selection The method of selection is not limited to this. For example, the user may specify the numbers of pixels included in the masked area or the numbers of pixels included in the non-masked area using the captured image 50 in which each pixel is numbered. Note that the numbering may be applied to all pixels or to some of the pixels.

[0030] The probability calculation unit 333 shown in FIG. 2 calculates the probability that each pixel in the captured image 50 is a pixel that constitutes a mask region. There are various known methods for calculating the probability that an arbitrary pixel on an image is included in a specific region, such as the Growcut method. As an example, a pixel selected as a mask region by the region selection receiving unit 332 has a 100% probability of being a mask region. The probability that other pixels are mask regions is calculated based on the similarity with the pixel selected as the mask region. The similarity between pixels can be calculated using parameters such as the pixel's color, depth, and distance from the pixel selected as the mask region. Hereinafter, the pixel selected as the mask region is referred to as a "mask region pixel," and the pixel for which the probability is calculated is referred to as a "target pixel."

[0031] For example, when calculating the probability using color, the RGB values ​​[Rx, Gx, Bx] of the target pixel and the RGB values ​​[R0, G0, B0] of the mask area pixel can be used to calculate the probability P1 that the target pixel is a mask area pixel using the following formula (1). P1 takes a value between 0 and 1, and the larger the value, the higher the probability that the target pixel is a mask area pixel. Note that the "255" in the denominator of the following formula (1) is the value when the gradation is expressed in 8 bits, that is, when each of the values ​​of Rx, Gx, Bx, R0, G0, and B0 is expressed as an integer between 0 and 255. When the gradation is not 8 bits, the value corresponding to "255" in the following formula (1) is the value obtained by subtracting 1 from the number of gradations.

[0032]

number

[0033] For example, when calculating the probability using distance, the probability P2 that the target pixel is a mask area pixel can be calculated using the distance [Dx] of the target pixel and the distance [D0] of the mask area pixel with the following formula (2). P2 takes a value between 0 and 1, and the larger the value, the higher the probability that the target pixel is a mask area pixel. Note that in the following formula (2), the distances Dx and D0 are scaled to values ​​between 0 and 255 to match the scale of the above formula (1).

[0034]

number

[0035] For example, when calculating the probability using color and distance, the probability P3 that the target pixel is a mask area pixel can be calculated using the RGB values ​​[Rx, Gx, Bx] of the target pixel, the RGB values ​​[R0, G0, B0] of the mask area pixel, the distance [Dx] of the target pixel, and the distance [D0] of the mask area pixel, using the following formula (3). P3 takes a value between 0 and 1, and the higher the value, the higher the probability that the target pixel is a mask area pixel. In the following formula (3), the color gradation is expressed by 8 bits, that is, integers from 0 to 255. The pixel distance is also scaled to a value from 0 to 255. α is the color weight, and β is the distance weight, and they can be set arbitrarily depending on which parameter is emphasized. The weights α and β do not need to be set. In other words, α and β can be set to 1.

[0036]

number

[0037] When depth or the like is used, the probability that the target image is a mask area pixel can be calculated in a similar manner. When calculating the probability using multiple parameters, as exemplified in the above formula (3), for example, weights may be assigned to the probabilities calculated using each parameter, and then these may be added together. When multiple mask area pixels are selected, the probability calculation unit 333 may calculate the probability using, for example, the average value of the parameter values ​​for probability calculation in all mask area pixels, or may calculate the probability using the parameter value of the mask area pixel located closest to the target pixel.

[0038] The probability calculation unit 333 may also use a coordinate conversion table between the imaging device 20 and the projector 10 to calculate the probability that each pixel of the captured image 50 is a pixel that constitutes the mask region. In the coordinate conversion table obtained by the structured light projection method, the value changes continuously when adjacent coordinates are on the same plane, whereas the value changes discontinuously when there is a change in the height direction between adjacent coordinates. Using this, the probability calculation unit 333 can estimate the position of the boundary between a plane and a three-dimensional object. In this embodiment, the probability calculation unit 333 can calculate the probability that each pixel of the captured image 50 is the surface 400 by estimating the position of the boundary between the object 40 and the wall surface 42 and the position of the boundary between the surface 400 and the side surface 402 of the object 40.

[0039] Furthermore, when a non-mask region is selected by the region selection receiving section 332, the probability calculation section 333 may calculate the probability that the pixels in the target image are pixels that form the non-mask region.

[0040] FIG. 5 is a diagram showing an example of a probability distribution image 52 generated by the probability calculation unit 333. 5 In , the probability that each pixel is a mask area is classified into five types of shading. Here, α1 to α4 are positive numbers less than 1 and greater than 0, and α1>α2>α3>α4. Shading pattern 1 is, for example, an area where the probability of being a mask area is α1 or more. Shading pattern 2 is an area where the probability of being a mask area is less than α1 and greater than α2. Shading pattern 3 is an area where the probability of being a mask area is less than α2 and greater than α3. Shading pattern 4 is an area where the probability of being a mask area is less than α3 and greater than α4. Shading pattern 5 is an area where the probability of being a mask area is less than α4.

[0041] Note that the probability calculation unit 333 only needs to calculate the probability that each pixel of the captured image 50 is a masked area or a non-masked area, and does not necessarily need to generate the probability distribution image 52 as shown in FIG. 5. Information indicating the probability that each pixel of the captured image 50 is a masked area or a non-masked area is called probability distribution information. In addition, in this embodiment, the information processing device 30 generates the probability distribution information, but this is not limited to this, and for example, the information processing device 30 may acquire probability distribution information generated by another information processing device through communication or the like. In addition, for example, probability distribution information generated by the probability calculation unit 333 in the past may be stored in the storage device 320, and the processing device 330 may read it out.

[0042] 2 receives a threshold value input from a user. This threshold value is a value used by a mask image generating unit 335 (described later) when determining whether or not each pixel of the captured image 50 is a pixel that constitutes a mask region. More specifically, the threshold value is a value indicating the probability that each pixel of the captured image 50 is a pixel that constitutes a mask region. The mask image generating unit 335 (described later) generates mask images 560, 580 by considering pixels whose probability of being in a mask region is equal to or greater than the threshold value received by the threshold receiving unit 334 as pixels that constitute a mask region.

[0043] 6 is a diagram showing an example of a state in which a threshold input is received by the threshold receiving unit 334. In this embodiment, a display control unit 336 (described later) causes the touch panel 310 to display a temporary projection image 54 including a temporary mask image 540, and at the same time causes the touch panel 310 to display a slider bar 70, which is a user interface for receiving a threshold input. The threshold receiving unit 334 receives an operation on the slider bar 70 as a threshold input.

[0044] First, the temporary projection image 54 will be described. In this embodiment, the temporary projection image 54 includes a temporary mask image 540, an input image 542, and a background image 544. The temporary mask image 540 is generated by a mask image generating unit 335, which will be described later. The temporary mask image 540 is an image in which pixels in the probability distribution image 52 shown in FIG. 5 that have a probability of being a mask region equal to or greater than a predetermined temporary threshold are set as the mask region. In FIG. 6, the temporary threshold is set to α2. That is, the temporary mask image 540 in FIG. 6 has pixels in the probability distribution image 52 shown in FIG. 5 that have a probability of being a mask region equal to or greater than α2, that is, the regions shown by pattern 1 and pattern 2 as the mask region. The temporary threshold may be a fixed value preset in the program 322, or may be a value specified by the user. The temporary mask image 540 is composed of lines indicating the edges of the mask region. The region surrounded by the lines is the mask region. The input image 542 is an image to be projected within the mask region. The input image 542 may be an image inputted from the outside to the processing device 330, or a monochromatic input image 542 may be generated inside the processing device 330. The background image 544 is an image that becomes the background of the temporary mask image 540, and in the example of FIG. 6, an image including the object 40 captured by the imaging device 20, that is, a captured image 50, is used. By superimposing the temporary mask image 540 and the input image 542 on the background image 544 using such a captured image 50, the user can grasp in more detail the image when actually projected by the projector 10. The background image 544 can also be said to be an image showing a state in which the input image 542 is not displayed outside the mask area.

[0045] The display control unit 336 may display the temporary projection image 54 on the touch panel 310 of the information processing device 30 as shown in Fig. 6, or may actually project the temporary projection image 54 from the projector 10 onto the object 40, for example. By actually projecting the temporary projection image 54 onto the object 40, the user can grasp, for example, the positional relationship between the object 40 and the mask area, the color balance, and the like more accurately and in detail. When the temporary projection image 54 is projected from the projector 10 onto the object 40, it is generally considered that the temporary projection image 54 is composed of a temporary mask image 540 and an input image 542, but may further include a background image 544.

[0046] Next, the slider bar 70 will be described. The slider bar 70 includes a bar 700 extending in a predetermined direction and a slider 702 movable on the bar 700. In the example of FIG. 6, the bar 700 extends in the left-right direction of the screen of the touch panel 310. The center position of the bar 700 in the left-right direction is the reference position of the slider 702. When the user moves the slider 702 to the right from the reference position, the threshold acceptance unit 334 accepts a value larger than the provisional threshold as the threshold. In this case, the user instructs to set a mask region by narrowing down the pixels whose probability of being a mask region is higher than the provisional threshold. Also, when the user moves the slider 702 to the left from the reference position, the threshold acceptance unit 334 accepts a value smaller than the provisional threshold as the threshold. In this case, the user instructs to set a mask region including pixels whose probability of being a mask region is lower than the provisional threshold.

[0047] When the temporary projection image 54 is actually projected from the projector 10 onto the object 40, for example, only the slider bar 70 may be displayed on the touch panel 310. Alternatively, together with the slider bar 70, an image of the object 40 onto which the temporary projection image 54 is projected from the projector 10, captured by the imaging device 20, may be displayed. Alternatively, a screen display as shown in FIG. 6 may be performed simultaneously with the projection from the projector 10.

[0048] The mask image generating unit 335 shown in FIG. 2 generates mask images 560, 580 for specifying the projection area of ​​the image from the projector 10, that is, the mask area. The mask images 560, 580 are exemplified in FIG. 7 and FIG. 8. More specifically, the mask image generating unit 335 generates mask images 560, 580 indicating a mask area, which is an area in the captured image 50 where the input image 542 is displayed, or a non-mask area, which is an area in the captured image 50 where the input image 542 is not displayed, by using the threshold input to the threshold receiving unit 334. The mask images 560, 580 correspond to the second image. The mask area can also be said to be an area of ​​the captured image 50 corresponding to the projection area of ​​the imaging area 44. As described above, the mask image generating unit 335 generates the mask images 560, 580 by assuming that pixels in the probability distribution image 52 generated by the probability calculation unit 333, whose probability of being a mask area is equal to or greater than the threshold received by the threshold receiving unit 334, constitute the mask area. Each time a change in the threshold is input to the threshold acceptance unit 334, the mask images 560 and 580 generated by the mask image generation unit 335 also change.

[0049] The display control unit 336 displays the projection images 56 and 58 in which the mask images 560 and 580 are superimposed on the input images 562 and 582 .

[0050] 7 and 8 are diagrams showing an example of the display mode of the projection images 56, 58 by the display control unit 336. FIG. 7 is a display example when an operation input for increasing the threshold is performed on the temporary projection image 54 shown in FIG. 6, that is, when the slider 702 is moved to the right side from the reference position. The slider 702 in FIG. 7 and FIG. 8 is displayed at a position corresponding to the threshold set in the displayed mask images 560, 580. In FIG. 7, it is assumed that the threshold is specified as α1 by the operation of the slider bar 70. In FIG. 7, the projection image 56 includes the mask image 560, the input image 562, and the background image 564 which is the captured image 50 including the object 40, and is displayed on the touch panel 310. In the mask image 560, the pixels in the probability distribution image 52 shown in FIG. 5 that have a probability of being a mask area of ​​α1 or more, that is, the area shown by the pattern 1, are masked areas. That is, the mask image 560 has a smaller area than the temporary mask image 540 in the temporary projection image 54 shown in FIG. 6.

[0051] FIG. 8 is a display example in the case where an operation input for decreasing the threshold value is performed on the temporary projection image 54 shown in FIG. 6, that is, where the slider 702 is moved to the left side from the reference position. In FIG. 8, it is assumed that the threshold value is specified to α3 by operating the slider bar 70. In FIG. 8, the projection image 58 includes a mask image 580, an input image 582, and a background image 584 which is the captured image 50 including the object 40, and is displayed on the touch panel 310. In the mask image 580 in FIG. 8, the pixels in the probability distribution image 52 shown in FIG. 5 that have a probability of being a mask region of α3 or more, that is, the regions shown by the patterns 1 to 3, are masked regions. That is, the mask image 580 has a larger area than the temporary mask image 540 in the temporary projection image 54 shown in FIG. 6.

[0052] The display control unit 336 may display the projection images 56, 58 on the touch panel 310 of the information processing device 30 as shown in Fig. 7 and Fig. 8, or may actually project the projection images 56, 58 from the projector 10 onto the object 40, for example. By actually projecting the projection images 56, 58 onto the object 40, the user can more accurately and in detail grasp, for example, the positional relationship between the object 40 and the mask region after threshold adjustment, the color balance, and the like. When the projection images 56, 58 are projected from the projector 10 onto the object 40, the projection images 56, 58 are generally considered to be composed of mask images 560, 580 and input images 562, 582, but may further include background images 564, 584.

[0053] 7 and 8, a slider bar 70 that accepts a threshold input from the user is displayed on the touch panel 310. The configuration and function of the slider bar 70 are the same as those described with reference to FIG. 6. When the user operates the slider bar 70 while the projection images 56, 58 including the mask images 560, 580 are being displayed, the mask image generation unit 335 changes the pixels to be included in the mask area according to the amount of operation of the slider bar 70. Then, the display control unit 336 causes the projection images 56, 58 including the changed mask images 560, 580 to be displayed on the touch panel 310.

[0054] Fig. 9 is a flowchart showing the flow of a display method executed by the processing device 330 of the information processing device 30 in accordance with the program 322. In Fig. 9, the display destination of the projection images 56, 58 is the touch panel 310. The processing device 330 functions as a captured image acquisition unit 331 to acquire a captured image 50 capturing an area including the object 40 from the imaging device 20 (step S200).

[0055] The processing device 330 functions as the area selection receiving unit 332 to receive designation of pixels constituting the mask area from the user (step S202). That is, the processing device 330 displays the captured image 50 acquired in step S200 on the touch panel 310, and receives drawing input on the captured image 50 from the user. The user touches a location on the captured image 50 that corresponds to the mask area to draw. Note that in step S202, the processing device 330 may receive designation of pixels constituting the non-mask area.

[0056] The processing device 330 functions as the probability calculation unit 333 to calculate the probability that each pixel of the captured image 50 is a mask area (step S204). At this time, the processing device 330 calculates the probability based on, for example, the similarity between each pixel of the captured image 50 and the pixels constituting the mask area specified in step S202.

[0057] The processing device 330 functions as the display control unit 336 to display the temporary projection image 54 including the temporary mask image 540 on the touch panel 310 (step S206). The temporary mask image 540 is an image in which pixels whose probability calculated in step S204 is equal to or greater than a predetermined temporary threshold value are used as a mask region. The processing device 330 also displays a slider bar 70 for adjusting the threshold value on the touch panel 310 together with the temporary projection image 54, as shown in FIG.

[0058] The processing device 330 functions as the threshold accepting unit 334 to accept a threshold setting operation from the user (step S208). As described above, the threshold is a value for determining pixels to be included in the mask region. The processing device 330 accepts an operation on the slider bar 70 as an input of the threshold.

[0059] The processing device 330 functions as the mask image generating unit 335 and generates the mask images 560, 580 based on the threshold value set in step S208 (step S210). That is, the processing device 330 generates the mask images 560, 580 in which pixels whose probability of being a mask region is equal to or greater than the threshold value calculated in step S208 are regarded as mask regions.

[0060] The processing device 330, functioning as the display control unit 336, causes the projection images 56, 58 including the mask images 560, 580 generated in step S210 to be displayed on the touch panel 310 (step S212). At this time, the processing device 330 also causes a slider bar 70 for resetting the threshold value to be displayed together with the projection images 56, 58.

[0061] The user looks at the projection images 56, 58 displayed on the touch panel 310 and judges whether the image projected on the object 40 is as imagined or whether further adjustment is required. For example, in step S208 or step S212, the processing device 330 may display a button on the touch panel 310 to receive an input of whether or not to complete the adjustment, and may judge whether or not to complete the adjustment based on the user's operation (step S214). As an example of the display, at least a button to be selected when completing the adjustment may be displayed. When the adjustment is to be continued, a button to be selected when continuing may be displayed, or the adjustment may be judged to be continued when an operation is performed on the slider bar 70 without displaying a button. Until the adjustment by the user is completed (step S214: NO), the processing device 330 returns to step S208 and repeats the subsequent processes. When the button instructing completion is pressed and it is judged that the adjustment by the user is completed (step S214: YES), the processing device 330 ends the process of this flowchart.

[0062] As described above, the processing device 330 of the information processing device 30 according to the embodiment executes the program 322 to function as the captured image acquisition unit 331, the region selection acceptance unit 332, the probability calculation unit 333, the threshold acceptance unit 334, the mask image generation unit 335, and the display control unit 336. The threshold acceptance unit 334 accepts input of a threshold. The mask image generation unit 335 uses the threshold to generate mask images 560, 580 indicating a mask region that is a region in which the input images 562, 582 are displayed in the captured image 50, or a non-mask region that is a region in which the input images 562, 582 are not displayed in the captured image 50. The display control unit 336 displays the projection images 56, 58 in which the mask images 560, 580 are superimposed on the input images 562, 582. Since the mask images 560, 580 are generated based on the input threshold, the number of mask images 560, 580 that can be generated is greater than when the threshold is a fixed value. Therefore, the user can easily adjust the mask area to suit his / her sensibilities.

[0063] The threshold value is a value used to determine whether or not each pixel of the captured image 50 is a pixel that constitutes a mask region. Therefore, the user can easily set the range of pixels included in the mask region.

[0064] Moreover, the threshold value is a value indicating the probability that each pixel of the captured image 50 is a pixel that constitutes the mask region. Therefore, the user can set the range of pixels included in the mask region based on the probability that each pixel is a pixel that constitutes the mask region.

[0065] Moreover, the mask region is either a predetermined portion of the mask images 560, 580 corresponding to the object 40 shown in the captured image 50, or a portion of the mask images 560, 580 other than the predetermined portion. This allows the processing device 330 to accurately distinguish between the mask region and other regions. Furthermore, the user can easily set the mask region according to the shape of the object 40.

[0066] Moreover, the region selection receiving unit 332 displays the captured image 50 and then receives input of information for selecting a mask region using the captured image 50. Furthermore, the mask image generating unit 335 extracts a mask region based on a threshold value and the information for selecting a mask region when generating the mask images 560, 580. This allows the user to directly specify the region to be included in the mask region on the captured image 50, and makes it possible to generate mask images 560, 580 that more accurately reflect the user's intention.

[0067] Furthermore, receiving input of information for selecting a mask region in the region selection receiving unit 332 includes receiving drawing input for the captured image 50, and the information for selecting a mask region includes information on pixels specified by the drawing input. This allows the user to easily and intuitively specify a portion of the captured image 50 that the user wishes to include in the mask region, thereby improving user convenience.

[0068] Furthermore, accepting an input of information for selecting a mask region in region selection accepting unit 332 includes canceling at least a part of a drawing input. This allows the user to easily cancel a drawing input that has been made incorrectly, thereby improving the convenience for the user when making a drawing input.

[0069] Furthermore, accepting input of information for selecting a mask region in the region selection accepting unit 332 includes designating that any pixel in the captured image 50 is a pixel that constitutes the mask region, or that any pixel in the captured image 50 is a pixel that constitutes the non-mask region. This makes it possible to generate mask images 560, 580 by designating pixels not only in the mask region but also in the non-mask region, thereby improving user convenience.

[0070] 2 .Modifications Each of the above-mentioned exemplary embodiments may be modified in various ways. Specific modified embodiments that may be applied to each of the above-mentioned embodiments are illustrated below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory. By visually checking the first lattice point 147a whose display mode has been changed, the user of the display control device 40 can confirm whether the first lattice point 147a is the desired lattice point 147. By confirming that the first lattice point 147a is the desired lattice point 147, the user can confirm that the cursor 200 is located at the desired position.

[0071] In this embodiment, the processing device 330 displays the slider bar 70 as an interface for accepting the threshold value designation from the user, but the interface is not limited to this. For example, two buttons, one of which corresponds to an increase in the threshold value and the other to a decrease in the threshold value, may be the interface. In this case, the processing device 330 increases the threshold value by a predetermined value each time the user presses the button associated with an increase in the threshold value. Also, the processing device 330 decreases the threshold value by a predetermined value each time the user presses the button associated with a decrease in the threshold value. The two buttons may be marked with marks that indicate opposite directions, such as triangles, or the buttons themselves may be shaped to indicate opposite directions. This allows the user to operate intuitively. Also, the interface may be, for example, a form in which the user can input a numerical value. In this case, the user inputs a numerical value that specifies the threshold value. Also, an interface may be provided that accepts the designation of the threshold value by voice input, for example.

[0072] In addition, in this embodiment, the purpose is to set the surface 400 of the object 40 as the mask region, and the mask region is set by extracting a portion of the captured image 50 captured by the imaging device 20 that has a probability of corresponding to the surface 400 equal to or greater than a threshold value. That is, the mask region is a predetermined portion identified by the object 40 shown in the captured image 50. However, for example, a region other than the predetermined portion identified by the object 40 shown in the captured image 50, such as the wall surface 42 on which the object 40 is attached, may be specified as the mask region. In this case, the user may select the mask region designation section 660 of the region type designation section 66 shown in FIG. 4 and then draw in a portion corresponding to the wall surface 42 of the captured image 50, or may select the non-mask region designation section 662 and then draw in a portion corresponding to the surface 400 of the captured image 50. [Explanation of symbols]

[0073] 1...projection system, 10...projector, 20...imaging device, 30...information processing device, 40...object, 42...wall surface, 44...imaging area, 300...communication device, 310...touch panel, 320...storage device, 322...program, 330...processing device, 331...captured image acquisition unit, 332...area selection reception unit, 333...probability calculation unit, 334...threshold reception unit, 335...mask image generation unit, 336...display control unit.

Claims

1. Accepting an input of a threshold value; generating a second image using the threshold value to indicate a masked region of the first image that displays an input image or a non-masked region of the first image that does not display the input image; displaying an image for projection in which the second image is superimposed on the input image; and Including, The threshold value is a value used to determine whether or not each pixel of the first image is a pixel that constitutes the mask region, and is a value indicating a probability that each pixel of the first image is a pixel that constitutes the mask region. Display method.

2. The mask area is either a predetermined portion of the second image corresponding to an object shown in the first image, or a portion of the second image other than the predetermined portion. The display method according to claim 1.

3. Displaying the first image; and receiving an input of information for selecting the mask region using the first image; Producing the second image includes: extracting the mask region based on the threshold and information for selecting the mask region. The display method according to any one of claims 1 to 2.

4. Receiving input of information for selecting the mask region receiving a drawing input for the first image; the information for selecting the mask region includes information on a pixel designated by the drawing input; The display method according to claim 3.

5. Receiving input of information for selecting the mask region canceling at least a portion of the drawing input. The display method according to claim 4.

6. Receiving input of information for selecting the mask region Specifying that any pixel of the first image is a pixel constituting the mask region, or that any pixel of the first image is a pixel constituting the non-mask region, The display method according to any one of claims 3 to 5.

7. Receiving an input of a threshold value; generating a second image using the threshold value to indicate a masked region of the first image that displays an input image or a non-masked region of the first image that does not display the input image; displaying an image for projection in which the second image is superimposed on the input image; and Displaying the first image; Receiving an input of information for selecting the mask region using the first image; Including, Producing the second image includes: extracting the mask region based on the threshold and information for selecting the mask region; Receiving input of information for selecting the mask region Specifying pixels corresponding to an object appearing in the first image or pixels corresponding to a portion other than the object appearing in the first image. Display method.

8. Receiving an input of a threshold value; generating a second image using the threshold value to indicate a masked region of the first image that displays an input image or a non-masked region of the first image that does not display the input image; displaying an image for projection in which the second image is superimposed on the input image; and Displaying the first image; Receiving an input of information for selecting the mask region using the first image; Including, Producing the second image includes: extracting the mask region based on the threshold and information for selecting the mask region; Receiving input of information for selecting the mask region receiving a drawing input for the first image; the information for selecting the mask region includes information on a pixel designated by the drawing input; Display method.

9. On the computer, Accepting an input of a threshold value; generating a second image using the threshold value to indicate a masked region of the first image that displays an input image or a non-masked region of the first image that does not display the input image; displaying an image for projection in which the second image is superimposed on the input image; and Run the command, The threshold value is a value used to determine whether or not each pixel of the first image is a pixel that constitutes the mask region, and is a value indicating a probability that each pixel of the first image is a pixel that constitutes the mask region. program.

10. A computer comprising: Accepting an input of a threshold value; generating a second image using the threshold value to indicate a masked region of the first image that displays an input image or a non-masked region of the first image that does not display the input image; displaying an image for projection in which the second image is superimposed on the input image; and Displaying the first image; Receiving an input of information for selecting the mask region using the first image; Run the command, Producing the second image includes: extracting the mask region based on the threshold and information for selecting the mask region; Receiving input of information for selecting the mask region receiving a drawing input for the first image; the information for selecting the mask region includes information on a pixel designated by the drawing input; program.

Images