Display method and recording medium

Abstract

A display method and a recording medium are provided to facilitate adjustment of a mask area. The display method includes: receiving input of first information indicating the amount of expansion or contraction processing for a first mask image including the mask area; performing the expansion or contraction processing corresponding to the first information on the first mask image with the edge of the mask area as a reference to generate a second mask image; and displaying a projection image obtained by superimposing the second mask image on the input image.

Description

Technical Field

[0001] The present invention relates to a display method and a recording medium. Background Art

[0002] Patent document 1 discloses a masked image including a masked area and a non-masked area. If the masked image is superimposed on an input image, the portion superimposed with the masked area is masked and cannot be seen, while the portion superimposed with the non-masked area is not masked and can be seen. Patent document 1 discloses the following technology: for a first masked processed image having a non-masked area of ​​a first size, an enlargement and reduction process is performed to generate a second masked processed image in which the non-masked area is enlarged or reduced from the first size to a second size.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2019-164191

[0004] In the image enlargement and reduction process, the size can be changed with reference to a predetermined point such as the lower left corner of the original image while maintaining the original shape of the image. However, if the enlargement and reduction process is performed on an image whose positional relationship with the input image etc. is problematic, such as a masked image, the positional relationship between the input image etc. and the masked image after the enlargement and reduction process may deviate. Therefore, the shape may not be what the user expects. Summary of the invention

[0005] A display method according to one embodiment of the present invention includes: accepting input of first information, wherein the first information indicates the amount of expansion processing or contraction processing for a first masking image including a masking area; performing the expansion processing or contraction processing corresponding to the first information on the first masking image based on the edge of the masking area to generate a second masking image; and displaying a projection image obtained by overlapping the second masking image with the input image.

[0006] A program of one embodiment of the present invention causes a computer to execute: accepting input of first information, the first information indicating the amount of expansion processing or contraction processing for a first masking image including a masking area; performing the expansion processing or contraction processing corresponding to the first information on the first masking image with an edge of the masking area as a reference to generate a second masking image; and displaying a projection image obtained by superimposing the second masking image on an input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a schematic diagram showing an example of a projection system according to an embodiment.

[0008] Figure 2 It is a block diagram showing the functional structure of the processing device in the first embodiment.

[0009] Figure 3 This is a diagram showing an example of the manner in which the operation accepting unit accepts the expansion and contraction amount.

[0010] Figure 4 This is a diagram showing an example of the manner in which the operation accepting unit accepts the expansion and contraction amount.

[0011] Figure 5 It is a schematic diagram for explaining the expansion process and the contraction process.

[0012] Figure 6 It is a schematic diagram for explaining the expansion process and the contraction process.

[0013] Figure 7 It is a schematic diagram for explaining the expansion process and the contraction process.

[0014] Figure 8 This is an example of a display method of the second mask image by the display control unit.

[0015] Fig. 9 This is an example of a display method of the second mask image by the display control unit.

[0016] Fig.10 This is a diagram showing an example of an effect specifying screen.

[0017] Fig.11 This is a diagram showing an example of an effect specifying screen.

[0018] Fig.12 This is a flowchart showing the flow of the projection method executed by a processing device of an information processing device according to a program.

[0019] Fig.13 It is a block diagram showing the functional structure of the processing device in the second embodiment.

[0020] Fig.14 This is a diagram showing an example of a method of accepting the zoom-in / zoom-out amount by the operation accepting unit.

[0021] Fig.15 This is an example of a display method of the third mask image by the display control unit.

[0022] Fig.16 This is an example of a display method of the third mask image by the display control unit.

[0023] Description of symbols

[0024] 1: projection system; 10: projector; 20: shooting device; 30: information processing device; 40: object; 42: wall; 300: communication device; 310: touch panel; 320: storage device; 322: program; 330: processing device; 331: first shielding image generating unit; 332: operation accepting unit; 333: second shielding image generating unit; 334: display control unit; 335: effect adding unit; 336: third shielding image generating unit; ED: edge. DETAILED DESCRIPTION

[0025] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in the accompanying drawings, the size or scale of each part is appropriately different from the actual one, and there are also parts schematically shown for easy understanding. In addition, the scope of the present invention is not limited to these modes unless the description of the present invention is particularly limited in the following description.

[0026] 1. First Implementation

[0027] Figure 1 1 is a schematic diagram showing an example of a projection system 1 according to an embodiment. Figure 1 As shown, the projection system 1 includes a projector 10 , a photographing device 20 and an information processing device 30 . Figure 1 The projection system 1 shown is a system for projecting an image onto an object 40 having a three-dimensional shape. In the present embodiment, the object 40 is formed by cutting a plate-like member having a predetermined thickness into the shape of the letter "E". The object 40 has an unillustrated adhesive surface that contacts the wall surface 42, a surface 400 that is the surface on the opposite side of the adhesive surface, and a side surface 402 that connects the adhesive surface and the surface 400. In the present embodiment, the surface 400 of the object 40 is set as the projection area of ​​the image. The projection area of ​​the image from the projector 10 corresponds to the shielding area described later.

[0028] The projector 10 includes a liquid crystal light valve, a projection lens, and a liquid crystal drive unit for projecting a projection image 50 described later. In addition, the projector 10 includes an ultra-high pressure mercury lamp or a metal halide lamp as a light source. The projector 10 is connected to the information processing device 30 in a communicable manner, for example, via a cable. The projector 10 obtains projection image data from the information processing device 30 through communication via the cable. The projector 10 projects the projection image 50 represented by the obtained projection image data onto the shielded area. Hereinafter, the shielded area is an area where the input image 502 described later is displayed, and the area outside the shielded area is an area where the input image 502 is not displayed, that is, an area that is not shielded. In the present embodiment, the communication between the projector 10 and the information processing device 30 is, for example, wired communication in accordance with standards such as Ethernet and USB (Universal Serial Bus). However, the communication between the projector 10 and the information processing device 30 may also be wireless communication in accordance with standards such as Wi-Fi. In addition, Wi-Fi and Ethernet are registered trademarks.

[0029] The photographing device 20 is, for example, a camera including a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) as a photographing element that converts the focused light into an electrical signal. In the following, for the sake of simplicity, it is assumed that the photographing device 20 photographs a still image. In addition, the photographing device 20 may also photograph a dynamic image instead of a still image. The photographing device 20 photographs an image of a photographing area 44 including an object 40. In the present embodiment, the photographing area 44 includes the object 40 and a wall surface 42. The photographing device 20 is connected to the information processing device 30 in a communicative manner, for example, via a cable, similarly to the projector 10. The photographing device 20 sends photographic image data representing the photographed image to the information processing device 30. In the present embodiment, the communication between the photographing device 20 and the information processing device 30 is, for example, wired communication in accordance with standards such as Ethernet and USB, but may also be wireless communication in accordance with standards such as Wi-Fi. In addition, in the present embodiment, the imaging device 20 is separate from the information processing device 30 and the projector 10 , but the imaging device 20 may be mounted on either the information processing device 30 or the projector 10 .

[0030] The information processing device 30 is an example of an electronic device, such as a personal computer. Figure 1As shown, the information processing device 30 includes a communication device 300, a touch panel 310, a storage device 320, and a processing device 330. The communication device 300 is connected to the projector 10 via a cable. In addition, the imaging device 20 is connected to the communication device 300 via a cable. The communication device 300 receives the captured image data transmitted from the imaging device 20. In addition, the communication device 300 transmits projection image data representing an image projected onto the object 40 to the projector 10 under the control of the processing device 330.

[0031] The touch panel 310 is a device that integrates a display device that displays various information and an input device for the user to input information. The input device is, for example, a transparent sheet-shaped contact sensor. The input device is set in a manner that covers the display surface of the display device. The input device uses an electrostatic capacitance determined by an object in contact with the input device and the input device to detect the touch position. The input device outputs data representing the detected touch position to the processing device 330. Thus, the user's operation content on the touch panel 310 is transmitted to the processing device 330. In the present embodiment, the touch panel 310 serves as both a display device and an input device, but the display device and the input device may also be provided separately. Specifically, the information processing device 30 may also have a display as a display device, and a keyboard and a mouse as input devices.

[0032] The storage device 320 is a recording medium readable by the processing device 330. The storage device 320 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory).

[0033] A program 322 to be executed by the processing device 330 is pre-stored in the non-volatile memory of the storage device 320. The volatile memory of the storage device 320 is used by the processing device 330 as a work area when executing the program 322. The program 322 may also be referred to as an "application program", "application software" or "application". The program 322 is obtained from a server (not shown) or the like via the communication device 300, for example, and then stored in the storage device 320.

[0034] The processing device 330 is configured to include, for example, a processor such as a CPU (Central Processing Unit), that is, a computer. The processing device 330 may be configured by a single computer or by a plurality of computers. The processing device 330 reads the program 322 from the non-volatile memory to the volatile memory when an operation is performed on the input device to instruct the start of execution of the program 322. The processing device 330 executes the program 322 read from the volatile memory.

[0035] Figure 2 is a block diagram showing the functional structure of the processing device 330. The processing device 330 operates according to the program 322 as Figure 2 The first shielding image generating unit 331 , the operation accepting unit 332 , the second shielding image generating unit 333 , the display controlling unit 334 , and the effect adding unit 335 shown function. Figure 2 The first shielding image generating unit 331 , the operation accepting unit 332 , the second shielding image generating unit 333 , the display controlling unit 334 , and the effect adding unit 335 shown are software modules realized by causing the processing device 330 to operate according to the program 322 .

[0036] The first shielding image generating unit 331 generates a first shielding image 500 for determining a projection area of ​​an image from the projector 10, that is, a shielding area. Figure 3 In the present embodiment, the first shielded image generation unit 331 uses a captured image obtained by capturing a captured area 44 including an object 40 to generate a first shielded image 500 having an edge ED corresponding to the contour of the surface 400 of the object 40 in the captured image. More specifically, the first shielded image generation unit 331 obtains a captured image obtained by capturing a captured area 44 including an object 40 from the imaging device 20. The first shielded image generation unit 331 determines an area corresponding to the surface 400 of the object 40 in the captured image, and generates a first shielded image 500 indicating the edge ED of the area.

[0037] Various methods are known for determining a predetermined area in an image. For example, a method is known in which the processing device 330 performs threshold processing on the color and depth values ​​of each pixel in the image to determine a specific area. In addition, a method is known in which, for example, the touch panel 310 displays a captured image including the object 40, and the user manually draws a mask area on the touch panel 310 while observing the captured image. The generation of the first mask image 500 by the first mask image generation unit 331 can use any of these methods.

[0038] In the present embodiment, the information processing device 30 generates the first shielded image 500, but the present invention is not limited thereto. For example, the information processing device 30 may obtain the first shielded image 500 generated by another information processing device through communication, etc. In addition, for example, the first shielded image 500 generated by the first shielded image generating unit 331 in the past may be stored in the storage device 320, and the processing device 330 may read it out.

[0039] The operation accepting unit 332 accepts input of information indicating the amount of expansion or contraction processing for the first shielding image 500 including the shielding area. Hereinafter, the expansion or contraction processing is referred to as "expansion or contraction processing", the amount of the expansion or contraction processing is referred to as "expansion or contraction amount", and the information indicating the amount of the expansion or contraction processing is referred to as "expansion or contraction amount information". The "expansion or contraction amount information" corresponds to the first information.

[0040] Here, the expansion process is a process of moving the edge ED of the masked area outward, and the contraction process is a process of moving the edge ED of the masked area inward. The size of the masked image after the expansion and contraction process changes along the edge ED, so the shape changes from the original image. In contrast, the enlargement and reduction process is a process of extending or shortening the edge ED of the masked area by a specified magnification. The masked image after the enlargement and reduction process changes only in size while maintaining the shape of the original image with a predetermined point as a reference.

[0041] Figure 3 3 is a diagram showing an example of a method of accepting the expansion and contraction amount by the operation accepting unit 332. In the present embodiment, the display control unit 334 described later displays the slide bar 52 as a user interface for accepting the designation of the expansion and contraction amount on the touch panel 310 while the projection image 50 including the first shielding image 500 is displayed on the touch panel 310. The operation accepting unit 332 accepts the operation amount on the slide bar 52 as the expansion and contraction amount.

[0042] First, the projection image 50 will be described. In the present embodiment, the projection image 50 includes at least a first mask image 500 and an input image 502 . Figure 3 The projection image 50 illustrated in the figure includes a first shielding image 500, an input image 502, and a background image 504. The first shielding image 500 is composed of lines representing the edge ED of the shielding area. The area surrounded by the lines is the shielding area. The input image 502 is an image projected into the shielding area. The input image 502 is set to be non-displayed in the area outside the shielding area. The input image 502 can be an image input to the processing device 330 from the outside, or a monochrome input image 502 can be generated inside the processing device 330. The background image 504 is an image that becomes the background of the input image 502 whose display range is defined by the edge ED of the first shielding image 500. Figure 3 In the example of , the background image 504 is a monochrome image of a color different from that of the input image 502. The background image 504 can also be said to be an image showing a state where the input image 502 is not displayed outside the masked area.

[0043] In the following, the drawings and the specification sometimes illustrate Figure 3 The projection image 50, the first shielding image 500, the input image 502, and the background image 504 are different in form, but are not limited to the specific forms. When the names of the images are described as a general term, the numbers of the projection image 50, the first shielding image 500, the input image 502, and the background image 504 are also used. In particular, the projection image 50 includes the projection image 54 including the second shielding image 540 described later.

[0044] Figure 4 FIG. 2 is a diagram showing another example of the method of accepting the expansion and contraction amount of the operation accepting unit 332. Figure 4 In the example of , the projection image 53 includes a first mask image 530 indicating an edge ED of the mask area, an input image 532 , and a background image 534 . Figure 4 The first mask image 530 and the input image 532 are Figure 3 The first mask image 500 is the same as the input image 502. On the other hand, Figure 4 The background image 534 is an image obtained by photographing a portion where the projector 10 performs projection, for example, an image including the object 40 and the wall surface 42 photographed by the camera 20. By superimposing the first shielding image 530 and the input image 532 on such a background image 534, the user can understand the image when the projector 10 is actually projecting in more detail.

[0045] In addition, the display control unit 334 Figure 3 , Figure 4 In addition to displaying the projection image 50 on the touch panel 310 of the information processing device 30, for example, the projection image 50 may be actually projected from the projector 10 onto the object 40. By actually projecting the projection image 50 onto the object 40, the user can more accurately and in detail grasp, for example, the positional relationship between the object 40 and the shielded area, the balance of tones, etc. When the projection image 50 is projected from the projector 10 onto the object 40, it is generally considered that the projection image 50 is composed of the first shielded image 500 and the input image 502, but it may further include a background image 504.

[0046] Next, the slide bar 52 will be described. The slide bar 52 includes a bar 520 extending in a predetermined direction and a slider 522 that can move on the bar 520. Figure 3 as well as Figure 4In the example of , the bar 520 extends in the left-right direction of the screen of the touch panel 310. The center position of the bar 520 in the left-right direction is the reference position of the slider 522. The right side of the bar 520 is the expansion operation area, and the left side is the contraction operation area. When the user moves the slider 522 from the reference position to the right, the operation receiving unit 332 receives the movement distance of the slider 522 as the amount of expansion processing. When the user moves the slider 522 from the reference position to the left, the operation receiving unit 332 receives the movement distance of the slider 522 as the amount of contraction processing. That is, the operation receiving unit 332 converts the movement distance of the slider 522 into the above-mentioned expansion and contraction amount information.

[0047] When the projection image 50 is actually projected from the projector 10 onto the object 40, for example, only the slide bar 52 may be displayed on the touch panel 310. Alternatively, an image obtained by photographing the object 40 projected with the projection image 50 from the projector 10 by the photographing device 20 may be displayed together with the slide bar 52. Alternatively, Figure 3 , Figure 4 Such screen display may be performed simultaneously with the projection from the projector 10 .

[0048] Figure 2 The second mask image generating unit 333 shown in FIG. 3 performs an expansion process or a contraction process on the first mask image 500 according to the expansion and contraction amount information based on the edge ED of the first mask image 500 to generate a second mask image 540. Figure 8 shown. Figure 5 to Figure 7 is a schematic diagram for explaining the expansion process and the contraction process. Figure 5 to Figure 7 In the example, from the perspective of visual recognition, the inner side of the shielded area is indicated by a shadow. Figure 5 to Figure 7 Each grid of corresponds to, for example, one pixel.

[0049] Figure 5 The second mask image generating unit 333 generates a mask image in the vicinity of the edge ED of the first mask image 500. Figure 5 When the first mask image 500 shown in FIG. Figure 6 The position of the edge ED is moved to the outside of the shielded area as shown. That is, in the present embodiment, the dilation processing is a processing for moving the edge ED of the shielded image to the outside of the shielded area by the amount of pixels corresponding to the dilation and contraction amount information. Moving the position of the edge ED to the outside means including the adjacent pixels to the pixels including the edge ED of the first shielded image 500 in the shielded area when the adjacent pixels are not included in the shielded area. In this case, the adjacent pixels include pixels whose vertices touch each other.

[0050] In addition, the second mask image generating unit 333 Figure 5When the first mask image 500 shown in FIG. Figure 7 The position of the edge ED is moved to the inside of the shielded area as shown. That is, the shrinking process is a process of moving the edge ED of the shielded image to the inside of the shielded area by the amount of pixels corresponding to the expansion and contraction amount information. Moving the position of the edge ED to the inside means that even if there is one pixel not included in the shielded area among the pixels adjacent to the pixel including the edge ED of the first shielded image 500, the pixel including the edge ED is excluded from the shielded area. The adjacent pixels in this case also include pixels whose vertices are connected to each other.

[0051] The second mask image generating unit 333 performs an expansion process or a contraction process of an amount corresponding to the expansion and contraction amount information received by the operation receiving unit 332. Specifically, the second mask image generating unit 333 determines the number of pixels by which the edge ED is moved in the expansion process or the contraction process according to the amount of movement of the slider 522 by the user. The larger the amount of movement of the slider 522 by the user, the larger the number of pixels by which the edge ED is moved in the expansion process or the contraction process.

[0052] Figure 2 The display control unit 334 shown displays the projection image 50 in which the second shielding image 540 is superimposed on the input image 532. At this time, the operation receiving unit 332 may also receive input of information indicating the amount of expansion or contraction processing for the second shielding image 540. That is, the user may be able to adjust the position of the edge ED of the shielding region again while observing the second shielding image 540 displayed by the display control unit 334.

[0053] Figure 8 as well as Fig. 9 This is an example of the display method of the second mask image 540 by the display control unit 334 . Figure 8 It includes Figure 4 FIG. 5 is a display example of a projection image 54 of a second mask image 540 generated by dilating a first mask image 530. Figure 8 In the figure, the projection image 54 includes the second shielding image 540, the input image 542, and the background image 544 which is a captured image including the object 40, and is displayed on the touch panel 310. The edge ED2 of the second shielding image 540 after the expansion process moves to the outside of the edge ED1 of the first shielding image 530 shown by the dotted line.

[0054] In the following drawings and the specification, examples are sometimes shown. Figure 8The second masking image 540 and the projection image 54 including the second masking image 540 may be in different forms, but are not limited to individual specific forms. When recording the names of the images as a general term, the labels of the second masking image 540 and the projection image 54 including the second masking image 540 are also used.

[0055] Fig. 9 It includes Figure 4 FIG. 5 is a display example of a projection image 55 of a second mask image 550 generated by shrinking the first mask image 530. Fig. 9 In the figure, the projection image 55 includes the second shielding image 550, the input image 552, and the background image 554 which is a captured image including the object 40, and is displayed on the touch panel 310. The edge ED2 of the second shielding image 540 after the contraction process is moved to the inner side than the edge ED1 of the first shielding image 530.

[0056] like Figure 8 , Fig. 9 As shown, in addition to displaying the projection image 54 on the touch panel 310 of the information processing device 30, the display control unit 334 may actually project the projection image 54 from the projector 10 to the object 40, for example. By actually projecting the projection image 54 onto the object 40, the user can more accurately and in detail grasp, for example, the positional relationship between the object 40 and the shielding area after expansion and contraction, the balance of color tones, etc. When the projection image 54 is projected from the projector 10 onto the object 40, it is generally considered that the projection image 54 is composed of the second shielding image 540 and the input image 542, but it may also include a background image 544.

[0057] That is, displaying the projection image 54 by the display control unit 334 includes: using the projector 10 to project the projection image 54 to the area including the object 40; and causing the touch panel 310 of the information processing device 30 to display the projection image 54. Figure 8 as well as Fig. 9 As illustrated, when the projection image 54 is displayed on the touch panel 310 of the information processing device 30 , the projection image 54 is displayed so as to be superimposed on a captured image of a region including the object 40 .

[0058] In addition, Figure 8 as well as Fig. 9 In the embodiment, a slide bar 52 for accepting input of expansion and contraction amount information from a user is displayed on the touch panel 310. The structure, function and use of the slide bar 52 Figure 3 and Figure 4 The structure and function of the instructions are the same. Figure 8 and Fig. 9The slider 522 is displayed at a position corresponding to the expansion and contraction amount applied to the displayed second shielding image 540. When the user operates the slider 52 during the display of the projection image 54 including the second shielding image 540, the second shielding image generating unit 333 performs expansion and contraction processing on the second shielding image 540 by an amount corresponding to the operation amount of the slider 52. Then, the display control unit 334 displays the projection image 50 including the shielding image after the expansion and contraction processing.

[0059] Figure 2 The effect adding unit 335 shown adds an effect specified by the user to the projection image 50. An effect is an image processing that produces a certain visual effect for a person viewing the image. When the user specifies the addition of an effect, the effect adding unit 335 performs image processing corresponding to the specified effect on the projection image 50. The display control unit 334 displays the projection image 50 to which the image processing by the effect adding unit 335 has been performed. The user can specify the addition of an effect at any timing. For example, Figure 3 or Figure 4 In addition to the timing designation effect of displaying the projection image 50 including the first shielding image 500 on the touch panel 310, it is also possible to perform the following steps. Figure 8 or Fig. 9 Thus, the timing designation effect is added when the projection image 54 including the second mask image 540 is displayed on the touch panel 310 .

[0060] As an example of a specific effect, for example, the input image 502 is displayed only in a part of the masked area, the display of the input image 502 in a part of the masked area is stopped, etc. In addition, for example, the input image 502 moves in the masked area, the input image 502 is replaced, and the color of the input image 502 changes, etc. are also examples of effects.

[0061] Fig.10 and Fig.11 This is a diagram showing an example of an effect specifying screen. Fig.10 as well as Fig.11 To Figure 4 The projection image 53 shown in the figure is used as an example to specify the addition of an effect. Fig.10 and Fig.11 In the example of the effect selection menu 60, the effect selection menu 60 is displayed on the touch panel 310. In the effect selection menu 60, two effects are provided as examples of selectable effects: "1. Make the inner side of the edge glow" and "2. Make the outer side of the edge glow". When the user selects "1. Make the inner side of the edge glow", Fig.10 As shown in FIG. 1 , a projection image 62 is displayed in which image processing is performed to emit light from the edge ED of the first shielding image 530 to a predetermined distance inside the shielding region. In addition, when the user selects “2. Make the outer side of the edge emit light”, as shown in FIG. Fig.11 As shown, a projection image 64 is displayed in which image processing has been performed to emit light from the edge ED of the first shielding image 530 toward a predetermined distance outside the shielding region.

[0062] In use Fig.10 , Fig.11 In the case of the effect based on the edge ED of the shielding area as shown in the example, there may be a need to separate the edge of the object 40 and the edge ED of the shielding area at equal intervals. In the enlargement and reduction process, only the size of the shielding image is changed while maintaining the shape, so it is impossible to meet the above needs, for example, it is necessary to re-create the shielding image from the beginning. By enabling expansion and contraction processing based on the edge ED as in this embodiment, the shielding area can be easily adjusted according to the user's sensibility, and the degree of freedom of the image projection method on the object 40 can be increased.

[0063] Fig.12 3 is a flowchart showing the flow of the projection method executed by the processing device 330 of the information processing device 30 according to the program 322. Fig.12 , the display destination of the projection image 50 is the touch panel 310. The processing device 330 obtains a captured image obtained by capturing an area including the object 40 from the imaging device 20 (step S200). The processing device 330 functions as a first shielding image generating unit 331, thereby generating a first shielding image 500 using the captured image obtained in step S200 (step S202). The first shielding image 500 has an edge ED corresponding to the outline of the object 40 in the captured image. The processing device 330 functions as a display control unit 334, thereby displaying the projection image 50 including the first shielding image 500 generated in step S202 on the touch panel 310 (step S204).

[0064] The processing device 330 functions as the effect adding unit 335, thereby accepting the designation of an effect from the user, and when the designation of an effect is accepted, the projection image 50 subjected to image processing corresponding to the effect is displayed on the touch panel 310 (step S206). In addition, when it is not necessary to add an effect, step S206 may not be executed.

[0065] The processing device 330 functions as the operation accepting unit 332, thereby accepting the expansion and contraction operation of the first shielding image 500 from the user (step S208). That is, the processing device 330 displays the projection image 50 including the first shielding image 500 and the slide bar 52 for accepting the expansion and contraction operation on the touch panel 310, and accepts the input of the expansion and contraction amount information for the first shielding image 500. When the expansion and contraction operation from the user is accepted, the processing device 330 functions as the second shielding image generating unit 333, thereby performing the expansion and contraction processing on the first shielding image 500 to generate the second shielding image 540 (step S210). That is, the processing device 330 performs the expansion processing or contraction processing according to the expansion and contraction amount information on the first shielding image 500 with the edge ED of the first shielding image 500 as a reference, and generates the second shielding image 540.

[0066] The processing device 330 functions as a display control unit 334, thereby displaying the projection image 54 including the second shielding image 540 generated in step S210 on the touch panel 310 (step S212). Furthermore, the processing device 330 functions as an effect adding unit 335, thereby displaying the projection image 50 subjected to image processing corresponding to the effect on the touch panel 310 when the designation of the effect is accepted in step S206 (step S214). It is also possible to accept a change of the effect in step S214, and when no effect is specified in step S206, it is also possible to accept a new designation of the effect in step S214. In addition, when it is not necessary to add an effect, step S214 may not be executed.

[0067] The user observes the projection image 54 displayed on the touch panel 310 and determines whether the image projected on the object 40 is as expected or needs further adjustment. The processing device 330 may also display a button for accepting input of whether the adjustment is completed on the touch panel 310 in step S212 or step S214, and determine whether the adjustment is completed based on the user's operation (step S216). As an example of display, at least the button selected in the case of completion is displayed. In the case of continuing the adjustment, the button selected in the case of continuing may be displayed, or the button may not be displayed. In the case of an operation on the slide bar 52, it is determined that the adjustment is continued. Before the user's adjustment is completed (step S216: No), the processing device 330 returns to step S208 and repeats the subsequent processing. If the button indicating completion is pressed, it is determined that the user's adjustment is completed (step S216: Yes), and the processing device 330 ends the processing of this flowchart.

[0068] As described above, the processing device 330 of the information processing device 30 of the first embodiment functions as the first shield image generation unit 331, the operation receiving unit 332, the second shield image generation unit 333, the display control unit 334, and the effect adding unit 335 by executing the program 322. The operation receiving unit 332 receives input of the expansion and contraction amount information for the first shield image 500 including the shield area. The second shield image generation unit 333 performs expansion processing or contraction processing on the first shield image 500 according to the expansion and contraction amount information based on the edge ED of the shield area, and generates the second shield image 540. The display control unit 334 displays the projection image 54 in which the second shield image 540 is superimposed on the input image 542. The second shield image 540 is generated by expanding and contracting the first shield image 500 by the expansion and contraction amount based on the edge ED of the first shield image 500, so the position of the second shield image 540 and the position of the first shield image 500 are not offset by the image processing. Therefore, the mask area can be easily adjusted in accordance with the user's sensibility.

[0069] In addition, the expansion processing performed by the second mask image generating unit 333 is a processing to move the edge ED to the outside of the mask area by the amount of pixels corresponding to the expansion and contraction amount information, and the contraction processing is a processing to move the edge ED to the inside of the mask area by the amount of pixels corresponding to the expansion and contraction amount information. Thus, the user can easily perform the operation of expanding or reducing the mask area based on the edge ED.

[0070] Furthermore, the first mask image generator 331 generates a first mask image 500 having an edge ED corresponding to the contour of the object 40 in the captured image using a captured image of a region including the object 40. Thus, the user can easily obtain the first mask image 500 matching the shape of the object 40.

[0071] In addition, the display control unit 334 displays the projection image 50, including at least one of projecting the projection image 50 onto the area including the object 40 using the projector 10 and displaying the projection image 50 on the touch panel 310 of the information processing device 30. When the projection image 50 is projected using the projector 10, the user can accurately and in detail understand, for example, the positional relationship between the object 40 and the shielding area, the balance of color tones, etc. In addition, when the projection image 50 is displayed on the touch panel 310 of the information processing device 30, the user can also confirm the second shielding image 540 after the expansion and contraction processing, for example, at a place far away from the object 40 and the projector 10.

[0072] In addition, when the display control unit 334 causes the touch panel 310 of the information processing device 30 to display the projection image 50, the projection image 50 may be displayed overlapping with the captured image of the area including the object 40. Thus, when the user causes the touch panel 310 to display the projection image 50 for confirmation, the user can more clearly understand the projection state of the input image 502 onto the object 40.

[0073] 2. Second Implementation

[0074] In the following exemplary embodiments, elements having the same functions or effects as those of the first embodiment are assigned the same reference numerals as those used in the first embodiment, and detailed descriptions thereof are appropriately omitted.

[0075] The second embodiment is an embodiment in which the first masked image 500 or the second masked image 540 can be magnified or reduced in addition to the expansion and contraction processing. As described above, the magnification and reduction processing is a processing to extend or shorten the edge ED of the masked area at a specified magnification. The magnification processing is a processing to interpolate pixels between pixels of the image of the processing object, and the reduction processing can also be said to be a processing to thin out the pixels of the image of the processing object. The masked image after the magnification and reduction processing is based on a specified point, and only the size changes while maintaining the shape of the original image. That is, the image after the magnification and reduction processing becomes a shape similar to the image before the processing. Hereinafter, the first masked image 500 or the second masked image 540 to be the processing object will be referred to as the "processing object masked image". In addition, in Figure 14 to Figure 16 In the figures, the symbol ED represents the edge of the masked image in each figure.

[0076] Fig.13 322, in addition to the first shielded image generating unit 331, the operation accepting unit 332, the second shielded image generating unit 333, the display control unit 334, and the effect adding unit 335, which are the same functional units as those in the first embodiment, the processing device 330 that operates according to the program 322 also functions as a third shielded image generating unit 336. The third shielded image generating unit 336 is also a software module realized by causing the processing device 330 to operate according to the program 322, as is the case with the other functional units.

[0077] In the second embodiment, the operation accepting unit 332 accepts input of information indicating the amount of enlargement or reduction processing for the processing target shielded image 700. Hereinafter, the enlargement or reduction processing is referred to as "enlargement or reduction processing", the amount of the enlargement or reduction processing is referred to as "enlargement or reduction amount", and the information indicating the amount of the enlargement or reduction processing is referred to as "enlargement or reduction amount information". The "enlargement or reduction amount information" corresponds to the second information.

[0078] Fig.14 3 is a diagram showing an example of a method of accepting the enlargement / reduction amount by the operation accepting unit 332. In the second embodiment, the display control unit 334 displays the slide bar 52 for accepting the designation of the expansion / reduction amount and the slide bar 72 for accepting the designation of the enlargement / reduction amount while the touch panel 310 displays the projection image 70 including the processing target shielding image 700. The projection image 70 includes the processing target shielding image 700, the input image 702, and the background image 704.

[0079] The slide bar 72 includes a bar 720 extending in a predetermined direction and a slider 722 movable on the bar 720, similarly to the slide bar 52. Fig.14 In the example of , the bar 720 extends in the left-right direction of the screen, the right side is the zoom-in operation area, and the left side is the zoom-out operation area. The center position of the bar 720 in the left-right direction is the reference position of the slider 722. When the user moves the slider 722 to the right, the operation receiving unit 332 receives the movement distance of the slider 722 as the amount of zoom-in processing. When the user moves the slider 722 to the left, the operation receiving unit 332 receives the movement distance of the slider 722 as the amount of zoom-out processing. That is, the operation receiving unit 332 converts the movement distance of the slider 722 into zoom-in and zoom-out amount information.

[0080] Fig.13 The third mask image generation unit 336 shown in FIG. 1 performs an enlargement process or a reduction process on the processing target mask image 700 according to the enlargement / reduction amount information to generate third mask images 740 and 750. The third mask image 740 is shown in FIG. Fig.15 In FIG. 7 , the third mask image 750 is illustrated in FIG. Fig.16 Specifically, the third mask image generation unit 336 determines the number of pixels to be interpolated in the enlargement process or the number of pixels to be thinned out in the reduction process according to the amount of movement of the slider 722 by the user. The greater the amount of movement of the slider 722 in the right direction by the user, the greater the number of pixels to be interpolated in the enlargement process by the third mask image generation unit 336. In addition, the greater the amount of movement of the slider 722 in the left direction by the user, the greater the number of pixels to be thinned out in the reduction process by the third mask image generation unit 336.

[0081] Fig.13The display control unit 334 shown in FIG. 1 displays projection images 74 and 75 in which the third mask images 740 and 750 are superimposed on the input images 742 and 752. The projection image 74 is shown in FIG. Fig.15 In the example, the projection image 75 is shown in Fig.16 At this time, the operation accepting unit 332 may also accept input of information indicating the amount of expansion and contraction processing or enlargement and reduction processing for the third shielding images 740 and 750. That is, the user may be able to adjust the shielding image again while observing the third shielding images 740 and 750 displayed by the display control unit 334.

[0082] Fig.15 as well as Fig.16 This is an example of the display method of the third mask images 740 and 750 by the display control unit 334. Fig.15 as well as Fig.16 The slider 722 is displayed at a position corresponding to the amount of enlargement or reduction applied to the displayed third shielding images 740 and 750 . Fig.15 It means that it contains Fig.14 FIG. 7 is a diagram of a projection image 74 of a third mask image 740 generated by enlarging the mask image 700 to be processed. Fig.15 , the projection image 74 includes the third shielding image 740, the input image 742, and the background image 744 which is a captured image including the object 40, and is displayed on the touch panel 310. The third shielding image 740 after the enlargement process is enlarged at the same ratio in the vertical direction and the horizontal direction with point O as the reference compared to the processing target shielding image 700.

[0083] Fig.16 It means that it contains Fig.14 FIG. 7 is a diagram of a projection image 75 of a third mask image 750 generated by reducing the mask image 700 to be processed. Fig.16 , the projection image 75 includes a third shielding image 750, an input image 752, and a background image 754 which is a captured image including the object 40, and is displayed on the touch panel 310. The third shielding image 750 after the reduction process is reduced at an equal ratio in the vertical direction and the horizontal direction with point O as a reference compared to the processing target shielding image 700.

[0084] That is, in the second embodiment, the operation accepting unit 332 accepts input of information indicating the amount of enlargement or reduction processing for the processing target shield image 700 as the first shield image 500 or the second shield image 540. The third shield image generating unit 336 performs enlargement or reduction processing on the processing target shield image 700 according to the information indicating the amount of enlargement or reduction processing, and generates third shield images 740 and 750. The display control unit 334 displays the projection images 74 and 75 in which the third shield images 740 and 750 are superimposed on the input images 742 and 752. Thus, the user can instruct the enlargement and reduction processing in addition to the expansion and contraction processing of the shield image, and can adjust the shield area in accordance with the user's sensibility more simply and in detail.

[0085] 3. Modifications

[0086] The above-described embodiments can be modified in many ways. The following are specific modified embodiments that can be applied to the above-described embodiments. Two or more embodiments arbitrarily selected from the following embodiments can be appropriately combined within the scope of non-contradiction.

[0087] In the present embodiment, the processing device 330 displays the slide bar 52 as an interface for accepting the designation of the expansion and contraction amount from the user, but the interface is not limited to this. For example, two buttons, one corresponding to the expansion processing amount and the other corresponding to the contraction processing amount, may also be an interface. In this case, the processing device 330 moves the edge ED of the processing object shielded image 700 outward by a specified unit pixel each time the user presses the button corresponding to the expansion processing amount once. In addition, the processing device 330 moves the edge ED of the processing object shielded image 700 inward by a specified unit pixel each time the user presses the button corresponding to the contraction processing amount once. The two buttons may also be, for example, triangles with marks indicating opposite directions, or the buttons themselves may be in a shape indicating opposite directions. Thus, the user can easily and intuitively operate. In addition, for example, a table in which the user can input numerical values ​​may also be an interface. In this case, the user specifies the expansion and contraction amount by, for example, inputting the number of pixels to move the edge ED during the expansion and contraction processing. In addition, for example, an interface for accepting the designation of the expansion and contraction amount by voice input may also be provided.

[0088] In the second embodiment, the processing device 330 displays the slide bar 52 for accepting the designation of the expansion and contraction amount and the slide bar 72 for accepting the designation of the enlargement and reduction amount at the same time. However, the processing device 330 may, for example, display an interface for specifying the content of the processing that the user wants to perform next, and after the user specifies the content of the processing, display an interface for specifying the processing amount of the processing. In addition, the processing device 330 may, for example, accept the enlargement processing operation when the user performs a pinch operation on the processing object shielding image 700 on the touch panel 310 while displaying the slide bar 52 for accepting the designation of the expansion and contraction amount, and accept the zoom-in processing operation when the user performs a pinch-in operation.

[0089] In addition, in the present embodiment, the processing device 330 accepts expansion and contraction processing and enlargement and reduction processing as processing of the masked image. However, the processing device 330 may also accept processing other than these, such as parallel translation, rotation, inversion, and other processing, and may also provide an interface for accepting such processing together with an interface for accepting expansion and contraction processing and enlargement and reduction processing.

Claims

1. A display method, comprising: Using a captured image obtained by capturing an area including an object, generating a first masked image including a masked area, wherein the masked area has an edge corresponding to a contour of the object in the captured image; accepting input of first information indicating an amount of expansion processing or contraction processing for the first mask image; For the first masked image, the dilation process or the contraction process corresponding to the first information is performed based on the edge to generate a second masked image; as well as A projection image is displayed in which the second mask image is superimposed on the input image.

2. The display method according to claim 1, wherein: The expansion process is a process of moving the edge to the outside of the mask area by an amount of pixels corresponding to the first information. The shrinking process is a process of moving the edge toward the inside of the mask area by an amount of pixels corresponding to the first information.

3. The display method according to claim 1 or 2, wherein: Displaying the projection image includes at least one of: projecting the projection image onto a region including the object using a projector; and causing a display device included in an electronic device to display the projection image.

4. The display method according to claim 3, wherein: The causing the display device to display the projection image includes causing the projection image to be displayed in an overlapping manner with a captured image obtained by capturing a region including the object.

5. The display method according to claim 1 or 2, wherein: The display method further includes: accepting input of second information indicating an amount of enlargement or reduction processing for the first shielded image or the second shielded image; performing the enlargement process or the reduction process corresponding to the second information on the first shielded image or the second shielded image to generate a third shielded image; and A projection image is displayed in which the third mask image is superimposed on an input image.

6. A recording medium having a program recorded thereon, the program causing a computer to execute the following processing: Using a captured image obtained by capturing an area including an object, generating a first masked image including a masked area, wherein the masked area has an edge corresponding to a contour of the object in the captured image; accepting input of first information indicating an amount of expansion processing or contraction processing for the first mask image; For the first masked image, the dilation process or the contraction process corresponding to the first information is performed based on the edge to generate a second masked image; as well as A projection image is displayed in which the second mask image is superimposed on the input image.

Images