Image editing method, image editing system, and storage medium

Abstract

Provided are an image editing method, an image editing system, and a storage medium, which enable a user without professional knowledge to easily perform projection mapping. In a real space in which a projector (10) and a projection target object (SC) are arranged, an information processing apparatus (20) causes a camera (30) to capture a center portion of a range of a projection region containing the projection target object (SC) and displaying a first guide image by the projector (10) when a position at which the center portion of the captured range overlaps the center portion of the first guide image, thereby acquiring a captured image. The information processing apparatus (20) generates a projection image by causing a user to edit the captured image, and transmits projection image data representing the generated projection image to the projector (10).

Description

Technical Field

[0001] This invention relates to image editing methods, image editing systems, and storage media. Background Technology

[0002] In recent years, projection mapping, which involves projecting various images onto objects with three-dimensional shapes using a projector for various performances, has become increasingly popular. Hereinafter, the object onto which the image is projected from the projector will be referred to as the projection object. For projection mapping to be performed, a two-dimensional image projected from the projector, i.e., a two-dimensional image from the projector's viewpoint, needs to be generated beforehand. Various techniques related to the generation of two-dimensional images from the projector's viewpoint have been proposed; as one example, the technique disclosed in Patent Document 1 can be cited. In the technique disclosed in Patent Document 1, a three-dimensional projection object is obtained from the projector's viewpoint image; a three-dimensional model corresponding to the projection object is prepared as the projection content; this three-dimensional model is transformed into a two-dimensional image consistent with the projection device's viewpoint image; thereby generating a two-dimensional image projected from the projector.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2017-192040

[0004] In order to perform projection mapping using the technology disclosed in Patent Document 1, users are required to possess expertise such as insights into the shape of the projected object as seen from the projector's viewpoint and insights into the three-dimensional model. However, users who wish to perform projection mapping do not necessarily possess such expertise. In the technology disclosed in Patent Document 1, there is a problem that users without such expertise cannot easily implement projection mapping. Summary of the Invention

[0005] One aspect of the image editing method disclosed herein includes the following processing: acquiring a captured image by taking a picture of a projection area within a real space from a position within the real space using a camera, wherein the projection area within the real space is an area containing a projection object that becomes the projection destination of an image from a projector and is displayed as a first image by the projector, the position being a location where the center portion of the camera's shooting range overlaps with the center portion of the first image; and sending a second image obtained by editing the captured image to the projector.

[0006] One aspect of the image editing system disclosed herein includes: a camera; and a processing device that controls the camera, the processing device performing the following processing: acquiring a captured image by taking a picture of a projection area in a real space from a position in the real space, wherein the projection area in the real space is an area containing a projection object that becomes the projection destination of an image from a projector and is displayed by the projector as a first image, the position being a location where the center portion of the camera's shooting range overlaps with the center portion of the first image; and sending a second image obtained by editing the captured image to the projector.

[0007] The procedure disclosed herein enables a computer to perform the following processes: acquiring a captured image by taking a picture of a projection area within a real space from a position within the real space, wherein the projection area within the real space is an area containing a projection object that becomes the projection destination of an image from a projector and is displayed by the projector as a first image, the position being a location where the center portion of the camera's field of view overlaps with the center portion of the first image; and sending a second image obtained by editing the captured image to the projector. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating a structural example of an image editing system 1 according to an embodiment of the present disclosure.

[0009] Figure 2 This is a diagram illustrating an example of the structure of the information processing device 20.

[0010] Figure 3 This is a diagram representing an example of the first guiding image GA1.

[0011] Figure 4 This is a diagram showing an example of the setting of the projected object SC relative to the projection area.

[0012] Figure 5 This is a diagram showing the relationship between the captured image GA2, the second guiding image GA3, and the overlaid image GA4.

[0013] Figure 6 This is a diagram showing an example of the display of the overlapping image GA4, message M1, and user interface image GA5.

[0014] Figure 7 This is a diagram showing an example of an overlapping image GA4 when the central portion of the first guide image GA1 is approximately aligned with the central portion of the second guide image GA3.

[0015] Figure 8 This is a diagram representing an example of a projected image GB1 edited based on a reference image.

[0016] Figure 9 This is a diagram showing an example of the projection of the projector 10 onto the projected image GB1.

[0017] Figure 10 This is a flowchart illustrating the process of an image editing method executed by the processing device 210 according to the program PA.

[0018] Label Explanation

[0019] 1: Image editing system; 10: Projector; 20: Information processing device; 30: Camera; 40: Terminal device; 50: Communication network; 60: Image storage device; 210: Processing device; 210a: Projection control unit; 210b: Display control unit; 210c: Notification unit; 210d: Acquisition unit; 210e: Output unit; 220, 410: External IF devices; 230: Storage device; 420: Display device; 430: Input device; PA: Program; SC: Projection object. Detailed Implementation

[0020] Various technically preferred limitations are attached to the embodiments described below. However, the embodiments of this disclosure are not limited to the methods described below.

[0021] 1. Implementation Method

[0022] Figure 1 This diagram illustrates a structural example of the image editing system 1 according to an embodiment of the present disclosure. The image editing system 1 is an information processing system for editing image data representing an image projected from a projector 10 onto a projection area containing a projection object SC. In this embodiment, the projection object SC is a circular fan depicting a figure on a substrate paper. The projector 10 and the projection object SC are located in the living room of a user using the projector 10. The living room of the user using the projector 10 is an example of an actual space in this disclosure.

[0023] The projector 10 is a device that projects an image represented by image data by projecting image light corresponding to the image data provided from the image providing device. In this embodiment, the image providing device is the information processing device 20, which will be described in detail later. Figure 1 Although detailed illustrations are omitted, the projector 10 includes a projection lens, a liquid crystal driving unit, a liquid crystal panel, and a light source unit. The liquid crystal driving unit drives the liquid crystal panel according to image data provided from the image providing device, thereby depicting the image represented by the image data on the liquid crystal panel. The light source unit includes, for example, a halogen lamp or a laser diode. Light from the light source unit is modulated in the liquid crystal panel according to each pixel and projected as image light through the projection lens.

[0024] In this embodiment, the projection area is a defined area within the room containing the projector 10. An image is projected from the projector 10 onto the projection area, and then displayed within the projection area. Hereinafter, the image projected by the projector 10 will be referred to as the projected image. The image data representing the projected image is sometimes referred to as the projected image data. The projection area contains the projected object SC. In this embodiment, by displaying the projected image in the projection area, the projection mapping of the projected object SC is achieved. The projected image can be a still image or a moving image. In this embodiment, the projected image is a still image.

[0025] like Figure 1 As shown, the image editing system 1 includes an information processing device 20, a camera 30, and a terminal device 40. Figure 1 In addition to the image editing system 1, the diagram also shows a projector 10, a projection object SC, a communication network 50, and an image storage device 60.

[0026] The information processing device 20 is, for example, a stick-type personal computer. The information processing device 20, for example, has a male connector according to standards such as USB (Universal Serial Bus). The projector 10 has a female connector corresponding to this male connector. The information processing device 20 is electrically connected to the projector 10 by inserting the male connector of the information processing device 20 into the female connector of the projector 10. The information processing device 20 communicates with the camera 30 and the terminal device 40 wirelessly or via wired means.

[0027] Camera 30 is a device for photographing the projection area. Camera 30 is, for example, mounted on a tripod with its optical axis facing the projection object SC within the user's living space. In this embodiment, the user can arbitrarily change the position of camera 30 within the living space. Camera 30 performs photography under the control of information processing device 20 and outputs image data representing the captured image to information processing device 20. Hereinafter, the image data representing the captured image will be referred to as captured image data.

[0028] The communication network 50 is, for example, an electrical communication line such as the Internet. An image storage device 60 is connected to the communication network 50. The image storage device 60 is, for example, a data server. One or more image data are stored in the image storage device 60. Specific examples of image data stored in the image storage device 60 include projected image data edited by the image editing system 1, or captured image data representing images captured by the camera 30. By having the image storage device 60 store image data, the user of the projector 10 can reuse edited projected images or captured images of the projection area.

[0029] Terminal device 40 is a smartphone used by the user of projector 10. For example... Figure 1 As shown, the terminal device 40 includes an external IF device 410, a display device 420, and an input device 430. The external IF device 410 includes a communication circuit that communicates with the image storage device 60 and the information processing device 20 via a communication network 50. IF is an abbreviation for Interface. The display device 420 includes a liquid crystal display and a driving circuit for the liquid crystal display. Under the control of the information processing device 20, the terminal device 40 displays various images on the display device 420. The input device 430 is a transparent sheet-like pressure-sensitive sensor that covers the display area of ​​the display device 420 and accepts user input operations. The terminal device 40 sends input operation data representing the user's input operation to the input device 430 to the information processing device 20 via the external IF device 410. Thus, the user's input operation is transmitted to the information processing device 20. An image drawing tool is pre-installed in the terminal device 40; this image drawing tool is software used to generate projected images by editing images captured by the camera 30. This image drawing tool corresponds to the editing function in this disclosure.

[0030] The information processing device 20 performs the following actions based on the input operation performed on the terminal device 40: capturing images of the projection area based on the camera 30, editing the projection image data corresponding to the user's input operation on the input device 430, and outputting the edited projection image data to the projector 10, as detailed later. The projector 10 projects the projection image represented by the projection image data output from the information processing device 20 onto the projection area.

[0031] Figure 2 This is a diagram illustrating an example of the structure of the information processing device 20. (See diagram for example.) Figure 2 As shown, the information processing device 20 includes a processing unit 210, an external information processing unit 220, and a storage unit 230. The processing unit 210 is configured, for example, to include a processor such as a CPU (Central Processing Unit), i.e., a computer. The processing unit 210 can be composed of a single processor or multiple processors. The processing unit 210 operates according to a program PA stored in the storage unit 230, thereby functioning as the control center of the information processing device 20.

[0032] The external IF device 220 includes the aforementioned male connector. When the male connector is inserted into the female connector of the projector 10, and the information processing device 20 is electrically connected to the projector 10, the external IF device 220 outputs data or signals provided by the processing device 210 to the projector 10. The external IF device 220 includes communication circuitry for communicating with the camera 30 or the terminal device 40.

[0033] Storage device 230 is a recording medium readable by processing device 210. Storage device 230 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory).

[0034] The non-volatile memory of storage device 230 contains a program PA that causes processing device 210 to execute the image editing method of this disclosure. The volatile memory of storage device 230 is used by processing device 210 as a working area when executing program PA.

[0035] When the processing unit 210 detects a connection between the information processing unit 20 and the projector 10, it reads the program PA from non-volatile memory into volatile memory and begins executing the read program PA. The processing unit 210, which operates according to the program PA, detects a connection between the terminal device 40 and the camera 30, and then... Figure 2 The projection control unit 210a, display control unit 210b, notification unit 210c, acquisition unit 210d, and output unit 210e shown in the diagram perform their functions. Figure 2 The projection control unit 210a, display control unit 210b, notification unit 210c, acquisition unit 210d, and output unit 210e shown are software modules implemented by operating the processing device 210 according to the program PA. Figure 2 The functions of the projection control unit 210a, display control unit 210b, notification unit 210c, acquisition unit 210d, and output unit 210e shown are as follows.

[0036] The projection control unit 210a outputs a first signal to the projector 10, indicating that an image representing the projection range of the projector 10 should be projected. The projection range of the projector 10 is determined based on the orientation of the optical axis of the projector 10 and the field of view of the projector 10. Hereinafter, the image representing the projection range of the projector 10 will be referred to as the first guide image GA1. Specific examples of the first guide image GA1 include, for example, a rectangular image uniformly painted with a first color such as white, or a rectangular image with a border made of a second color different from the first color. In this embodiment, the second color is red, but the second color is not limited to red. Figure 3 This is a diagram representing an example of the first guiding image GA1. In Figure 3In this diagram, the shading represents the first color, and the dashed lines represent lines drawn with the second color. That is, in this embodiment, the first guide image GA1 is a rectangular image uniformly painted with the first color and bordered with the second color, and the area enclosed by the dashed lines becomes the projection area. The first guide image GA1 is an example of the first image in this disclosure. The projector 10, receiving the first signal, projects the first guide image GA1 onto the projection area. Figure 4 As shown, the visual confirmation indicates that the user has set the projected object SC within the projection range shown by the first guide image GA1 displayed in the projection area. Figure 4 In this text, the depiction of the shading lines representing the first color is omitted. That is, the projected object SC is included in the projection area, and the first guide image GA1 is displayed.

[0037] The display control unit 210b outputs a second signal indicating the start of shooting to the camera 30. Upon receiving the second signal, the camera 30 takes pictures at a predetermined interval, such as a few milliseconds, and outputs the captured image data to the information processing device 20. As described above, the camera 30 is positioned in the user's room with its optical axis facing the projection object SC. Therefore, in the captured image GA2 of the camera 30, the projection area displaying the first guide image GA1 and the projection object SC placed in the projection area are reflected.

[0038] Whenever image data is input from camera 30, display control unit 210b generates a value based on the image data. Figure 5 The overlapping image GA4 is shown. Figure 5 This is a diagram showing the relationship between the overlapping image GA4, the captured image GA2 represented by the captured image data, and the second guide image GA3. The second guide image GA3 is an image representing the shooting range of the camera 30. The shooting range of the camera 30 is determined based on the orientation of the optical axis of the camera 30 and the field of view of the camera 30. In this embodiment, the second guide image GA3 is an image of a rectangular frame that divides the outer edge of the captured image GA2. In this embodiment, the frame is colored with a third color that is different from both the first and second colors. Figure 5 The shading based on the third color is represented by a single-dotted line. In this embodiment, the third color is blue, but the third color is not limited to blue. Figure 5 As shown, the overlaid image GA4 is an image in which the captured image GA2 is overlaid with the second guiding image GA3.

[0039] like Figure 6 As shown, the display control unit 210b displays the overlay image GA4 on the display device 420. The display control unit 210b also displays a user interface image GA5 on the display device 420, prompting the user to input a decision regarding the position of the camera 30. (As shown...) Figure 6As shown, the user interface image GA5 in this disclosure is an image of a virtual operator.

[0040] The notification unit 210c notifies the camera 30 to move to a position where the center portion of the first guide image GA1 overlaps with the center portion of the second guide image GA3. The center portion of the first guide image GA1 refers to the area inside a first circle of a predetermined radius centered on the center position of the first guide image GA1. The center portion of the second guide image GA3 refers to the area inside a second circle of a predetermined radius centered on the center position of the second guide image GA3. The center position of the first guide image GA1 is the intersection of the two diagonals of the first guide image GA1. Similarly, the center position of the second guide image GA3 is the intersection of the two diagonals of the second guide image GA3. The radii of the circles defining the center portions of the first and second guide images GA1 and GA3 can be determined to be appropriate values ​​through experiments, etc. The overlap of the center portions of the first guide image GA1 and the second guide image GA3 means that the distance between the centers of the first and second circles is less than the aforementioned radii.

[0041] As described above, the first guide image GA1 corresponds to the projection range of the projector 10, and the second guide image GA3 corresponds to the shooting range of the camera 30. Therefore, the overlap of the central portion of the first guide image GA1 and the central portion of the second guide image GA3 is equivalent to the overlap of the central portion of the projection range of the projector 10 and the central portion of the shooting range of the camera 30.

[0042] In this embodiment, the notification unit 210c is as follows: Figure 6 As shown, a message M1 is displayed on the display device 420, prompting the camera 30 to move to a position where the center portion of the first guide image GA1 overlaps with the center portion of the second guide image GA3. In this embodiment, displaying message M1 on the display device 420 is equivalent to notifying the camera 30 to move to a position where the center portion of the first guide image GA1 overlaps with the center portion of the second guide image GA3. The notification unit 210c can also make the above notification by outputting the voice of the message M1. The user who sees message M1 checks the overlap between the center portion of the first guide image GA1 and the center portion of the second guide image GA3 in the overlapping image GA4 displayed on the display device 420 while moving the camera 30. As the camera 30 moves, the overlap between the center portion of the first guide image GA1 and the center portion of the second guide image GA3 in the overlapping image GA4 displayed on the display device 420 changes.

[0043] The acquisition unit 210d determines, based on the overlapping image GA4, whether the size of the overlapping area between the center portion of the first guide image GA1 and the center portion of the second guide image GA3 is greater than or equal to a threshold. This threshold can also be determined based on the area of ​​the first or second circle described above; as a specific example of the threshold in this method, a value of 60% to 90% of the area of ​​the first or second circle can be given. If the size of the overlapping area is less than the threshold, the acquisition unit 210d instructs the notification unit 210c to continue displaying message M1. If the size of the overlapping area is greater than or equal to the threshold, the acquisition unit 210d instructs the notification unit 210c to stop displaying message M1. Figure 7 This diagram shows an example of the overlapping image GA4 and the user interface image GA5 when the size of the overlapping region is above a threshold. Because the size of the overlapping region is above the threshold, therefore... Figure 7 In the example shown, message M1 is not displayed. Figure 7 As shown, when message M1 is no longer displayed on display device 420, the user performs a touch operation on user interface image GA5.

[0044] Upon detecting a touch operation on the user interface image GA5, the acquisition unit 210d first outputs a third signal to the projector 10, instructing the projection of the first guide image GA1 to stop. Next, the acquisition unit 210d writes the captured image data output from the camera 30 as reference image data, representing a reference image used as a reference for editing the projected image, into the volatile memory of the storage device 230. The reference image data is image data representing the captured image GA2, which represents the projection area where the first guide image GA1 is not displayed.

[0045] The output unit 210e obtains projection image data representing the projected image by allowing the user to edit the reference image. In this embodiment, the output unit 210e first sends the reference image data to the terminal device 40 and activates the image drawing tool. The terminal device 40, which is executing the image drawing tool, displays the image represented by the reference image data on the display device 420. While referring to the position and shape of the projected object SC reflected in the reference image, the user uses the image drawing tool to edit the projected image used to decorate the projected object SC.

[0046] Figure 8 This is a diagram illustrating an example of a projected image edited based on a reference image. In Figure 8 In the reference image, the projected object SC is represented by a dashed line, and the image of the star, which is given a shadow by the horizontal line, is a projected image drawn by the user to decorate the projected object SC. Figure 8The shading of the horizontal lines in the image represents colors such as yellow, which are assigned to the image of the stars. The output unit 210e obtains image data representing the projected image edited based on the reference image from the terminal device 40 as projected image data. The projected image edited based on the reference image is an example of the second image in this disclosure. The output unit 210e outputs the projected image data to the projector 10. The projector 10 projects the projected image represented by the projected image data output from the information processing device 20 onto the projection area. As a result, such as Figure 9 As shown, images of stars used to decorate the projected object SC are displayed around it. Figure 9 The horizontal shading in the image represents colors such as yellow, which are assigned to the image of stars.

[0047] The position of the camera 30 when capturing the reference image is the point where the center portion of the first guide image GA1 overlaps with the center portion of the second guide image GA3. The overlap of the center portions of the first guide image GA1 and the second guide image GA3 means that the center portion of the projection range of the projector 10 overlaps with the center portion of the shooting range of the camera 30. This overlap also means that the orientation of the optical axis of the projector 10 is approximately the same as the orientation of the optical axis of the camera 30, and the field of view of the projector 10 is approximately the same as the field of view of the camera 30. Therefore, the position and shape of the projected object SC in the reference image are approximately the same as the position and shape of the projected object SC when viewed from the projector 10. Therefore, by referring to the position and shape of the projected object SC reflected in the reference image while drawing a projected image for decorating the projected object SC, the user can generate a projected image that matches the position and shape of the projected object SC.

[0048] Processing device 210, which operates according to procedure PA, executes... Figure 10 The image editing method shown. Figure 10 As shown, the image editing method of this embodiment includes projection control processing SA110, display control processing SA120, notification processing SA130, acquisition processing SA140, and output processing SA150.

[0049] In the projection control process SA110, the processing device 210 functions as the projection control unit 210a. In the projection control process SA110, the processing device 210 outputs a first signal to the projector 10, instructing the projector 10 to display the first guide image GA1.

[0050] In the display control processing SA120, the processing unit 210 functions as the display control unit 210b. In the display control processing SA120, the processing unit 210 outputs a second signal instructing the camera to capture the image to the camera 30. Next, the processing unit 210 displays an overlaid image GA4 obtained by overlaying the captured image GA2 (captured by the camera 30) with a second guiding image GA3, and a user interface image GA5 on the display device 420.

[0051] In notification processing SA130, processing device 210 functions as notification unit 210c. In notification processing SA130, processing device 210 notifies the camera 30 to move to a position where the center portion of the first guide image GA1 overlaps with the center portion of the second guide image GA3 by displaying message M1 on display device 420.

[0052] In the acquisition process SA140, the processing device 210 functions as the acquisition unit 210d. In the acquisition process SA140, the processing device 210 determines whether the size of the overlapping area is above or below a threshold. If the size of the overlapping area is below the threshold, the acquisition unit 210d instructs the notification unit 210c to continue displaying message M1. If the size of the overlapping area is above or below the threshold, the acquisition unit 210d instructs the notification unit 210c to stop displaying message M1. The user, taking advantage of the message M1 no longer being displayed on the display device 420, performs a touch operation on the user interface image GA5. Upon detecting a touch operation on the user interface image GA5, the processing device 210 outputs a third signal to the projector 10 instructing the projector to stop the projection of the first guide image GA1, and writes the captured image data of the captured image GA2, representing the projection area where the first guide image GA1 is not displayed, i.e., the reference image data representing the reference image, into the volatile memory of the storage device 230.

[0053] In the output processing SA150, the processing unit 210 functions as the output unit 210e. In the output processing SA150, the processing unit 210 edits a reference image represented by the user's reference image data to obtain projected image data representing the projected image. Then, the processing unit 210 outputs the projected image data to the projector 10.

[0054] As described above, in this embodiment, the user of the projector 10 is required to perform touch operations based on the position of the camera 30 determined by message M1, and to edit the projected image by referring to the image GA2 captured by the camera 30. In the technology disclosed in Patent Document 1, it is necessary to prepare a three-dimensional model corresponding to the projected object SC, and to transform the coordinates of this three-dimensional model into a two-dimensional image consistent with the viewpoint image of the projection device. This requires expertise in both three-dimensional models and coordinate transformations. In contrast, in this embodiment, it is not necessary to prepare a three-dimensional model corresponding to the projected object SC, nor is it necessary to perform coordinate transformations when editing the projected image. Therefore, according to this embodiment, users without expertise in three-dimensional models can easily perform projection mapping.

[0055] 2. Deformation

[0056] The above embodiments can be modified as follows.

[0057] (1) In the above embodiment, an application example of the present disclosure to the projection mapping of decorations in the room of a user implementing the projector 10 was described. However, the present disclosure can be applied to the projection mapping of merchandise displays in stores such as retail stores, as well as to the projection mapping of performances in theme parks or event venues.

[0058] (2) In the above embodiment, a projector 10 projects an image onto a projection object SC. However, multiple projectors 10 positioned at different locations relative to the projection object SC can also project the image. By projecting images from multiple projectors 10 positioned at different locations relative to the projection object SC, increased brightness in the projection mapping can be achieved. Furthermore, by projecting images from multiple projectors 10 positioned at different locations relative to the projection object SC, shadows can be minimized, achieving a projection mapping that is visible from any angle (360°).

[0059] (3) In the above embodiment, when a touch operation is performed on the user interface image GA5, the acquisition unit 210d outputs a third signal to stop the display of the first guide image GA1, and then acquires reference image data. However, the output of the third signal may be omitted. In the method of omitting the output of the third signal, the first guide image GA1 is included in the captured image GA2 represented by the captured image data output from the camera 30. The acquisition unit 210d may also directly write the captured image data representing the captured image GA2 containing the first guide image GA1 as reference image data into the volatile memory of the storage device 230, in which the reference image includes the first guide image GA1. In addition, in the method of omitting the output of the third signal, the acquisition unit 210d may also generate reference image data by performing image processing on the captured image data to eliminate or reduce the first guide image GA1. As a specific example of image processing to eliminate or reduce the first guide image GA1, an example is image processing that replaces the pixel value of the pixel corresponding to the first guide image GA1 with the pixel value of the surrounding pixels that are not corresponding to the first guide image GA1. As another specific example of image processing to eliminate or thin the first guide image GA1, one could cite an image processing method that replaces the pixel value of the pixel corresponding to the first guide image GA1 with the average of the pixel value of that pixel and the pixel values ​​of a plurality of surrounding pixels. According to this method, the period from the input operation indicating the shooting to the actual shooting is shortened.

[0060] (4) In the above embodiment, if the acquisition unit 210d determines that the size of the overlapping area is less than a threshold, it causes the notification unit 210c to continue urging the camera 30 to move; if the size of the overlapping area is determined to be above the threshold, it causes the notification unit 210c to stop. However, if the acquisition unit 210d determines that the size of the overlapping area is above the threshold, it may also cause the notification unit 210c to urge the user to input a decision on the position of the camera 30. Alternatively, if the acquisition unit 210d determines that the size of the overlapping area is above the threshold, it may store the reference image data without waiting for user input. According to the latter approach, if the size of the overlapping area is above the threshold, the reference image data can be automatically stored, reducing user inconvenience. In the latter approach, the display of the user interface image GA5 is not required. Furthermore, determining whether the size of the overlapping area is less than the threshold is not necessary and can be omitted.

[0061] (5) In the above embodiment, the second guide image GA3 is an image representing the shooting range of the camera 30. However, in order to clearly show the central portion of the shooting range of the camera 30, the display control unit 210b may display a third guide image representing the central portion of the shooting range of the camera 30 on the display device 420, either in place of the second guide image GA3 or together with the second guide image GA3. In this manner, the position in the actual space where the central portion of the shooting range overlaps with the central portion of the first guide image GA1 refers to the central portion of the third guide image in the overlapping image GA4 or the position where the third guide image itself overlaps with the central portion of the first guide image GA1.

[0062] The third guide image can also be an image smaller than the second guide image GA3. Specific examples of a third guide image smaller than the second guide image GA3 include the aforementioned image of the second circle, an image of a polygon inscribed or circumscribed by the aforementioned second circle, or an image of multiple line segments intersecting the center of the second circle. By displaying the third guide image, smaller than the second guide image GA3, together with the second guide image GA3, the overall shooting range of the camera 30 and the central portion of that shooting range can be clearly shown to the user. When a third guide image, smaller than the second guide image GA3, is displayed instead of the second guide image GA3, the central portion of the shooting range of the camera 30 can be clearly shown to the user.

[0063] The third guide image can also be an image of the same size as the second guide image GA3. A specific example of a third guide image of the same size as the second guide image GA3 could be an image comprising a rectangular frame dividing the outer edge of the captured image GA2 and its diagonal, or an image of the diagonal. The third guide image is an example of the third image in this disclosure. In the former case, by displaying the third guide image instead of the second guide image GA3, the overall shooting range of the camera 30 and the central portion of that shooting range can be clearly shown to the user. In the latter case, when the third guide image is displayed together with the second guide image GA3, the overall shooting range of the camera 30 and the central portion of that shooting range can be clearly shown to the user; when the third guide image is displayed instead of the second guide image GA3, the central portion of the shooting range of the camera 30 can be clearly shown to the user.

[0064] Alternatively, in order to clearly show the central portion of the first guide image GA1, an image representing the central portion of the first guide image GA1 may be included in the first guide image GA1. Specific examples of images representing the central portion of the first guide image GA1 include the aforementioned image of the first circle, an image of a polygon inscribed or circumscribed with the aforementioned first circle, or an image of multiple line segments intersecting the center of the first circle.

[0065] (6) In the above embodiment, the output unit 210e initiates the image drawing tool by sending the reference image data to the terminal device 40, taking the storage of the reference image data to the volatile memory as the trigger. However, the output unit 210e may also initiate the storage of the reference image data to the volatile memory by displaying a user interface image, such as a virtual operation tool for initiating the image drawing tool, on the display device 420. In this manner, the output unit 210e initiates the image drawing tool on the terminal device 40 based on the input operation for the user interface image.

[0066] (7) The notification processing SA130 in the above embodiment can also be omitted. In the case of omitting the notification processing SA130, the notification unit 210c can also be omitted. This is because even if the notification processing SA130 is omitted, the projected image can be easily generated. When the projector 10 projects the first guide image GA1 based on the power-on of the projector 10, the projection control unit 210a and the projection control processing SA110 can be omitted. Since the shooting range of the camera 30 can be determined based on the captured image GA2, and the center part of the shooting range can also be determined, the overlap of the second guide image GA3 can also be omitted.

[0067] (8) In the above embodiments, the camera 30 and the terminal device 40 are independent devices, but the camera 30 may also be included in the terminal device 40. For example, if the terminal device 40 is a smartphone equipped with a camera, the camera of the smartphone can perform the function of the camera 30. In addition, in the above embodiments, the information processing device 20 is a device that is different from the terminal device 40, the camera 30, and the projector 10. However, the information processing device 20 may also be included in any one of the terminal device 40, the camera 30, and the projector 10.

[0068] (9) In the above embodiments, the projection control unit 210a, display control unit 210b, notification unit 210c, acquisition unit 210d, and output unit 210e are software modules. However, any one or more, or all of the projection control unit 210a, display control unit 210b, notification unit 210c, acquisition unit 210d, and output unit 210e may also be hardware modules such as ASICs (Application Specific Integrated Circuits). Even if any one or more, or all of the projection control unit 210a, display control unit 210b, notification unit 210c, acquisition unit 210d, and output unit 210e are hardware modules, the same effect as in the above embodiments can be achieved.

[0069] (10) The program PA can be manufactured as a standalone unit or provided for a fee or free of charge. Examples of providing the program PA include writing it into a computer-readable recording medium such as flash ROM, or providing it by downloading it via an electrical communication line such as the Internet. By operating a general computer according to the program PA provided in these ways, the computer can execute the image editing method of this disclosure.

[0070] 3. The method mastered according to the implementation method and at least one of the variations.

[0071] This disclosure is not limited to the embodiments and variations described above, and can be implemented in various ways without departing from its spirit. For example, this disclosure can also be implemented in the following ways. The technical features in the above embodiments corresponding to the technical features in the various methods described below can be appropriately replaced or combined to solve part or all of the problems of this disclosure, or to achieve part or all of the effects of this disclosure. In addition, such technical features can be appropriately deleted in this specification unless they are described as essential features.

[0072] One aspect of the image editing method disclosed herein includes an acquisition process and an output process. In the acquisition process, an image is acquired by a camera of a projection area containing a projection object that becomes the projection destination of an image from a projector, and through which a first guiding image is displayed by the projector. The projection area is a region within the actual space in which the projector and the projection object are configured. The position of the camera that captures the image is a position within the actual space where the center portion of the camera's field of view overlaps with the center portion of the first guiding image. In the output process, a projected image is obtained by editing the captured image acquired in the acquisition process, and this projected image is sent to the projector. According to this image editing method, even users without specialized knowledge can easily perform projection mapping.

[0073] A more preferred image editing method may further include displaying an overlapping image on a display device, the overlapping image being obtained by capturing an image of a projection area on which a first guide image is projected using a camera, and then overlaying it with a third guide image representing the central portion of the shooting range. The third guide image is an example of the third image in this disclosure. The actual spatial position of the camera in this method is where the central portion of the first guide image overlaps with the central portion of the third guide image or the third guide image itself in the overlapping image. According to this method, the user can determine the camera position while simultaneously determining the degree of overlap between the central portion of the camera's shooting range and the central portion of the projector's projection range based on the first and third guide images in the overlapping image.

[0074] A more preferred image editing method may further include: upon obtaining the captured image, displaying a user interface image for invoking editing functions for editing the captured image, wherein the second image in this method can be obtained by editing the captured image using the editing functions. According to this method, the user can invoke editing functions for editing the captured image by operating the user interface image.

[0075] A more preferred image editing method may also include: outputting a notification prompting the camera to move to a position within the actual space where the center portion of the shooting range overlaps with the center portion of the first image. According to this method, the user can be prompted to move the camera to a position within the actual space where the center portion of the camera's shooting range overlaps with the center portion of the first guiding image.

[0076] A more preferred image editing method may include: determining whether the size of the overlapping region is above a threshold; and outputting the aforementioned notification if the size of the overlapping region is less than the threshold. The overlapping region refers to the area in the overlapping image where the center portion of the camera's shooting range overlaps with the center portion of the first guiding image. According to this method, if the size of the overlapping region is less than the threshold, the user can be prompted to move the camera to a position within the actual space where the center portion of the camera's shooting range overlaps with the center portion of the first guiding image.

[0077] A more preferred image editing method may include: having the camera capture the projection area when the size of the overlapping area is greater than or equal to a threshold. According to this method, when the size of the overlapping area is greater than or equal to a threshold, the camera can automatically capture the projection area, reducing the user's inconvenience.

[0078] In a more preferred image editing method, the captured image can also be obtained by capturing an image of the projection area where the first guide image is not displayed. According to this method, the projected image can be edited based on the captured image of the projection area where the first guide image is not displayed.

[0079] One aspect of the image editing system disclosed herein includes a camera and a processing device for controlling the camera. The processing device performs an acquisition process and an output process. In the acquisition process, the processing device acquires an image captured by the camera of a projection area containing a projection object that becomes the projection destination of an image from a projector and is displayed by the projector as a first guiding image. The projection area is a region within the actual space in which the projector and the projection object are configured. The position of the camera that captures the image is a position within the actual space where the center portion of the camera's field of view overlaps with the center portion of the first guiding image. In the output process, the processing device sends projection image data representing the projection image obtained by editing the image captured in the acquisition process to the projector. According to the image editing system of this aspect, even users without specialized knowledge can easily perform projection mapping.

[0080] One method of the procedure disclosed herein enables a computer to perform acquisition and output processing. In the acquisition processing, the computer, operating according to this procedure, acquires an image taken by a camera of a projection area containing a projection object that becomes the projection destination of an image from a projector, and through which a first guiding image is displayed by the projector. The projection area is a region within the actual space in which the projector and the projection object are configured. The position of the camera that took the image is a position within the actual space where the center portion of the camera's field of view overlaps with the center portion of the first guiding image. In the output processing, the computer, operating according to this procedure, sends projection image data representing the projected image obtained by editing the image acquired in the acquisition processing to the projector. According to this procedure, even users without specialized knowledge can easily perform projection mapping.

Claims

1. An image editing method, comprising: An image is obtained by taking a picture of a projection area in the actual space from a camera at the following position in the actual space, wherein the projection area in the actual space is an area that includes a projection object that becomes the projection destination of an image from a projector and displays a first image through the projector, and the position is the location where the center part of the camera's shooting range overlaps with the center part of the first image. The second image, obtained by editing the captured image, is sent to the projector; and An overlaid image is displayed on a display device. The overlaid image is formed by superimposing an image obtained by taking a picture of the projection area displaying the first image using the camera, and a third image representing the central portion of the shooting range. The location is where the center portion of the third image overlaps with the center portion of the first image in the overlapping image.

2. The image editing method according to claim 1, wherein, The image editing method further includes: upon obtaining the captured image, displaying a user interface image, the user interface image being used to invoke editing functions for editing the captured image. The second image is obtained by editing the captured image using the editing function.

3. The image editing method according to claim 1, wherein, The image editing method further includes: outputting a notification prompting the camera to move to the position.

4. The image editing method according to claim 3, wherein, The image editing method further includes: determining whether the size of the area where the central portion of the first image overlaps with the central portion of the shooting range in the overlapping images is less than a threshold. The notification is output when the size of the region is less than the threshold.

5. The image editing method according to claim 4, wherein, If the size of the area is above the threshold, the camera is used to capture images of the projection area.

6. The image editing method according to claim 1, wherein, The captured image is obtained by having the camera capture an image of the projection area where the first image is not displayed.

7. An image editing system, comprising: camera; as well as Processing device, which controls the camera, The processing device performs the following processing: An image is obtained by taking a picture of a projection area in the actual space from a camera at the following position in the actual space, wherein the projection area in the actual space is an area that includes a projection object that becomes the projection destination of an image from a projector and displays a first image through the projector, and the position is the location where the center part of the camera's shooting range overlaps with the center part of the first image. The second image, obtained by editing the captured image, is sent to the projector; and An overlaid image is displayed on a display device. The overlaid image is formed by superimposing an image obtained by taking a picture of the projection area displaying the first image using the camera, and a third image representing the central portion of the shooting range. The location is where the center portion of the third image overlaps with the center portion of the first image in the overlapping image.

8. A storage medium storing a program that causes a computer to perform the following processes: An image is obtained by taking a picture of the projection area within the actual space from the following positions within the actual space: The projection area within the actual space is the area containing the projection object that becomes the projection destination of the image from the projector and the area through which the first image is displayed by the projector. The location is the position where the center portion of the camera's shooting range overlaps with the center portion of the first image. The second image, obtained by editing the captured image, is sent to the projector; as well as An overlaid image is displayed on a display device. The overlaid image is formed by superimposing an image obtained by taking a picture of the projection area displaying the first image using the camera, and a third image representing the central portion of the shooting range. The location is where the center portion of the third image overlaps with the center portion of the first image in the overlapping image.

Images