Display method and display system

Abstract

To provide a display method and a display system that can easily designate the details of an image to be displayed and the arrangement of a plurality of images.SOLUTION: A projector picks up, with an imaging apparatus, a picked-up image including a marker M1 arranged on a projection surface P of a projection device and a marker M2 arranged on the projection surface P. The projector displays, with the projection device, a projection image I1 corresponding to the marker M1 and a projection image I2 corresponding to the marker M2 on the projection surface P in an arrangement corresponding to the arrangement of the marker M1 and the marker M2 in the picked-up image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a display method and a display system. [Background technology]

[0002] Conventionally, technologies have been developed to assist workers by projecting images relating to the procedures for cooking or other tasks near the work object. For example, Patent Document 1 below discloses a cooking assistance device including a digital camera unit, a projector unit, an identification processing unit, a cooking method determination unit, and a cooking method instruction unit. The digital camera unit captures images of ingredients placed on a counter in a system kitchen. The identification processing unit identifies the types of ingredients shown in the captured image, and the cooking method determination unit determines a cooking method for the identified ingredients. The cooking method instruction unit causes the projector unit to project an image indicating the cooking method for the identified ingredients. The image projected by the projector unit is displayed superimposed on the ingredients placed on the counter. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-191745 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, the image is displayed superimposed on or near the food ingredient, and there is no mention of the user deciding the display position of the image. However, it is expected that various items, such as materials and tools, used for the work will be lined up on the work table during work. If the display position of the image is predetermined, the image may be displayed superimposed on the items, which may make it difficult for the user to understand the content of the image. [Means for solving the problem]

[0005] A display method according to one aspect of the present invention includes capturing an image including a first marker arranged on a projection surface and a second marker arranged on the projection surface, and displaying a first image corresponding to the first marker and a second image corresponding to the second marker on the projection surface in an arrangement corresponding to the arrangement of the first marker and the second marker in the captured image.

[0006] A display system according to one aspect of the present invention includes an imaging device that captures an image including a first marker arranged on a projection surface and a second marker arranged on the projection surface, and a projection device that displays a first image corresponding to the first marker and a second image corresponding to the second marker on the projection surface in an arrangement that corresponds to the arrangement of the first marker and the second marker in the captured image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an overview of a projector according to a first embodiment. [Figure 2] 1 is a block diagram showing the configuration of a projector according to a first embodiment. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a control unit according to the first embodiment. [Figure 4] 2A to 2C are diagrams schematically showing the appearance of a marker in the first embodiment. [Figure 5] FIG. 2 is a diagram schematically illustrating an arrangement state of markers in the first embodiment. [Figure 6] FIG. 3 is a diagram schematically illustrating a method for determining the arrangement of projected images in the first embodiment. [Figure 7] FIG. 3 is a diagram schematically illustrating a method for determining the arrangement of projected images in the first embodiment. [Figure 8] FIG. 2 is a diagram schematically illustrating the arrangement of projected images in the first embodiment. [Figure 9] 5A and 5B are diagrams schematically showing other arrangements of projected images in the first embodiment. [Figure 10] FIG. 2 is a diagram schematically illustrating a projection image database. [Figure 11] 10 is a flowchart showing the flow of a display method executed by a control unit of the projector according to a program. [Figure 12] FIG. 10 is a diagram schematically illustrating an arrangement of markers in a second embodiment. [Figure 13] FIG. 10 is a diagram schematically illustrating the arrangement of projected images in the second embodiment. [Figure 14] 5A to 5C are schematic diagrams showing the appearance of a marker in a second embodiment. [Figure 15] 5A to 5C are schematic diagrams showing the appearance of a marker in a second embodiment. [Figure 16] 10A to 10C are diagrams schematically showing the appearance of a marker in a third embodiment. [Figure 17] FIG. 11 is a block diagram showing the functional configuration of a control unit in a third embodiment. [Figure 18] FIG. 11 is a diagram schematically illustrating an example of the arrangement of markers and objects in the third embodiment. [Figure 19] FIG. 10 is a diagram schematically illustrating the arrangement of projected images in a third embodiment. [Figure 20] FIG. 10 is a diagram schematically showing another example of the arrangement of markers and objects in the third embodiment. [Figure 21] FIG. 10 is a diagram schematically showing another arrangement of projected images in the third embodiment. [Figure 22] FIG. 1 is a diagram schematically illustrating a show window display using a projector. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions and some parts are shown schematically to facilitate understanding. Furthermore, the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited thereto.

[0009] [First embodiment] 1 is a diagram illustrating an overview of a projector 10 according to a first embodiment. The projector 10 according to the embodiment is an example of a display system including an imaging device 11 and a projection device 12. The projector 10 displays a projection image I on a projection surface P. In this embodiment, the projector 10 displays the projection image I relating to cooking procedures on a counter 26 in a kitchen 20. The kitchen 20 includes, for example, a stove 22, a sink 24, and the counter 26.

[0010] The projector 10 is attached to a wall surface 28 that stands upright from a countertop 26 of the kitchen 20. In this embodiment, the housing of the projector 10 is a rectangular parallelepiped, and an imaging lens 11A of the imaging device 11 and a projection lens 12A of the projection device 12, which will be described later with reference to FIG. 2, are exposed from the surface facing the countertop 26.

[0011] In this embodiment, the projection surface P of the projection device 12 is adjusted to include almost the entire area of the counter 26. The imaging area S of the imaging device 11 is adjusted to include the entire area of the projection surface P. The imaging area S of the imaging device 11 preferably includes the entire projection surface P, but may include a range wider than the projection surface P. In this embodiment, for convenience of explanation, it is assumed that the projection surface P and the imaging area S are the same area. In this embodiment, the projection surface P is on the counter 26, but it may also include, for example, at least a portion of the stove 22 or the sink 24.

[0012] A marker M is placed on the projection surface P. As will be described in detail later, the marker M is used to specify the content of the projection image I to be projected onto the projection surface P and to specify the display position of the projection image I on the projection surface P. While FIG. 1 illustrates one marker M and one projection image I, in actual use, multiple markers M are placed within the imaging area S, and multiple projection images I corresponding to each marker M are projected onto the projection surface P. An object O is placed on the projection surface P. As will be described in detail later, the projection image I has content related to the object O.

[0013] 2 is a block diagram showing the configuration of the projector 10. The projector 10 includes an imaging device 11, a projection device 12, an operation unit 13, a communication unit 14, a storage unit 15, and a control unit 16.

[0014] The imaging device 11 is, for example, a camera equipped with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), which is an imaging element that converts condensed light into an electrical signal. Hereinafter, for simplicity of explanation, it is assumed that the imaging device 11 captures still images. Note that the imaging device 11 may capture moving images instead of still images. As described above, in this embodiment, the imaging area S of the imaging device 11 includes the entire projection surface P of the projection device 12 on the countertop 26. Therefore, the captured image of the imaging device 11 includes objects placed on the countertop 26, such as the object O and the marker M.

[0015] Projection device 12 includes a liquid crystal light valve, a projection lens, and a liquid crystal driver for projecting projection image I. Projection device 12 also includes, as a light source, an ultra-high pressure mercury lamp or a metal halide lamp, for example. Projection device 12 acquires projection image data from control unit 16, for example, and projects projection image I represented by the acquired projection image data onto projection surface P on countertop 26.

[0016] The operation unit 13 accepts instructions and operations from the user to the projector 10. The operation unit 13 may be, for example, a touch panel using a display provided on the housing of the projector 10, or operation buttons provided on the housing. If the operation unit 13 is a touch panel, the operation unit 13 outputs data indicating the detected touch position to the control unit 16. If the operation unit 13 is an operation button, the operation unit 13 outputs data identifying the pressed button to the control unit 16. In this way, the content of the instruction operation by the user to the operation unit 13 is transmitted to the control unit 16.

[0017] The communication unit 14 is, for example, a network card such as a wireless LAN (Local Area Network) card. The communication unit 14 connects to a wide area communication network such as the Internet via a nearby router. The communication unit 14 then connects to an image server 30 connected to the wide area communication network, and transmits and receives various information to and from the image server 30. The image server 30 is provided with a projection image database DB in which multiple projection image data D (see FIG. 10) are stored. The projection image I shown in FIG. 1 and other figures is the projection image data D visualized by the projection device 12. Under the control of the control unit 16, the communication unit 14 transmits an image transmission request to the image server 30 to request transmission of the projection image data D. The communication unit 14 also receives the projection image data D transmitted from the image server 30 and outputs it to the control unit 16.

[0018] FIG. 10 is a diagram illustrating a schematic diagram of a projection image database DB. The projection image database DB stores multiple projection image data D. The projection image data D may be, for example, still image data with a file extension such as "jpg" or video data with a file extension such as "mp4." In this embodiment, the projection image data D records images related to the steps for preparing a specific dish. Examples of images related to the steps for preparing a specific dish include still images displaying text information indicating the names and amounts of ingredients, and video images capturing how to cut and handle the ingredients. That is, in this embodiment, the projection image database DB stores multiple sets of projection image data D for each dish to be cooked. For example, for a dish having steps of (Step 1) preparing ingredients, (Step 2) cutting the ingredients, and (Step 3) heating the ingredients, a set of three images is stored: (Image 1) a still image displaying the names and amounts of ingredients and text information indicating the cooking steps, (Image 2) a video image capturing how to cut the ingredients, and (Image 3) a video image capturing how to heat the ingredients.

[0019] These images are created to be paired with objects O, such as cooking utensils and ingredients used during cooking. For example, image 1 is created to correspond to a cutting board that is mainly used in the non-heating process of cooking, image 2 is created to correspond to ingredients to be cut in step 2, and image 3 is created to correspond to a cooking utensil that heats ingredients. In other words, if image 1 is the first image, the cutting board is the first object, image 2 is the second image, and the ingredients to be cut in step 2 are the second object, image 1 is an image related to the content of the work using the first object, and image 2 is an image related to the content of the work using the second object. As will be described later, a user can display an image corresponding to object O in the vicinity of object O placed on counter 26 by placing marker M of an image corresponding to object O near object O.

[0020] The image identification information N is identification information for uniquely identifying each projection image data D. In this embodiment, the image identification information N includes a three-digit number that is dish identification information for identifying the dish, and a three-digit number that is procedure identification information for identifying the procedure. In the example shown in FIG. 10, the dish with dish identification information 053 includes three images (053_001.jpg, 053_002.mp4, 053_003.mp4). 053_001.jpg corresponds to, for example, the above (Image 1), 053_002.mp4 corresponds to, for example, the above (Image 2), and 053_003.mp4 corresponds to, for example, the above (Image 3).

[0021] The image server 30 can search the projection image database DB using the image identification information N as an index to identify the storage area of the projection image data D corresponding to the image identification information N. In other words, the image identification information N is information for identifying the recording location of a specific projection image.

[0022] In this embodiment, when multiple projection images are projected, the projection areas of each projection image are set to the same size. Therefore, it is preferable that the aspect ratios and image sizes of the projection images stored in the projection image database DB are uniform. Furthermore, the format of the image identification information N described above is an example, and the image identification information N may be in any format as long as it can uniquely identify each projection image data D. For example, a serial number may be assigned to all projection image data D in the projection image database DB, and this serial number may be used as the image identification information N.

[0023] 2 is a recording medium readable by the control unit 16. The storage unit 15 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM).

[0024] A program 15A to be executed by the control unit 16 is stored in advance in the nonvolatile memory of the storage unit 15. The volatile memory of the storage unit 15 is used by the control unit 16 as a work area when executing the program 15A. The program 15A may also be referred to as an "application program," "application software," or "app." The program 15A is obtained, for example, from a server (not shown) via the communication unit 14, and then stored in the storage unit 15.

[0025] 2 includes a processor such as a CPU (Central Processing Unit), i.e., a computer. The control unit 16 may be configured with a single computer or multiple computers. When an operation to start execution of the program 15A is performed on the operation unit 13, the control unit 16 reads the program 15A from the non-volatile memory to the volatile memory. The control unit 16 executes the program 15A read into the volatile memory.

[0026] Fig. 3 is a block diagram showing the functional configuration of control unit 16. Control unit 16 operating in accordance with program 15A functions as imaging control unit 160, marker detection unit 161, display image acquisition unit 162, image layout determination unit 163, and display control unit 164 shown in Fig. 3. Imaging control unit 160, marker detection unit 161, display image acquisition unit 162, image layout determination unit 163, and display control unit 164 shown in Fig. 3 are software modules realized by operating control unit 16 in accordance with program 15A.

[0027] Before describing each functional unit of the control unit 16, the marker M will be described. FIG. 4 is a schematic diagram showing the appearance of the marker M in the first embodiment. More specifically, FIG. 4A shows the front surface of the marker M, and FIG. 4B shows the back surface of the marker M. In this embodiment, the marker M is a plate-shaped member, for example, several centimeters square. In this embodiment, since the marker M is used in the kitchen 20, it is preferable that the marker M be waterproofed, for example, by being made of a waterproof material, or by having its surface waterproofed or printed with waterproof ink. The marker M may be included in the package with the projector 10 when the projector 10 is purchased, or may be included as an attachment to a cookbook, magazine, or the like. The marker M may also be printed using print data downloaded by a user from, for example, the website of the projector 10 manufacturer or a food manufacturer, or a recipe website.

[0028] 4A, a two-dimensional code C, such as a QR code (registered trademark), is added to the front surface of the marker M. When the marker M is placed in the imaging area S of the imaging device 11 and the two-dimensional code C appears in the image captured by the imaging device 11, the marker detection unit 161 of the control unit 16, which will be described later, functions as a two-dimensional code reader and calculates information indicated by the two-dimensional code C. Note that the two-dimensional codes C of the multiple markers M are different from one another, and the information obtained when each marker M is read is also different.

[0029] The information obtained when the front surface of the marker M is read, i.e., the information indicated by the two-dimensional code C, is image identification information N for uniquely identifying specific image data from among the multiple projection image data D stored in the image server 30. As described above, the image identification information N includes three-digit dish identification information and three-digit procedure identification information. For example, the image identification information N indicated by the two-dimensional code C illustrated in FIG. 4A is "053_002," where "053" is the dish identification information and "002" is the procedure identification information. The display image acquisition unit 162, which will be described later, includes the image identification information N in an image transmission request to the image server 30. This allows the display image acquisition unit 162 to obtain the projection image data D corresponding to the read marker M.

[0030] 4B, user information F, which is information intended for the user, is added to the back of the marker M. In this embodiment, the user information F includes dish name information F1, step number information F2, object information F3, and placement information F4.

[0031] The dish name information F1 corresponds to the dish identification information of the image identification information N, and indicates the name of the dish whose cooking steps are shown in the image displayed using the marker M. In the example of FIG. 4B, "Sautéed Pacific saury" is displayed as the dish name information F1. The step number information F2 corresponds to the step identification information of the image identification information N, and indicates which cooking step the image displayed using the marker M relates to. In the example of FIG. 4B, "2" is displayed as the step number information F2.

[0032] The object information F3 indicates which object O the image displayed using the marker M is related to. In the example of FIG. 4B, "Pacific saury" is displayed as the object information F3. Note that the notation above the marker M uses the word "corresponding item" to indicate the item corresponding to the marker M as the word equivalent to "object." The placement information F4 displays how the marker M should be placed relative to the object O. In the example of FIG. 4B, the placement information F4 displays "Place the marker M above the corresponding item if they are arranged horizontally, or to the left of the corresponding item if they are arranged vertically." Horizontal and vertical placement refer to the direction in which multiple markers M are arranged, and can also be called the arrangement of the markers.

[0033] By reading the user information F, the user can know which step of which dish this marker M displays, and in what position marker M should be placed relative to which object O. In the example of FIG. 4B, by reading the dish name information F1 and step number information F2, the user understands that this marker M displays an image relating to the "second" step in making "sautéed sanma." Furthermore, by reading the object information F3 and placement information F4, the user understands that this marker M should be placed "above or to the left of" the "sanma," an ingredient in sautéed sanma.

[0034] In this embodiment, the marker M has a front surface and a back surface, each of which has a two-dimensional code C and user-oriented information F attached thereto, but for example, the two-dimensional code C and user-oriented information F may be attached to the front surface of the marker M. Furthermore, the image identification information N is not limited to being recorded in the two-dimensional code C, and may instead be recorded in a one-dimensional code such as a barcode, or image identification information N written in numbers or letters may be read by OCR (Optical Character Recognition).

[0035] Next, a method for using the markers M will be described. When using the projector 10, the user prepares a set of markers M related to the dish to be cooked. At this time, the user reads the dish name information F1 displayed on the back of the marker M to identify the marker M for the desired dish. Note that the user information F may also display how many steps are included in the cooking process for this dish, i.e., how many markers M are included in the set of markers M related to this dish. This makes it easier for the user to prepare all the necessary markers M. The user prepares the objects O listed in the object information F3 displayed on the back of each marker M and places them on the counter 26. At this time, it is preferable for the user to arrange the objects O in the order listed in the step number information F2. This allows the projected image I to be displayed in the order of the cooking process, allowing the user to smoothly understand the cooking steps.

[0036] Next, the user places markers M corresponding to each object O near the objects O placed on the counter 26. At this time, it is preferable that the user places the markers M according to the placement described in the placement information F4. At this time, the user may, for example, place only a portion of the set of markers M related to the dish to be cooked this time on the counter 26. For example, for a dish that requires many ingredients or has complicated cooking steps, there is a possibility that the set will contain a large number of markers M. In this case, the markers M may be placed on the counter 26 in multiple batches, for example.

[0037] FIG. 5 is a diagram schematically illustrating the arrangement of markers M. FIG. 5 illustrates, from above, the portion of counter 26 that will become projection surface P and imaging area S. When viewed from a user standing facing counter 26, projection surface P forms an XY plane with the horizontal direction as the X axis and the depth direction as the Y axis. The following explanation will be given for a case in which a user is making "sauteed saury." The dish identification information for sauteed saury is assumed to be "053." Sauteed saury includes three steps, with three step numbers 001, 002, and 003. Therefore, the image identification information N for images related to sauteed saury is assumed to be 053_001, 053_002, and 053_003. The following objects O are associated with each step: Step 1: cutting board, Step 2: saury, and Step 3: frying pan.

[0038] Hereinafter, the markers M used when making "sauteed saury" will be referred to as markers M1, M2, and M3. A two-dimensional code C indicating "053_001" is attached to the front surface of marker M1, a two-dimensional code C indicating "053_002" is attached to the front surface of marker M2, and a two-dimensional code C indicating "053_003" is attached to the front surface of marker M3. As described above, the image identification information N of images related to sauteed saury is 053_001, 053_002, and 053_003. Therefore, if marker M1 is the first marker, 053_001.jpg is the first image, and 053_001, which is the image identification information N of 053_001.jpg, is the first information, then the first marker has attached thereto the first information used to identify the recording location of the first image. Furthermore, if marker M2 is the second marker, 053_002.mp4 is the second image, and 053_002, which is the image identification information N of 053_002.mp4, is the second information, then the second marker has added to it second information used to identify the recording location of the second image.

[0039] Furthermore, "sautéed saury" is displayed as dish name information F1 on the back of each of markers M1, M2, and M3. Furthermore, "1" is displayed as step number information F2 on the back of marker M1, "2" is displayed as step number information F2 on the back of marker M2, and "3" is displayed as step number information F2 on the back of marker M3. Furthermore, "cutting board" is displayed as object information F3 on the back of marker M1, "saury" is displayed as object information F3 on the back of marker M2, and "frying pan" is displayed as object information F3 on the back of marker M3.

[0040] Therefore, if marker M1 is the first marker and the cutting board is the first object, the first marker corresponds to the first object used in the work performed by the user. Also, if marker M2 is the second marker and the saury is the second object, the second marker corresponds to the second object used in the work. Also, as in object information F3 in FIG. 4, the first marker includes an indication that indicates to the user that the first marker corresponds to the first object, and the second marker includes an indication that indicates to the user that the second marker corresponds to the second object.

[0041] Also, on the back of each of the markers M1, M2, and M3, placement information F4 is displayed, stating, "Place it above the corresponding item if they are arranged horizontally, or to the left of the corresponding item if they are arranged vertically." Note that when referring to the placement of the marker M or object O, "above" means the back side as viewed from the counter 26.

[0042] The user prepares three markers M1, M2, and M3 with "Sautéed Pacific saury" written in the dish name information F1 on the back. The user also prepares an object O described in the object information F3 of markers M1, M2, and M3. Specifically, the user prepares a cutting board O1 described in marker M1, a Pacific saury O2 described in marker M2, and a frying pan O3 described in marker M3.

[0043] The user places the prepared objects O1 to O3 on the counter 26. At this time, the user arranges the objects O in a line in the order of the step numbers corresponding to each object O. In the example of FIG. 5, from left to right, the cutting board O1 corresponding to step 1, the saury O2 corresponding to step 2, and the frying pan O3 corresponding to step 3 are arranged. In addition, the user places markers M1 to M3 corresponding to each object O1 to O3 on the counter 26. In the example of FIG. 5, the objects O1 to O3 are arranged horizontally. Therefore, the example of FIG. 5 corresponds to "horizontal arrangement" in the arrangement information F4, and the user places the markers M1 to M3 above each object O1 to O3 in order from left to right. Note that the objects O1 to O3 may also be arranged from right to left. In this case, the markers M1 to M3 are also arranged from right to left.

[0044] For example, as shown in Figure 9, a cutting board O1 corresponding to step 1, a saury O2 corresponding to step 2, and a frying pan O3 corresponding to step 3 may be arranged from front to back. This arrangement corresponds to "vertical arrangement" in arrangement information F4, and the user places markers M1 to M3 on the left side of each of the objects O1 to O3. The objects O1 to O3 may also be arranged from back to front. In this case, the markers M1 to M3 are also arranged from back to front.

[0045] Next, each functional unit of the control unit 16 will be described with reference to Fig. 3. The imaging control unit 160 controls the imaging of images by the imaging device 11. The imaging control unit 160 controls, for example, the timing of imaging in the imaging device 11. The timing of imaging may be determined by, for example, receiving an imaging instruction from a user using the operation unit 13. Alternatively, for example, the imaging device 11 may continuously capture a video, and when a state in which an object in the imaging area S remains motionless for a predetermined period of time or more, one frame of the video may be extracted and used as the captured image. In this embodiment, the imaging control unit 160 receives an imaging instruction from a user using the operation unit 13.

[0046] Furthermore, the imaging control unit 160 acquires image data of the image captured by the imaging device 11 from the imaging device 11. In this embodiment, the imaging device 11 is used to capture an image that shows two or more markers M. That is, the imaging device 11 captures an image that includes a first marker M1 placed on the counter 26, which is the projection surface P, and a second marker M2 placed on the projection surface P. Therefore, for example, if two or more markers M are not included in the captured image, the imaging control unit 160 may notify the user of an error and prompt the user to recapture the image.

[0047] The marker detection unit 161 detects markers M from the captured image by the imaging device 11. The marker detection unit 161 also calculates information indicated by the two-dimensional code C added to the markers M, and determines the positions of the multiple markers M in the captured image. More specifically, the marker detection unit 161 detects the two-dimensional code added to the markers M from the captured image by the imaging device 11, and calculates image identification information N according to the specifications of the two-dimensional code C. That is, the marker detection unit 161 acquires first information and second information based on the captured image. In this embodiment, the marker detection unit 161 calculates "053_001" from the two-dimensional code C of marker M1, "053_002" from the two-dimensional code C of marker M2, and "053_003" from the two-dimensional code C of marker M3.

[0048] As described above, two or more markers M appear in the captured image. The marker detection unit 161 detects the positions of these multiple markers M. For example, in the example of FIG. 5, the marker detection unit 161 detects the positions of three markers M1 to M3 lined up in the horizontal direction. Also, in the example of FIG. 9, the marker detection unit 161 detects the positions of three markers M lined up in the vertical direction.

[0049] The display image acquisition unit 162 acquires projection image data D corresponding to the marker M from the image server 30. More specifically, the display image acquisition unit 162 transmits an image transmission request including image identification information N calculated from the two-dimensional code C to the image server 30. The image server 30 transmits the projection image data D identified by the image identification information N to the projector 10. The display image acquisition unit 162 temporarily stores the projection image data D transmitted from the image server 30 in, for example, the storage unit 15. That is, the display image acquisition unit 162 acquires the first image from the storage location of the first image based on the first information. Furthermore, the display image acquisition unit 162 acquires the second image from the storage location of the second image based on the second information.

[0050] In this embodiment, the display image acquisition unit 162 acquires three pieces of image identification information N, namely, "053 _001", "053_00 2 "," 053_00 3 " and the image transmission request In response to the image transmission request, the display image acquisition unit 162 transmits the image to the image server 30. Three projection image data D, namely "053_001.jpg", "053_002.m p4" and "053_003.mp4".

[0051] The image layout determination unit 163 determines the layout of the projected image I on the projection surface P. More specifically, the image layout determination unit 163 determines the layout direction of the projected image I based on the positions of multiple markers M in the captured image. FIGS. 6 and 7 are diagrams schematically showing a method for determining the layout of the projected image I. In FIG. 6, multiple markers M1 to M3 are arranged horizontally, as in FIG. 5. When multiple markers M1 to M3 are arranged horizontally, the image layout determination unit 163 sets the upper side of the marker M as the projection area L0. This is why the layout information F4 for the markers M1 to M3 specifies that the marker M should be placed "above the corresponding item when arranged horizontally." In other words, the layout information F4 indicates the position where the marker M should be placed so that the object O and the projected image I do not overlap. Note that the image layout determination unit 163 may also arrange the projected image I below the marker M. In this case, the placement information F4 describes that the marker M should be placed "below the corresponding product when they are arranged side by side." The user may be able to specify the placement of the projection image I relative to the marker M using the operation unit 13 or the like.

[0052] When the markers M are arranged horizontally, the image arrangement determination unit 163 determines the arrangement of each projection image I so that the projection images I corresponding to each marker M are also arranged horizontally. Specifically, as shown in Fig. 7, the image arrangement determination unit 163 divides the projection area L0 horizontally into three areas, and sets a projection area L1 onto which the projection image I corresponding to marker M1 is projected, a projection area L2 onto which the projection image I corresponding to marker M2 is projected, and a projection area L3 onto which the projection image I corresponding to marker M3 is projected. Note that in Figs. 6, 7, etc., the edges of the projection areas L (L0, L1, etc.) are positioned away from the edges of the projection surface P from the viewpoint of visibility, but the edges of the projection areas L may be aligned with the edges of the projection surface P.

[0053] The display control unit 164 controls the projection device 12 to project the projection image I onto the projection surface P. More specifically, the display control unit 164 controls the projection device 12 to project the projection image I onto the projection surface P in accordance with the arrangement determined by the image arrangement determination unit 163.

[0054] In this embodiment, as shown in Fig. 8, the display control unit 164 controls the projection device 12 to display a projection image I1 of the projection image data "053_001.jpg" in the projection area L1 corresponding to the marker M1. The display control unit 164 also controls the projection device 12 to display a projection image I2 of the projection image data "053_002.mp4" in the projection area L2 corresponding to the marker M2. The display control unit 164 also controls the projection device 12 to display a projection image I3 of the projection image data "053_003.mp4" in the projection area L3 corresponding to the marker M3.

[0055] That is, the projection device 12 displays a first image corresponding to the first marker and a second image corresponding to the second marker on the projection surface P in an arrangement corresponding to the arrangement of the first marker and the second marker in the captured image. At this time, the arrangement direction of the first image and the second image on the projection surface P matches the arrangement direction of the first marker and the second marker in the captured image.

[0056] 9, for example, when multiple markers M1 to M3 are lined up vertically, the image layout determination unit 163 determines the left side of the marker M as the projection area L0. This is why the layout information F4 for the markers M1 to M3 specifies that the marker M should be placed "on the left side of the corresponding item when they are lined up vertically." In other words, the layout information F4 indicates the position where the marker M should be placed so that the object O and the projected image I do not overlap.

[0057] Furthermore, when the markers M are lined up vertically, the image layout determination unit 163 determines the layout of each projection image I so that the projection images I corresponding to each marker M are also lined up vertically. Specifically, as shown in Fig. 9, the image layout determination unit 163 divides the projection area L0 vertically into three areas, and sets a projection area L1 onto which the projection image I corresponding to marker M1 is projected, a projection area L2 onto which the projection image I corresponding to marker M2 is projected, and a projection area L3 onto which the projection image I corresponding to marker M3 is projected.

[0058] The display control unit 164 controls the projection device 12 to project the projection image I onto the projection surface P in accordance with the layout determined by the image layout determination unit 163. Note that the aspect ratio of the original projection image I is maintained when projected from the projection device 12. Therefore, the magnification of the projection image I differs depending on the shape of the projection area L. For example, even if there is a projection area L that is long in the horizontal direction as shown in FIG. 9, the projection image I is displayed at a size that matches the maximum magnification in the vertical direction.

[0059] 11 is a flowchart showing the flow of a display method executed by the control unit 16 of the projector 10 in accordance with the program 15A. Prior to executing the processing of FIG. 11, the user places a plurality of markers M and an object O on the projection surface P as shown in FIG.

[0060] The control unit 16 functions as the imaging control unit 160 to instruct the imaging device 11 to capture an image of the imaging area S (step S100). Also, the control unit 16 functions as the imaging control unit 160 to acquire the captured image captured by the imaging device 11 (step S102).

[0061] The control unit 16 functions as the marker detection unit 161 to detect multiple markers M from the captured image (step S104). The control unit 16 functions as the image layout determination unit 163 to determine the layout of the projected images I corresponding to each marker M (step S106). At this time, if the multiple markers M are lined up horizontally, the control unit 16 also arranges the projected images I horizontally. If the multiple markers M are lined up vertically, the control unit 16 also arranges the projected images I vertically.

[0062] The control unit 16 functions as the marker detection unit 161 to calculate the image identification information N indicated by the two-dimensional code C added to each marker M (step S108). The control unit 16 functions as the display image acquisition unit 162 to send an image transmission request for projection image data D of the projection image corresponding to each image identification information N to the image server 30 (step S110). The control unit 16 waits until it receives the projection image I from the image server 30 (step S112: NO loop).

[0063] When the projection image I is received from the image server 30 (step S112: YES), the control unit 16 functions as the display control unit 164 to control the projection device 12 to display the projection image I received in step S112 according to the layout determined in step S106 (step S114), and ends the processing of this flowchart. Note that in the above flowchart, step S106 and steps S108 to S112 may be performed in the reverse order, or may be processed simultaneously.

[0064] As described above, the projector 10 according to the first embodiment displays, on the projection surface P, a projection image I1 corresponding to the marker M1 and a projection image I2 corresponding to the marker M2 in an arrangement corresponding to the arrangement of the markers M1 and M2. This allows the user to change the arrangement of the projection images I1 and I2 on the projection surface P by changing the arrangement of the markers M1 and M2, and allows the user to easily specify the projection position of the projection image I.

[0065] Furthermore, the projector 10 aligns the arrangement direction of the projected image I1 and the projected image I2 on the projection surface P with the arrangement direction of the markers M1 and M2 in the captured image, allowing the user to intuitively specify the arrangement of the projected image I.

[0066] Furthermore, marker M1 corresponds to a cutting board O1, which is a first object used in the work performed by the user, and marker M2 corresponds to a saury O2, which is a second object used in the work, and projected image I1 is an image relating to the work using the board O1, and projected image I2 is an image relating to the work using the saury O2. Thus, the user can refer to images relating to the work using object O for each individual object O, thereby improving work efficiency.

[0067] Furthermore, the marker M1 includes object information F3 that indicates to the user that the marker M1 is a marker M corresponding to the cutting board O1, and the marker M2 includes object information F3 that indicates to the user that the marker M2 is a marker M corresponding to the saury O2. Thus, the user can easily understand which object O each of the multiple markers M corresponds to.

[0068] Furthermore, image identification information N for specifying the recording location of the projected image I1 is added to the marker M1, and image identification information N for specifying the recording location of the projected image I2 is added to the marker M2. The projector 10 calculates the image identification information N added to each marker M from the captured image, and acquires the projected image I1 and the projected image I2 from their respective recording locations based on the image identification information N. This allows the user to easily acquire the projected image I corresponding to each marker M.

[0069] For example, a user can place objects O anywhere on the projection surface P, and display a projection image I corresponding to each object O near each object O. In this embodiment, when considering the projection of cooking instructions in a kitchen 20, the names and amounts of ingredients and cooking instructions can be displayed next to a cutting board O1, allowing the user to proceed with cooking while checking this information. In addition, a user can divide the projection area L into multiple projection areas L and adjust the projection size of the projection image I simply by placing a marker M.

[0070] [Second embodiment] A second embodiment of the present invention will be described below. In the following exemplary embodiments, for elements whose actions and functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate. In the first embodiment, the user arranged multiple markers M in a line, and the projection device 12 displayed projected images I corresponding to each marker M in a row aligned with the direction in which the markers M were arranged. In contrast, the second embodiment makes it possible to display images corresponding to the markers M at individual positions based on the marker M.

[0071] 12 is a diagram schematically illustrating the arrangement of markers M in the second embodiment. In the second embodiment, the projection surface P and the imaging area S include the entire cooking counter 26 and a part of the stove 22. As in the first embodiment, the objects O are a cutting board O1, a saury O2, and a frying pan O3. That is, if marker M1 is the first marker, marker M2 is the second marker, and marker M3 is the third marker, the imaging device 11 captures an image including the third marker arranged on the projection surface P in addition to the first and second markers.

[0072] In general, a frying pan O3 is often used on the stove 22 and rarely on the counter 26. For example, as shown in FIG. 13, a cutting board O1 and a saury O2 may be placed on the counter 26, and the frying pan O3 may be placed on the stove 22. In such a case, it is difficult to arrange the markers M corresponding to each object O in a line as in the first embodiment. Therefore, in the second embodiment, a projected image I corresponding to a specific object O can be displayed independently, regardless of the positions of the other markers M.

[0073] 14 and 15 are schematic diagrams showing the appearance of the marker M in the second embodiment. More specifically, FIG. 14A shows the front side of marker M2 corresponding to sanma O2, and FIG. 4B shows the back side of marker M2. A The marker M3 corresponds to the frying pan O3. 15B shows the front side and FIG. 15B shows the back side of marker M3.

[0074] First, the information written on the back of each marker M2, M3 will be described. As shown in Figures 14B and 15B, user-oriented information F (F1 to F4) is added to the back of the marker M. Of these, the contents of the dish name information F1, step number information F2, and object information F3 are the same as those in the first embodiment.

[0075] On the other hand, unlike the first embodiment, the placement information F4 describes the placement type of each marker M. As shown in FIG. 14B, the placement information F4 for marker M2 describes that it is a "permuted placement type," and similarly to the first embodiment, describes the position at which the marker M should be placed relative to the object O. As in the first embodiment, the permuted placement type is a type to which markers M that are expected to be placed alongside other markers M belong. In this embodiment, marker M1 is also of the permuted placement type, as is marker M2. That is, because the cutting board O1 and the saury O2 are expected to be placed alongside each other on the counter 26, the markers M corresponding to these objects O are of the permuted placement type.

[0076] 15B, the placement information F4 of the marker M3 indicates that it is a "free placement type," and also indicates position information indicating the position at which the projected image I is to be displayed relative to the marker M. In the example of FIG. 15B, the position information indicates that the projected image I is to be projected to the "upper left" of the marker M3. In this embodiment, the position information indicates the display position of the projected image I3 relative to the position of the marker M3. That is, the third marker includes position information.

[0077] The free placement type is assigned to a marker M that is assumed to be difficult or inappropriate to place alongside other markers M. In this embodiment, the frying pan O3 is assumed to be placed on the stove 22 away from the cooking table 26 where the other objects O1 and O2 are located, so the marker M3 corresponding to the frying pan O3 is of the free placement type.

[0078] Figures 14A and 15 A As shown in FIG. 1, the front surfaces of the markers M2 and M3 are As with the above, a two-dimensional code C is added. The information to be stored includes information T on the layout type in addition to image identification information N. For example, the two-dimensional code C of the marker M2 shown in FIG. 14A contains information T about the placement type. The permutation arrangement flag indicating that the permutation arrangement type is included. The two-dimensional code C of the marker M3 shown in Fig. 1 contains the information T on the placement type, which is a free placement type. A free placement flag indicating that the image is a free type and a display position of the projected image I relative to the marker M are specified. It contains information specifying the A In the example, the information specifying the display position is "top left" Contains:

[0079] In the second embodiment, the image layout determination unit 163 determines the layout of the projection image I on the projection surface P based on the information T on the layout type. More specifically, when the markers M are of the permuted layout type, the image layout determination unit 163 determines the layout direction of the projection image I based on the layout direction between the markers M. When the markers M are of the free layout type, the image layout determination unit 163 determines the display position of the projection image I based on the position of the marker M in the captured image and information specifying the display position included in the information T on the layout type. As in the first embodiment, the display control unit 164 controls the projection device 12 to project the projection image I onto the projection surface P in accordance with the layout determined by the image layout determination unit 163.

[0080] 13 is a diagram showing a schematic arrangement of the projected image I in the second embodiment. The projected images I1 and I2 corresponding to the markers M1 and M2, which are of the permutation type, are displayed side by side in the horizontal direction. On the other hand, the projected image I3 corresponding to the marker M3, which is of the free arrangement type, is not displayed side by side with the other images but is displayed to the upper left of the marker M3. In other words, if the projected image I3 is the third image, displaying the projected image I further includes displaying the third image corresponding to the third marker M3 at a position corresponding to the position information.

[0081] In the above description, only the marker M3 corresponding to the frying pan O3 is of the free placement type, but this is not limiting, and all the markers M may be of the free placement type.

[0082] As described above, according to the second embodiment, the projector 10 can display the projection image I corresponding to the freely arranged marker M at any position. Therefore, even if the use position of a predetermined object O is far from the use positions of other objects O, the user can display the projection image I corresponding to the predetermined object O at any position, thereby further improving work efficiency.

[0083] [Third embodiment] The third embodiment of the present invention will be described below. In the following exemplary embodiment, for elements whose actions and functions are similar to those of the first or second embodiment, the reference numerals used in the description of the first or second embodiment will be used, and detailed description of each element will be omitted as appropriate.

[0084] In the first and second embodiments, the projection area L of the projection image I is determined based on the position of the marker M, without taking into consideration the position of the object O. In the third embodiment, the projection area L of the projection image I is determined based on the position of the object O on the projection surface P.

[0085] FIG. 16 is a diagram schematically illustrating the appearance of a marker M in the third embodiment. The third embodiment differs from the other embodiments in that no placement information F4 is written on the back of the marker M shown in FIG. 16B. This is because the display position of the projected image I relative to the marker M changes dynamically depending on the presence or absence of a surrounding object O. Also, as in the first embodiment, a two-dimensional code C is added to the front surface of the marker M shown in FIG. 16A. As in the first embodiment, the information obtained when this two-dimensional code C is read is image identification information N.

[0086] Fig. 17 is a block diagram showing the functional configuration of the control unit 16 in the third embodiment. In the third embodiment, an object detection unit 165 is provided in addition to the functional configuration shown in Fig. 3. The object detection unit 165 detects an object placed on the projection surface P from the captured image. That is, when marker M1 is the first marker, marker M2 is the second marker, and object O placed on the projection surface P is the object, the object detection unit 165 detects an image area corresponding to the object placed around the first marker and the second marker from the captured image.

[0087] Various techniques for detecting objects from captured images are known, and therefore detailed explanation of the processing by the object detection unit 165 will be omitted. Generally, since the projection surface P is a surface with a uniform color tone, etc., it is considered that the detection of the object O by the object detection unit 165 can be performed with relatively high accuracy. Note that the object detected by the object detection unit 165 is not limited to the object O associated with the marker M. The object detection unit 165 only needs to be able to detect an item that gets in the way of displaying the projection image I. In other words, the object is not limited to the object O associated with the marker M, but may be any item placed on the projection surface P.

[0088] 18 is a diagram schematically showing an example of the arrangement of markers M and objects O in the third embodiment. In the example of FIG. 18, a cutting board O1, a saury O2, and a frying pan O3 are arranged in this order from left to right on the back side of the projection surface P. Furthermore, markers M1 to M3 corresponding to the objects O1 to O3 are placed on the front side of the object O. The marker detection unit 161 detects the marker M from a captured image of the projection surface P. The object detection unit 165 detects the object O from a captured image of the projection surface P.

[0089] In the third embodiment, the image layout determination unit 163 determines the display position of the projection image I based on the layout of the multiple markers M in the captured image and the layout area of the object O in the captured image. That is, when the projection image I1 is the first image and the projection image I2 is the second image, the image layout determination unit 163 determines the display area for displaying the first image and the second image based on the layout of the first marker and the second marker in the captured image and the area of the image corresponding to the object O.

[0090] More specifically, the image layout determination unit 163 determines the layout direction of the projection image I based on the position of the marker M detected by the marker detection unit 161. For example, when multiple markers M1 to M3 are arranged horizontally as shown in Fig. 18, the image layout determination unit 163 determines that the projection images I1 to I3 corresponding to these markers M1 to M3 are to be displayed arranged horizontally.

[0091] Furthermore, the image layout determination unit 163 determines the projection position of the projection image I based on the layout area of the object O detected by the object detection unit 165. For example, when the objects O are arranged side by side on the projection surface P behind the markers M1 to M3 as shown in Fig. 18, the image layout determination unit 163 determines the area in front of the markers M1 to M3 as the projection area L0.

[0092] As shown in FIG. 19, the image placement determination unit 163 divides the projection area L0 horizontally into three parts, and sets a projection area L1 onto which a projection image I1 corresponding to the marker M1 is projected, a projection area L2 onto which a projection image I2 corresponding to the marker M2 is projected, and a projection area L3 onto which a projection image I3 corresponding to the marker M3 is projected.

[0093] The display control unit 164 controls the projection device 12 to project the projection images I1 to I3 onto the projection surface P in accordance with the layout determined by the image layout determination unit 163. That is, the first image and the second image are displayed based on the display area determined by the image layout determination unit 163.

[0094] Fig. 20 is a diagram schematically showing another example of the arrangement of markers M and objects O in the third embodiment. In the example of Fig. 20, the positions of markers M1 to M3 are substantially the same as in Fig. 18. That is, markers M1 to M3 are arranged side by side in the horizontal direction at approximately the center in the depth direction of projection surface P. On the other hand, the position of object O is different from that of Fig. 18. That is, a cutting board O1, which is an object O corresponding to marker M1, is arranged in front of marker M1. A saury O2, which is an object O corresponding to marker M2, is arranged behind marker M2. A frying pan O3, which is an object O corresponding to marker M3, is arranged behind marker M3.

[0095] In such a case, the image layout determination unit 163 first determines the layout direction of the projection image I based on the position of the marker M detected by the marker detection unit 161. In the example of Fig. 20, the markers M1 to M3 are lined up in the horizontal direction as in Fig. 18, so the image layout determination unit 163 determines that the projection images I1 to I3 corresponding to these markers M1 to M3 should be displayed lined up in the horizontal direction.

[0096] Furthermore, the image placement determination unit 163 determines the projection position of the projection image I based on the position of the object O detected by the object detection unit 165. In Fig. 20, the areas on the projection surface P where no object O is placed are an area L4 located on the opposite side of the cutting board O1 across the marker M1, and an area L5 located on the opposite side of the saury O2 and frying pan O3 across the markers M2 and M3.

[0097] 21, the image layout determination unit 163 determines an area L4 located on the opposite side of the cutting board O1 with the marker M1 in between as the projection area L1 onto which the projection image I1 is projected. The image layout determination unit 163 also divides an area L5 located on the opposite side of the saury O2 and frying pan O3 with the markers M2 and M3 in between into two areas, and determines them as the projection area L2 onto which the projection image I2 is projected and the projection area L3 onto which the projection image I3 is projected. The display control unit 164 controls the projection device 12 to project the projection images I1 to I3 onto the projection surface P in accordance with the layout determined by the image layout determination unit 163.

[0098] 20, the image layout determination unit 163 may divide the area L5 into thirds and set the projection areas L1 to L3. Also, for example, if too many objects O are placed on the projection surface P and the projection area L is not sufficient, the control unit 16 may present the user with a message such as "Please remove the objects from the projection surface or reduce the number of markers."

[0099] As described above, the projector 10 according to the third embodiment determines the display position of the projection image I based on the arrangement area of the object O in the captured image in addition to the positions of the multiple markers M in the captured image. Specifically, for example, if an area of the projection surface P where no object O is arranged is considered to be empty space, and there is empty space adjacent to the marker M, the projection image I is displayed in this empty space. Furthermore, if there is no empty space adjacent to the marker M, the projection image I is displayed in the empty space closest to the marker M. This makes it possible to prevent a decrease in visibility of the projection image I due to the projection area L being set so as to overlap the object O, and allows the user to more reliably grasp the content of the projection image I.

[0100] [Variations] The above-described embodiments can be modified in various ways. Specific modifications that can be applied to the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples can be combined as appropriate within the scope of not mutually contradictory.

[0101] [Variation 1] In this embodiment, the projector 10 displays images relating to cooking procedures in the kitchen 20, but the present invention is not limited to this and may display images relating to other types of work or procedures. For example, the projector 10 may be used to display images relating to craft procedures or how to use tools in DIY (Do It Yourself) projects. Furthermore, for example, the projector 10 may be used to display teaching materials, experimental videos, and the like in distance learning.

[0102] [Variation 2] The projector 10 may be used for signage purposes, for example. FIG. 22 is a diagram schematically illustrating a shop window display using the projector 10. FIG. 22 illustrates an example in which vases V (V1, V2) are displayed on a wall W of a store. The vases V1 and V2 are placed on shelves B1 and B2, respectively, that protrude vertically from the wall W. On the wall W behind the vases V1 and V2, flower images Ix are displayed by the projector 10. By viewing the vases V1 and V2 together with the images Ix, customers of the store can more specifically imagine how the vases V1 and V2 are used. Therefore, it is believed that this is more likely to increase customers' purchasing motivation than simply displaying the vases V1 and V2.

[0103] To display the image as shown in Figure 22, the bottom edge E of the image Ix must be aligned with the height of the openings Va1 and Va2 of the vases V1 and V2. However, the positions of the openings Va1 and Va2 of the vases V1 and V2, that is, the heights H1 and H2, which are the distances from the shelves B1 and B2 to the openings Va1 and Va2, vary depending on the type of vase. Therefore, every time the vase V being displayed changes, the store's window technician must adjust the display position of the image Ix, which is cumbersome.

[0104] Therefore, by using the projector 10 to allow the display position of the image Ix to be specified using a marker M, the display position of the image Ix can be easily adjusted. For example, if the projector 10 is adjusted so that the image Ix is displayed with the bottom end of the marker M aligned with the bottom end E of the image Ix, then the marker M can be attached to the wall W behind the openings Va1 and Va2 of the vases V1 and V2, resulting in a display such as that shown in FIG. 22. Furthermore, the information in the two-dimensional code of the marker M may include information specifying the display position of the image Ix, such as "align the bottom edge of the marker M with the bottom edge of the image Ix." Furthermore, according to the second modification, the content of the image Ix can be changed by replacing the marker M. This makes it easy to change the image Ix depending on the season, specifically, to replace it with an image of a flower appropriate for the season.

[0105] In this way, by using the projector 10, even a user without specialized knowledge can easily change the display position of an image or replace an image, making it easy to implement signage using images.

[0106] [Variation 3] In this embodiment, the projector 10 includes the imaging device 11 and the projection device 12, but these may be separate devices. For example, the imaging device, projection device, and information processing device, which are independent devices, may be connected by wire or wirelessly to form a display system. The information processing device may be a personal computer or a smartphone. Furthermore, the imaging device in the display system may be an imaging device included in an information processing device such as a personal computer or a smartphone.

[0107] [Variation 4] In this embodiment, the image server 30, upon receiving an image transmission request including image identification information N, searches the image database DB for a projection image I corresponding to the image identification information N and returns the image. However, the present invention is not limited to this. For example, the address of a storage location of a predetermined projection image I may be assigned to the marker M as the image identification information N, and the display image acquisition unit 162 may request the image server 30 to transmit the projection image I stored at that address. The address of the storage location of the projection image I may be, for example, a URL (Uniform Resource Locator). [Explanation of symbols]

[0108] 10...projector, 11...imaging device, 12...projection device, 13...operation unit, 14...communication unit, 15...memory unit, 15A...program, 16...control unit, 20...kitchen, 22...cooker, 24...sink, 26...cooking table, 28...wall, 30...image server, 160...imaging control unit, 161...marker detection unit, 162...display image acquisition unit, 163...image placement determination unit, 164...display control unit, 165...object detection unit, P...projection surface, S...imaging area.

Claims

1. A first marker arranged on the projection surface, a second marker arranged on the projection surface, and a capturing an image including an object disposed on a projection plane; A first arrangement area of the first marker and a second arrangement area of the second marker in the captured image detecting a region and a third placement region of the object; The third arrangement is determined based on the first arrangement region, the second arrangement region, and the third arrangement region. determining a display area that does not overlap with the placement area and displays the first image and the second image; If the display area is not sufficient, information to the effect that the display area is not sufficient is output. And, When the display area is sufficient, the first image corresponding to the first marker and the and displaying the second image corresponding to the second marker in the display area; The information that the display area is insufficient may be displayed as information to move the object or Or, the information is information to remove the first marker or the second marker from the projection surface. How to display.

2. In the displaying, The arrangement direction of the first image and the second image on the projection surface is The arrangement direction of the first marker and the second marker coincides with the arrangement direction of the first marker and the second marker. The display method according to claim 1.

3. the first marker corresponds to a first object used in a task performed by a user; the second marker corresponds to a second object used in the task; the first image is an image relating to the content of the work using the first object, the second image is an image relating to the content of the work using the second object; 3. The display method according to claim 1 or 2.

4. The first marker is a marker corresponding to the first object. and including an indicator indicating to the user that The second marker is a marker corresponding to the second object. and including an indication to the user that The display method according to claim 3.

5. The first marker is provided with first information used to identify the recording location of the first image. It has been added, The second marker is provided with second information used to identify the recording location of the second image. It has been added, acquiring the first information and the second information based on the captured image; acquiring the first image from a recording location of the first image based on the first information; 、 acquiring the second image from a recording location of the second image based on the second information; , further comprising: The display method according to any one of claims 1 to 4.

6. A first marker arranged on the projection surface, a second marker arranged on the projection surface, and a an imaging device that captures an image including an object placed on a projection plane; A first arrangement area of the first marker and a second arrangement area of the second marker in the captured image a third placement area of the object; The third arrangement is determined based on the first arrangement region, the second arrangement region, and the third arrangement region. a control device that determines a display area in which the first image and the second image are displayed without overlapping with the placement area; If the display area is insufficient, output information indicating that the display area is insufficient; When the display area is sufficient, the first image corresponding to the first marker and the a projection device that displays the second image corresponding to the second marker in the display area; Equipped with The information that the display area is insufficient may be displayed as information to move the object or Or, the information is information to remove the first marker or the second marker from the projection surface. A display system.

Images