Control device, control method, and control program

Abstract

Provided are a control device, a control method, and a control program for enabling notification related to particular content associated with a displayed image.　A control device (4) generates third image data based on first image data and second image data, and outputs the third image data. A frame image (60A) based on the second image data in a projected image (70) represented by the third image data is an image indicating that particular content is associated with the projected image (70). The frame image (60A) is also an image that surrounds an image (50A) to be projected that is based on the first image data in the projected image (70).

Description

Technical Field

[0001] The present invention relates to a control device, a control method, and a control program.

Background Art

[0002] In Patent Document 1, an image of a position marker used to specify the display position of a virtual object in augmented reality technology is acquired, and a video for projection mapping is completed by incorporating it into a video for projection mapping. The video is projected by a video projection device toward a showroom so that an image of the position marker is displayed on the floor of the showroom. A video creation device is described.

[0003] In Patent Document 2, in projection mapping in which video content is projected onto a three-dimensional object by a projector in consideration of the three-dimensional structure of the three-dimensional object, when a user captures the video content with a camera mounted on a smart device, it is described that AR content is superimposed and displayed in the captured video.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

[0005] One embodiment according to the technology of the present disclosure provides a control device, a control method, and a control program capable of notifying a specific content associated with a display image.

Means for Solving the Problems

[0006] A control device according to one aspect of the present invention is a control device comprising a processor, wherein the processor generates third image data based on first image data and second image data, outputs the third image data, and among the third image represented by the third image data, the second image based on the second image data is an image indicating that specific content is associated with the third image, and surrounds the first image based on the first image data among the third image data.

[0007] A control device according to one aspect of the present invention is a control device comprising a processor, wherein the processor generates third image data based on first image data and second image data, outputs the third image data, and the second image based on the second image data among the third images represented by the third image data is an image indicating the position of an image of a first object to which specific content is associated, which is included in the first image based on the first image data among the third images.

[0008] In one aspect of the present invention, a control method is provided in which the processor of a control device generates third image data based on first image data and second image data, outputs the third image data, and among the third image data represented by the third image data, the second image data based on the second image data is an image indicating that specific content is associated with the third image data, and surrounds the first image data based on the first image data among the third image data.

[0009] In one aspect of the present invention, a control method is provided in which the processor of a control device generates third image data based on first image data and second image data, outputs the third image data, and among the third images represented by the third image data, the second image based on the second image data is an image indicating the position of an image of a first object to which specific content is associated, which is included in the first image based on the first image data among the third images.

[0010] A control program in one aspect of the present invention is a control program that causes the processor of a control device to perform a process that generates a third image data based on a first image data and a second image data, and outputs the third image data, wherein the second image data based on the second image data, among the third image data represented by the third image data, is an image that indicates that specific content is associated with the third image data, and surrounds the first image data based on the first image data among the third image data.

[0011] A control program in one aspect of the present invention is a control program that causes the processor of a control device to perform a process that generates a third image data based on a first image data and a second image data, and outputs the third image data, wherein the second image data based on the second image data, among the third image data represented by the third image data, is an image that indicates the position of an image of a first object to which specific content is associated, which is included in the first image data based on the first image data among the third image data.

[0012] According to the present invention, it is possible to provide a control device, a control method, and a control program that can provide notifications regarding specific content associated with a displayed image. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of the projection system 100 of Embodiment 1. [Figure 2] This is a diagram showing an example of the configuration of the projection device 10. [Figure 3] This is a schematic diagram showing an example of the internal structure of the projection unit 1. [Figure 4] This figure shows an example of the external configuration of the projection device 10. [Figure 5] Figure 4 is a schematic cross-sectional view of a portion of the optical unit 106 of the projection device 10 shown in Figure 4. [Figure 6] This figure shows an example of the hardware configuration of the information terminal 80. [Figure 7] This figure shows an example of image projection and augmented reality content playback. [Figure 8] FIG. showing another example of projection by the projection device 10 and reproduction of augmented reality content. [Figure 9] FIG. showing an example of input of information on a projection area. [Figure 10] FIG. showing an example of image projection and reproduction of augmented reality content based on input of information on a projection area. [Figure 11] FIG. showing an example of a frame image 60 including identification information. [Figure 12] FIG. showing an example of guidance of the direction of a vehicle image 51A based on identification information 61. [Figure 13] FIG. (part 1) showing an example of reproduction of augmented reality content according to the attributes of the augmented reality content. [Figure 14] FIG. (part 2) showing an example of reproduction of augmented reality content according to the attributes of the augmented reality content. [Figure 15] FIG. showing an example of a frame image 60A indicating the position of an image of a first object with which augmented reality content is associated. [Figure 16] FIG. (part 1) showing an example of tracking of the frame image 60A when the image of the first object moves. [Figure 17] FIG. (part 2) showing an example of tracking of the frame image 60A when the image of the first object moves. [Figure 18] FIG. (part 1) showing a specific example of the frame image 60A based on information on a display area and the position of the vehicle image 51. [Figure 19] FIG. (part 2) showing a specific example of the frame image 60A based on information on a display area and the position of the vehicle image 51. [Figure 20] FIG. (part 3) showing a specific example of the frame image 60A based on information on a display area and the position of the vehicle image 51. [Figure 21] FIG. (part 4) showing a specific example of the frame image 60A based on information on a display area and the position of the vehicle image 51. [Figure 22] Schematic diagram showing another external configuration of the projection device 10. [Figure 23]Figure 22 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Figure 22. [Figure 24] This figure shows a modified example of the projection system 100. [Figure 25] This figure shows an example of the hardware configuration of the information terminal 110. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0015] (Embodiment 1) <Projection system 100 of Embodiment 1> Figure 1 shows an example of the projection system 100 of Embodiment 1. As shown in Figure 1, the projection system 100 includes a projection device 10 and an information terminal 80. The control device in the present invention is applied to the projection device 10, for example. The projection device 10 is a projection device capable of projecting onto a projection target object 6.

[0016] The projection object 6 is an object such as a screen that has a projection surface on which the projected image is displayed by the projection device 10. In the example shown in Figure 1, the projection surface of the projection object 6 is a rectangular plane. The top, bottom, left, and right of the projection object 6 in Figure 1 are assumed to be the top, bottom, left, and right of the actual projection object 6.

[0017] The projection range 11, shown by the dashed line, is the area of ​​the projection object 6 that is illuminated by the projection device 10. The projection range 11 is part or all of the projection range that can be projected by the projection device 10. In the example shown in Figure 1, the projection range 11 is rectangular.

[0018] The information terminal 80 is an information terminal such as a smartphone or tablet, having an imaging unit (for example, an imaging module 85 in Figure 6) and a display unit 86. The information terminal 80 stores (installs) an application that, when a specific image is included in the image captured by the imaging unit, displays a superimposed image on the image by superimposing predetermined augmented reality (AR) content on the image using the display unit 86.

[0019] <Configuration of projection device 10> Figure 2 shows an example of the configuration of the projection device 10. As shown in Figure 2, the projection device 10 comprises, for example, a projection unit 1, a control device 4, and an operation reception unit 2. The projection device 10 may further include a communication unit 5. The projection unit 1 is composed of, for example, a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following description, the projection unit 1 will be described assuming that it is a liquid crystal projector.

[0020] Control device 4 is an example of a control device of the present invention. Control device 4 controls projection by projection device 10. Control device 4 is a device that includes a control unit composed of various processors, a communication interface (not shown) for communicating with each unit, and a storage medium 4a such as a hard disk, SSD (Solid State Drive), or ROM (Read Only Memory), and comprehensively controls the projection unit 1. The various processors of the control unit of control device 4 include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and performs various processes; a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array); or a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to execute a specific process, such as an ASIC (Application Specific Integrated Circuit).

[0021] The structure of these various processors is, more specifically, an electrical circuit formed by combining circuit elements such as semiconductor elements. The control unit of the control device 4 may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0022] The operation reception unit 2 detects user instructions (user commands) by receiving various operations from the user. The operation reception unit 2 may be a button, key, joystick, etc., provided on the control device 4, or it may be a receiving unit, etc., that receives signals from a remote controller that remotely operates the control device 4.

[0023] The communication unit 5 is a communication interface that enables communication with other devices. The communication unit 5 may be a wired communication interface for wired communication or a wireless communication interface for wireless communication.

[0024] The projection unit 1, control device 4, operation reception unit 2, and communication unit 5 may be implemented by, for example, a single device (see, for example, Figures 4 and 5). Alternatively, the projection unit 1, control device 4, operation reception unit 2, and communication unit 5 may be implemented by multiple devices that can cooperate by communicating with each other.

[0025] <Internal structure of projection unit 1> Figure 3 is a schematic diagram showing an example of the internal configuration of the projection unit 1. The projection unit 1 of the projection device 10 shown in Figure 2 comprises a light source 21, a light modulation unit 22, a projection optical system 23, and a control circuit 24, as shown in Figure 3. The light source 21 includes a light-emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0026] The light modulation unit 22 consists of three liquid crystal panels (light modulation elements) that modulate each color of light, which is emitted from the light source 21 and separated into three colors (red, blue, and green) by a color separation mechanism (not shown), based on image information, and emit each color image; and a dichroic prism that mixes the color images emitted from the three liquid crystal panels and emits them in the same direction. Alternatively, each of the three liquid crystal panels may be equipped with red, blue, and green filters, and the white light emitted from the light source 21 may be modulated by each liquid crystal panel to emit each color image.

[0027] The projection optical system 23 receives light from the light source 21 and the light modulation unit 22, and is composed of, for example, a relay optical system that includes at least one lens. The light that passes through the projection optical system 23 is projected onto the projection object 6.

[0028] Of the projection target object 6, the area illuminated by light that passes through the entire range of the light modulation unit 22 becomes the projectable range that can be projected by the projection unit 1. Within this projectable range, the area illuminated by light that actually passes through from the light modulation unit 22 becomes the projection range 11 of the projection unit 1. For example, by controlling the size, position, and shape of the area through which light passes within the light modulation unit 22, the size, position, and shape of the projection range of the projection unit 1 can be changed within the projectable range.

[0029] The control circuit 24 controls the light source 21, the light modulation unit 22, and the projection optical system 23 based on the display data input from the control device 4, thereby projecting an image based on this display data onto the projection target object 6. The display data input to the control circuit 24 consists of three parts: red display data, blue display data, and green display data.

[0030] Furthermore, the control circuit 24 expands or shrinks the projection range of the projection unit 1 by changing the projection optical system 23 based on commands input from the control device 4. The control device 4 may also move the projection range of the projection unit 1 by changing the projection optical system 23 based on user operations received by the operation reception unit 2.

[0031] Furthermore, the projection device 10 includes a shift mechanism that mechanically or optically moves the projection range of the projection unit 1 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is the region in which the projected light incident on the projection optical system 23 passes through the projection optical system 23 appropriately in terms of light intensity reduction, color separation, peripheral curvature, etc.

[0032] The shift mechanism is implemented by at least one of the following: an optical shift mechanism that performs optical system shifting, and an electronic shift mechanism that performs electronic shifting.

[0033] The optical system shift mechanism is, for example, a mechanism that moves the projection optical system 23 in a direction perpendicular to the optical axis (see, for example, Figures 5 and 23), or a mechanism that moves the light modulation unit 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. Alternatively, the optical system shift mechanism may be a combination of moving the projection optical system 23 and moving the light modulation unit 22.

[0034] The electronic shift mechanism is a mechanism that performs a pseudo-projection range shift by changing the range through which light is transmitted in the optical modulation section 22.

[0035] Furthermore, the projection device 10 may be equipped with a projection direction changing mechanism that moves the projection range together with the image circle of the projection optical system 23. The projection direction changing mechanism is a mechanism that changes the projection direction of the projection unit 1 by changing the orientation of the projection unit 1 through mechanical rotation (see, for example, Figure 23).

[0036] <Mechanical configuration of projection device 10> Figure 4 shows an example of the external configuration of the projection device 10. Figure 5 is a schematic cross-sectional view of a part of the optical unit 106 of the projection device 10 shown in Figure 4. Figure 5 shows a cross-section along the optical path of the light emitted from the main body 101 shown in Figure 4.

[0037] As shown in Figure 4, the projection device 10 comprises a main body 101 and an optical unit 106 that protrudes from the main body 101. In the configuration shown in Figure 4, the operation reception unit 2, the control device 4, the light source 21, the light modulation unit 22, and the control circuit 24 in the projection unit 1, and the communication unit 5 are provided in the main body 101. The projection optical system 23 in the projection unit 1 is provided in the optical unit 106.

[0038] The optical unit 106 includes a first member 102 supported by the main body 101. The optical unit 106 may be configured to be detachably attached to (in other words, interchangeable) the main body 101.

[0039] As shown in Figure 5, the main body 101 has a housing 15 in which an opening 15a for allowing light to pass through is formed in the portion connected to the optical unit 106.

[0040] As shown in Figure 4, the housing 15 of the main body 101 is provided with a light source 21 and an optical modulation unit 12, which includes an optical modulation unit 22 (see Figure 3) that spatially modulates the light emitted from the light source 21 based on input image data to generate an image. The light emitted from the light source 21 is incident on the optical modulation unit 22 of the optical modulation unit 12, spatially modulated by the optical modulation unit 22, and then emitted.

[0041] As shown in Figure 5, the image formed by the light spatially modulated by the optical modulation unit 12 passes through the aperture 15a of the housing 15 and enters the optical unit 106, where it is projected onto the projection target 6, making the image G1 visible to the observer.

[0042] As shown in Figure 5, the optical unit 106 comprises a first member 102 having a hollow portion 2A connected to the inside of the main body 101, a first optical system 121 arranged in the hollow portion 2A, a lens 34, and a first shift mechanism 105.

[0043] The first member 102 is a member whose cross-sectional shape is, for example, rectangular, and has openings 2a and 2b formed on planes parallel to each other. The first member 102 is supported by the main body 101 with opening 2a positioned opposite opening 15a of the main body 101. Light emitted from the light modulation section 22 of the light modulation unit 12 of the main body 101 is incident on the hollow section 2A of the first member 102 through openings 15a and 2a.

[0044] The direction of incidence of light entering the hollow section 2A from the main body 101 is described as direction X1, the opposite direction of direction X1 is described as direction X2, and directions X1 and X2 are collectively referred to as direction X. In Figure 5, the direction from the front to the back of the paper and the opposite direction are described as direction Z. Of the directions Z, the direction from the front to the back of the paper is described as direction Z1, and the direction from the back to the front of the paper is described as direction Z2.

[0045] Furthermore, the direction perpendicular to directions X and Z is denoted as direction Y, and within direction Y, the upward direction in Figure 5 is denoted as direction Y1, and the downward direction in Figure 5 is denoted as direction Y2. In the example in Figure 5, the projection device 10 is positioned such that direction Y2 is the vertical direction.

[0046] The projection optical system 23 shown in Figure 3 is composed of a first optical system 121 and a lens 34 in the example shown in Figure 5. Figure 5 shows the optical axis K of this projection optical system 23. The first optical system 121 and the lens 34 are arranged along the optical axis K in this order, starting from the side of the light modulation unit 22.

[0047] The first optical system 121 includes at least one lens and guides light traveling in direction X1 incident from the main body 101 to the first member 102 to the lens 34.

[0048] The lens 34 is positioned at the end of the first member 102 so as to block the opening 2b formed at the end on the direction X1 side. The lens 34 projects light incident from the first optical system 121 onto the projection target 6.

[0049] The first shift mechanism 105 is a mechanism for moving the optical axis K of the projection optical system (in other words, the optical unit 106) in a direction perpendicular to the optical axis K (direction Y in Figure 5). Specifically, the first shift mechanism 105 is configured to change the position of the first member 102 in direction Y relative to the main body 101. The first shift mechanism 105 may be used to move the first member 102 manually or electrically.

[0050] Figure 5 shows the state in which the first member 102 has been moved to its maximum extent toward direction Y1 by the first shift mechanism 105. From this state shown in Figure 5, the first member 102 moves toward direction Y2 by the first shift mechanism 105, which changes the relative position between the center of the image formed by the light modulation unit 22 (in other words, the center of the display surface) and the optical axis K, thereby shifting (translating) the image G1 projected onto the projection object 6 toward direction Y2.

[0051] Alternatively, the first shift mechanism 105 may be a mechanism that moves the light modulation unit 22 in the direction Y instead of moving the optical unit 106 in the direction Y. In this case as well, the image G1 projected onto the projection target 6 can be moved in the direction Y.

[0052] <Hardware configuration of information terminal 80> Figure 6 shows an example of the hardware configuration of the information terminal 80. The information terminal 80 shown in Figure 1 comprises a processor 81, memory 82, communication interface 83, user interface 84, and imaging module 85, as shown in Figure 5. The processor 81, memory 82, communication interface 83, user interface 84, and imaging module 85 are connected, for example, by a bus 89.

[0053] The processor 81 is a circuit that performs signal processing, and is, for example, a CPU that controls the entire information terminal 80. The processor 81 may also be implemented using other digital circuits such as an FPGA or a DSP (Digital Signal Processor). Furthermore, the processor 81 may be implemented using a combination of multiple digital circuits.

[0054] Memory 82 includes, for example, main memory and auxiliary memory. Main memory is, for example, RAM (Random Access Memory). Main memory is used as the work area of ​​processor 81.

[0055] Auxiliary memory is non-volatile memory such as magnetic disks, optical disks, or flash memory. Various programs for operating the information terminal 80 are stored in the auxiliary memory. The programs stored in the auxiliary memory are loaded into main memory and executed by the processor 81.

[0056] Furthermore, the auxiliary memory may include portable memory that can be removed from the information terminal 80. Portable memory may include USB (Universal Serial Bus) flash drives, SD (Secure Digital) memory cards, and external hard disk drives.

[0057] The communication interface 83 is a communication interface that communicates with external devices of the information terminal 80. The communication interface 83 is controlled by the processor 81. The communication interface 83 may be a wired communication interface for wired communication, a wireless communication interface for wireless communication, or it may include both a wired communication interface and a wireless communication interface.

[0058] The user interface 84 includes, for example, an input device that accepts user input and an output device that outputs information to the user. The input device can be implemented by, for example, a pointing device (e.g., a mouse), a key (e.g., a keyboard), or a remote control. The output device can be implemented by, for example, a display or a speaker. Alternatively, the input and output devices may be implemented by a touch panel. The user interface 84 is controlled by the processor 81. The display unit 86 of the information terminal 80 shown in Figure 1 is implemented by, for example, a touch panel included in the user interface 84.

[0059] The imaging module 85 is an imaging unit that includes an imaging lens and an image sensor. For example, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor can be used as the image sensor.

[0060] <Image projection and augmented reality content playback> Figure 7 shows an example of image projection and playback of augmented reality content. The projection device 10 receives first image data representing the image to be projected 50. In the example in Figure 7, the image to be projected 50 is an image that includes a car image 51, which is a photograph or illustration of a sports car. The projection device 10 stores second image data representing the frame image 60 in the storage medium 4a. In the example in Figure 7, the frame image 60 is a frame image with a hollow rectangular shape.

[0061] The control device 4 of the projection device 10 generates a third image data representing the projection image 70 based on the first image data and the second image data. The control device 4 then outputs the generated third image data to the projection unit 1 of the projection device 10. As a result, the projection image 70 based on the third image data is projected onto the projection range 11.

[0062] The projected image 70 is an example of a third image represented by the third image data. In the example in Figure 7, the projected image 70 is an image in which the projection target image 50 is superimposed inside a frame image 60, which is a hollow rectangular frame image. The projected image 70 includes the projection target image 50A and the frame image 60A.

[0063] The projection target image 50A is the first image in the projection image 70, based on the first image data. The car image 51A is the image in the projection target image 50A, based on the car image 51. In the example in Figure 7, the projection image 70 is generated so that the projection target image 50 is contained within the frame image 60, so the projection target image 50A is identical to the projection target image 50.

[0064] The projected image 70 is not limited to this; for example, it may be obtained by replacing the outer region of the projection target image 50 (for example, a region of a certain width inside the projection target image 50, along the outer perimeter of the projection target image 50) with the frame image 60. In this case, the projection target image 50A in the projected image 70 is the image of the projection target image 50 excluding the outer region of the projection target image 50.

[0065] The frame image 60A is a second image in the projected image 70, based on the second image data. In the example in Figure 7, the projected image 70 is generated so that the frame image 60 is included directly outside the projection target image 50, so the frame image 60A is identical to the frame image 60.

[0066] The imaging range 85a indicates the imaging range captured by the imaging module 85 of the information terminal 80. In other words, an image representing the imaging range 85a is obtained by imaging with the imaging module 85 of the information terminal 80.

[0067] The information terminal 80 stores (installs) an application that, if the image captured by the imaging module 85 includes a car image 51A, displays the captured image along with a superimposed image 90, which is created by superimposing a glossy image 91 onto a specific portion of the car image 51A in the image, via the display unit 86. In the example in Figure 7, the entire projected image 70 is captured by the information terminal 80, and the superimposed image 90, which is created by superimposing the glossy image 91 onto the portion of the car image 51A in the projected image 70, is displayed by the display unit 86.

[0068] Frame image 60A is an image that indicates that augmented reality content is associated with the projected image 70. Augmented reality content is an example of specific content. For example, the user of the information terminal 80 is informed in advance that augmented reality content will be played when an image enclosed by frame image 60A is captured from the images projected onto the projection target object 6.

[0069] The user of the information terminal 80 sees the frame image 60A of the projected image 70 projected from the projection device 10, recognizes that augmented reality content is associated with the projected image 70, and takes an image of the projected image 70 using the information terminal 80. As a result, the display unit 86 displays an overlaid image 90, which is the projected image 70 with a glossy image 91 superimposed on it.

[0070] Furthermore, in the projected image 70, the frame image 60A is positioned so as to be in contact with at least a portion of the outer perimeter of the projection target image 50A (first image) and not to include the internal region of the projection target image 50A. The outer perimeter is the edge that has no width. In the example in Figure 7, the frame image 60A surrounds the projection target image 50A so as to be in contact with the entire outer perimeter of the projection target image 50A.

[0071] This makes it possible to suppress the decrease in visibility of the original projected content, the projected image 50A, while notifying the observer of the projected image 70 that augmented reality content is associated with the projected image 70.

[0072] Note that the frame image 60A only needs to surround the projection target image 50A and does not need to be in contact with it. For example, the projection image 70 may be an image in which the frame image 60A is positioned to surround the projection target image 50A with a small gap between them. Surrounding the projection target image 50A also includes sandwiching the projection target image 50A, as in the example in Figure 8.

[0073] Figure 8 shows another example of projection and playback of augmented reality content by the projection device 10. In the example in Figure 8, the frame image 60 consists of two L-shaped frame images. The projected image 70 is an image in which the frame image 60 is superimposed so as to touch the upper right and lower left corners of the rectangular projection target image 50. That is, in the projected image 70, the frame image 60A surrounds the projection target image 50A so as to touch a part of its outer perimeter.

[0074] <Image projection and augmented reality content playback based on input information from the projection area> Figure 9 shows an example of inputting information for the projection area. Figure 10 shows an example of image projection and playback of augmented reality content based on the input of information for the projection area. In Figures 9 and 10, the projection area 6a is the projection surface of the object 6 to be projected onto. In the example in Figure 9, the projection area 6a is L-shaped, not rectangular.

[0075] The control device 4 of the projection device 10 may accept input of information about the shape of the projection area 6a of the projection target object 6. The projection area 6a is an example of a display area where the projected image 70 (third image) is displayed. The input of information about the shape of the projection area 6a is performed, for example, via the operation reception unit 2.

[0076] Furthermore, in the example shown in Figure 9, the projection device 10 stores information indicating the width and color of the frame, etc., in the storage medium 4a as second image data for generating the frame image 60. Thus, the second image data is not limited to image data representing a predetermined frame image 60, but may also be data such as the color and width of the frame image for generating the frame image 60.

[0077] As shown in Figure 10, the control device 4 generates third image data based on the input information about the shape of the projection area 6a, first image data representing the projection target image 50, and second image data indicating the width and color of the frame. Specifically, the control device 4 generates a frame image 60 that follows the shape of the projection area 6a indicated by the input information and has the width and color indicated by the second image data. Then, the control device 4 generates third image data representing the projection image 70 with the projection target image 50 superimposed inside the frame image 60, and outputs the generated third image data to the projection unit 1.

[0078] As a result, a projection image 70 with a shape that matches the projection area 6a of the projection target object 6 is projected from the projection device 10 onto the projection target object 6.

[0079] <Other examples of how to generate frame image 60> The control device 4 may generate a frame image 60 based on the color of the input first image data (projection target image 50). For example, the control device 4 may determine the color of the portion of the projection target image 50 adjacent to the outer edge of the projection target image 50 (outer region) and generate a frame image 60 of a different color from the determined color.

[0080] This prevents the outer region of the projected image 50A in the projected image 70 from having the same color as the frame image 60A, thereby improving the visibility of the frame image 60A in the projected image 70. This helps to prevent situations where an observer of the projected image 70 does not realize that augmented reality content is associated with the projected image 70.

[0081] <Configuration in which augmented reality content is associated with frame image 60 (second image)> In this description, an application is stored in the information terminal 80 that displays a superimposed image 90, which is created by superimposing a glossy image 91 onto a specific portion of the car image 51 in the captured image, if the captured image includes a car image 51A. In other words, a configuration has been described in which augmented reality content is associated with the car image 51A (projection target image 50). However, the configuration is not limited to this. For example, a configuration in which augmented reality content is associated with a frame image 60 (second image) is also possible.

[0082] Specifically, an application stored in the information terminal 80 may be configured to display a superimposed image 90, which is created by superimposing a glossy image 91 onto a specific location inside the frame image 60 (the portion of the car image 51A) within the captured image, if the captured image includes a frame image 60.

[0083] Furthermore, even if the captured image contains only a portion of the frame image 60, the application stored in the information terminal 80 may be configured to display a superimposed image 90 in which the glossy image 91 is superimposed on the specific position by deriving the positional relationship between the portion of the frame image 60 and the specific position based on the shape of the portion of the frame image 60.

[0084] <Frame image 60 containing identification information> Figure 11 shows an example of a frame image 60 that includes identification information. In a configuration in which augmented reality content (e.g., a glossy image 91) is associated with the frame image 60, the frame image 60 may include identification information 61, as shown in Figure 11.

[0085] For example, the identification information 61 consists of multiple distinct images placed at multiple positions on the frame image 60. In the example shown in Figure 11, 20 pieces of identification information 61 are placed on the frame image 60 at intervals from each other, and in Figure 11, only one of these pieces of identification information 61 is labeled with the code "61". The identification information 61 is an example of identification information that can identify the relative position of the identification information 61 in the projected image 70 based on the frame image 60.

[0086] The 20 identification information pieces 61 shown in Figure 11 are, for example, different one-dimensional and two-dimensional codes, and the arrangement of the 20 identification information pieces 61 in the frame image 60 is predetermined. As a result, the information terminal 80 can determine the position of each identification information piece 61 included in the captured image in the projected image 70, that is, the relative position of that identification information piece 61 in the projected image 70.

[0087] As a result, even if the image captured by the imaging module 85 only contains a portion of the frame image 60A, the information terminal 80 can display a superimposed image 90, which is created by superimposing a glossy image 91 onto the captured image, based on the identification information 61 contained in that portion of the frame image 60A.

[0088] Furthermore, the identification information 61 is not limited to one-dimensional codes or two-dimensional codes; it may also be an image having a predetermined unique color or pattern, or an image of characters, numbers, symbols, etc.

[0089] Furthermore, even in a configuration where augmented reality content (e.g., a glossy image 91) is associated with the projection target image 50 (first image), the information terminal 80 may, for example, display a superimposed image 90 in which the glossy image 91 is superimposed on the captured image, based on the identification information 61 of the frame image 60, if the car image 51A is not included in the captured image or if only a part of the car image 51A is included in the captured image.

[0090] <Directional guidance for vehicle image 51A based on identification information 61> Figure 12 shows an example of directional guidance for a car image 51A based on identification information 61. In the example in Figure 12, augmented reality content (e.g., a glossy image 91) is associated with the car image 51A of the projection target image 50, but the imaging range 85a does not include the car image 51, while the imaging range 85a includes a portion of the frame image 60A.

[0091] In this case, the information terminal 80 may determine the direction of the car image 51A based on the identification information 61 contained in a part of the frame image 60A, and display a superimposed image 90 in which an arrow image 94 indicating the direction of the determined car image 51A is superimposed on the captured image. Furthermore, if the car image 51A is included in the imaging range 85a, the information terminal 80 may display a superimposed image 90 in which a glossy image 91 is superimposed on the captured image instead of the arrow image 94.

[0092] As a result, the user of the information terminal 80 refers to the arrow image 94 and performs an operation to change the imaging range 85a to include the area to the upper right of the current imaging range 85a. This operation may include, for example, changing the orientation of the information terminal 80, moving the information terminal 80 away from the projection target 6, or zooming out the imaging module 85.

[0093] As a result, the imaging range 85a includes the car image 51A, and a superimposed image 90 including the car image 51A and the glossy image 91 is displayed.

[0094] <Playback of augmented reality content according to its attributes> Figures 13 and 14 illustrate examples of augmented reality content playback according to the attributes of the augmented reality content. In the examples in Figures 13 and 14, a smoke image 92 and a meteor image 93 are associated with the projected image 70. The smoke image 92 is an example of first augmented reality content that is played when the car image 51A (first object) included in the projection target image 50A is included in the captured image (image field of view). The meteor image 93 is an example of second augmented reality content that is played regardless of whether the car image 51A is included in the captured image or not.

[0095] For example, the information terminal 80 operates as follows, according to the functions of the above application. That is, the information terminal 80 has pre-stored image data representing smoke image 92 and image data representing meteor image 93. The smoke image 92 is associated with attribute information indicating that it is a first augmented reality content that is played back when the car image 51A is included in the captured image. The meteor image 93 is associated with attribute information indicating that it is a second augmented reality content that is played back regardless of whether the car image 51A is included in the captured image or not.

[0096] In the example shown in Figure 13, since the imaging range 85a includes the car image 51A, the imaging module 85 of the information terminal 80 captures an image containing the car image 51A. In this case, the information terminal 80 displays a superimposed image 90, which is the captured image with the smoke image 92 and the meteor image 93 superimposed on it, on the display unit 86.

[0097] In the example shown in Figure 14, since the vehicle image 51A is not included in the imaging range 85a, the imaging module 85 of the information terminal 80 obtains an image that does not include the vehicle image 51A. In this case, the information terminal 80 displays a superimposed image 90, which is the image with only the meteor image 93 superimposed on it, on the display unit 86.

[0098] As shown in Figures 13 and 14, the augmented reality content associated with the projected image 70 may include a first augmented reality content (for example, a smoke image 92 displayed only when the car image 51A is visible) that is played back when the car image 51A (first object) included in the projection target image 50 (first image) is within the imaging field of view, and a second augmented reality content (for example, a meteor image 93 displayed even when the car image 51A is not visible) that is played back regardless of whether the car image 51A is within the imaging field of view or not.

[0099] In the examples shown in Figures 13 and 14, the frame image 60 may also include identification information 61, as shown in Figure 11. This allows the information terminal 80 to display a superimposed image 90, in which the meteor image 93 is superimposed on the captured image, based on the identification information 61 of the frame image 60, for example, when the car image 51A of the projection target image 50 is not included in the captured image (for example, the example in Figure 14), or when only a part of the car image 51A is included in the captured image.

[0100] (Embodiment 2) The differences between Embodiment 2 and Embodiment 1 will now be described. In Embodiment 1, the frame image 60A (second image) in the projected image 70 (third image) is described as an image that indicates that specific content is associated with the projected image 70, and is an image that surrounds the projected target image 50A (first image) in the projected image 70. However, the configuration is not limited to this.

[0101] In Embodiment 2, we will describe a configuration in which the frame image 60A (second image) in the projected image 70 (third image) is an image that indicates the position of the image of the first object to which augmented reality content is associated, which is included in the projection target image 50A.

[0102] <Position of the image of the first object to which the augmented reality content is associated> Figure 15 shows an example of a frame image 60A that indicates the position of the image of the first object to which augmented reality content is associated. In the example in Figure 15, the frame image 60 is used as an image that indicates the position of the car image 51A (image of the first object).

[0103] For example, the control device 4 stores object position information indicating the region (position) of the car image 51 in the projection target image 50. Based on this object position information, it generates a projected image 70 (third image) by superimposing the frame image 60 onto the projection target image 50 so that the frame image 60 surrounds the car image 51 in the projection target image 50.

[0104] In this case, the portion of the projected image 70 over which the frame image 60 is superimposed is the frame image 60A based on the second image data (frame image 60). Furthermore, the portion of the projected image 70 inside the frame image 60A and the portion outside the frame image 60A are the projection target image 50A based on the first image data (projection target image 50).

[0105] This allows the user of the projected image 70, who is an observer of the projected image 70, to be notified of the position of the car image 51A (image of the first object) to which the glossy image 91 (augmented reality content) is associated. The user of the information terminal 80 can recognize from the frame image 60A of the projected image 70 that the augmented reality content is associated with the car image 51A inside the frame image 60A, and takes an image with the information terminal 80 including the car image 51A in the imaging range 85a.

[0106] <Following of frame image 60A when the image of the first object moves> Figures 16 and 17 show an example of how the frame image 60A tracks the image of the first object as it moves. The projection target image 50 may be video content. For example, the projection target image 50 is video content in which a car image 51 moves within the projection target image 50. Figure 16 shows the positions of the car image 51 at consecutive times t1 to t3. In the example of Figure 16, the projection target image 50 is video content in which the car image 51 moves from left to right according to the elapsed playback time.

[0107] The control device 4 of the projection device 10 can acquire correspondence information between the elapsed time (e.g., time t1 to t3) of the playback of the projection target image 50 and the position of the car image 51 in the projection target image 50. This correspondence information may be stored in the storage medium 4a of the control device 4, or it may be stored in another device with which the control device 4 can communicate. Based on this correspondence information, the control device 4 may generate a projection image 70 that includes a frame image 60A that moves in accordance with the movement of the car image 51A, as shown in Figure 17.

[0108] Furthermore, the projection target image 50 may be video content in which the size of the car image 51 changes according to the elapsed playback time. In this case, the control device 4 can acquire correspondence information between the elapsed playback time of the projection target image 50 and the size of the car image 51 in the projection target image 50. Based on this correspondence information, the control device 4 may generate a projection image 70 that includes a frame image 60A whose size changes in accordance with the change in the size of the car image 51A (not shown).

[0109] Furthermore, the projection target image 50 may be video content in which the position and size of the car image 51 change according to the elapsed playback time. In this case, the control device 4 can acquire correspondence information between the elapsed playback time of the projection target image 50 and the position and size of the car image 51 in the projection target image 50. Based on this correspondence information, the control device 4 may generate a projection image 70 that includes a frame image 60A whose size changes in accordance with the changes in the position and size of the car image 51A (not shown).

[0110] <Specific example of a frame image 60A based on information in the display area and the position of the car image 51> Figures 18 to 21 show specific examples of frame images 60A based on information about the display area and the position of the car image 51. In the example in Figure 18, the projection target object 6 is a building with windows 171 and 172, and the projection range 11 includes windows 171 and 172. In this case, the control device 4 of the projection device 10 receives input of information indicating the area corresponding to windows 171 and 172 in the projection range 11 as information about the display area on which the projected image 70 is displayed. This information is input, for example, via the operation reception unit 2.

[0111] For example, as shown in Figure 16, suppose the projection target image 50 is a video content in which the car image 51 moves from left to right according to the elapsed playback time. The control device 4 can acquire correspondence information between the elapsed playback time of the projection target image 50 (for example, time t1 to t3) and the position of the car image 51 in the projection target image 50.

[0112] For example, the control device 4 includes the frame image 60A in the projected image 70, as shown in the example in Figure 19. That is, at time t1, when the car image 51A is located in the area corresponding to the window 171 in the projected image 70 (projection range 11), the control device 4 includes the frame image 60A in the area corresponding to the window 171 in the projected image 70. Furthermore, at time t2, when the car image 51A is located in the area corresponding to the space between the window 171 and the window 172 in the projected image 70, the control device 4 maintains the state in which the frame image 60A is included in the area corresponding to the window 172 in the projected image 70. Furthermore, at time t3, when the car image 51A is located in the area corresponding to the window 172 in the projected image 70, the control device 4 includes the frame image 60A in the area corresponding to the window 172 in the projected image 70.

[0113] Furthermore, the control device 4 may include a frame image 60A in the projected image 70, as shown in the example in Figure 20. That is, at time t2, when the car image 51A is located in the area corresponding to the space between window 171 and window 172 in the projected image 70, the control device 4 may include the frame image 60A in the projected image 70 so as to include the area of ​​the projected image 70 corresponding to window 171 and the area of ​​the projected image 70 corresponding to window 172. At time t2, the control device 4 may include the frame image 60A in the projected image 70 so as to include the frame image 60A along the outer periphery of the projected image 70, not limited to the example in Figure 20.

[0114] Furthermore, the control device 4 may include the frame image 60A in the projected image 70, as shown in the example in Figure 21. That is, at time t2 when the car image 51A is located in the area corresponding to the space between window 171 and window 172 in the projected image 70, the control device 4 may exclude the frame image 60A from the projected image 70 (remove the frame image 60A).

[0115] Furthermore, the control device 4 may, for example, include the frame image 60A in the area of ​​the projected image 70 corresponding to the window 171 when a predetermined time before time t1, in which the car image 51A is located in the area of ​​the projected image 70 corresponding to the window 171 (not shown). Similarly, the control device 4 may, for example, include the frame image 60A in the area of ​​the projected image 70 corresponding to the window 172 when a predetermined time before time t2, in which the car image 51A is located in the area of ​​the projected image 70 corresponding to the window 172.

[0116] As explained in Figures 18 to 21, the control device 4 may generate a third image data representing the projected image 70, including a frame image 60A (second image) based on information of the display area where the projected image 70 (third image) is displayed and the position of the car image 51 (image of the first object) in the projection target image 50.

[0117] In Embodiment 2, the configuration described was that the frame image 60A completely surrounds the vehicle image 51A. However, the frame image 60A in Embodiment 2 may also be an L-shaped frame image, such as the frame image 60A shown in Figure 8, that surrounds (encloses) the vehicle image 51A.

[0118] Furthermore, although a frame image 60A was described as the second image of Embodiment 2, the second image of Embodiment 2 is not limited to a frame image, but may be, for example, an arrow image indicating the position of the car image 51A. (Variations of each embodiment) <Variations of projection device 10> In Figures 4 and 5, the configuration of the projection device 10 was described in which the optical axis K is not bent. However, a configuration in which a reflective member is provided in the optical unit 106 to bend the optical axis K once or more is also possible.

[0119] Figure 22 is a schematic diagram showing other external configurations of the projection device 10. Figure 23 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Figure 22. In Figures 22 and 23, parts that are the same as those shown in Figures 4 and 5 are denoted by the same reference numerals and their descriptions are omitted.

[0120] As shown in Figure 22, the optical unit 106 includes a first member 102 supported by the main body 101, as well as a second member 103 supported by the first member 102. Note that the first member 102 and the second member 103 may be an integrated component.

[0121] As shown in Figure 23, the optical unit 106 includes, in addition to the first member 102, a second member 103 having a hollow portion 3A connected to the hollow portion 2A of the first member 102, a first optical system 121 and a reflecting member 122 arranged in the hollow portion 2A, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34 arranged in the hollow portion 3A, a first shift mechanism 105, and a projection direction changing mechanism 104.

[0122] In the examples shown in Figures 22 and 23, the apertures 2a and 2b of the first member 102 are formed on planes perpendicular to each other. Furthermore, the projection optical system 23 shown in Figures 22 and 23 consists of a reflective member 122, a second optical system 31, a reflective member 32, and a third optical system 33, in addition to the first optical system 121 and lens 34 shown in Figures 4 and 5. With such a projection optical system 23, as shown in Figure 23, the optical axis K is folded back by bending twice. The first optical system 121, reflective member 122, second optical system 31, reflective member 32, third optical system 33, and lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0123] The first optical system 121 guides light traveling in direction X1, which is incident from the main body 101 to the first member 102, to the reflecting member 122. The reflecting member 122 reflects the light incident from the first optical system 121 in direction Y1. The reflecting member 122 is made of, for example, a mirror. An aperture 2b is formed in the first member 102 on the optical path of the light reflected by the reflecting member 122, and this reflected light passes through the aperture 2b and proceeds to the hollow portion 3A of the second member 103.

[0124] The second member 103 has a roughly L-shaped cross-section, and an opening 3a is formed at a position facing the opening 2b of the first member 102. Light from the main body 101 that has passed through the opening 2b of the first member 102 is incident on the hollow portion 3A of the second member 103 through this opening 3a. Note that the cross-sectional shapes of the first member 102 and the second member 103 are arbitrary and not limited to those described above.

[0125] The second optical system 31 includes at least one lens and guides the light incident from the first member 102 to the reflecting member 32. The reflecting member 32 reflects the light incident from the second optical system 31 in direction X2 and guides it to the third optical system 33. The reflecting member 32 is made up of, for example, a mirror. The third optical system 33 includes at least one lens and guides the light reflected by the reflecting member 32 to lens 34.

[0126] The lens 34 is positioned at the end of the second member 103 so as to block the opening 3c formed at the end on the direction X2 side. The lens 34 projects light incident from the third optical system 33 onto the projection target 6.

[0127] Figure 23 shows the state in which the first member 102 has been moved to its maximum extent toward direction Y1 by the first shift mechanism 105. From the state shown in Figure 23, the first member 102 moves toward direction Y2 by the first shift mechanism 105, which changes the relative position between the center of the image formed by the light modulation unit 22 and the optical axis K, thereby shifting the image G1 projected onto the projection target 6 toward direction Y1.

[0128] The projection direction changing mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. This projection direction changing mechanism 104 allows the second member 103 to rotate freely around a rotation axis (specifically, the optical axis K) extending in direction Y. The projection direction changing mechanism 104 only needs to be able to rotate the optical system and is not limited to the arrangement shown in Figure 23. Furthermore, the number of rotation mechanisms is not limited to one; multiple mechanisms may be provided.

[0129] <Variations of the control device> Although the case in which the control device of the present invention is applied to the control device 4 of the projection device 10 has been described, the present invention is not limited to this configuration.

[0130] Figure 24 shows a modified example of the projection system 100. The projection system 100 shown in Figure 24 includes an information terminal 110 in addition to the configuration of the projection system 100 shown in Figure 1. The control device of the present invention may be applied to the information terminal 110.

[0131] The information terminal 110 is an information terminal capable of communicating directly or indirectly with the projection device 10. Communication between the information terminal 110 and the projection device 10 may be wired or wireless. The information terminal 110 performs various controls by the control device 4 by communicating with the projection device 10. In the example shown in Figure 24, the information terminal 110 is a notebook-type personal computer, but the information terminal 110 can be various types of information terminals, such as a desktop-type personal computer, smartphone, or tablet device.

[0132] For example, the first image data representing the projection target image 50 is input to the information terminal 110. The second image data representing the frame image 60 is stored in the storage medium of the information terminal 110. Based on these image data, the information terminal 110 generates a third image data representing the projection image 70 and outputs the generated third image data to the projection device 10 (projection unit). The projection device 10 projects the projection image 70 based on the third image data output from the information terminal 110.

[0133] Figure 25 shows an example of the hardware configuration of the information terminal 110. The information terminal 110 shown in Figure 1 comprises a processor 111, memory 112, a communication interface 113, and a user interface 114, as shown in Figure 25. The processor 111, memory 112, communication interface 113, and user interface 114 are connected, for example, by a bus 119. The processor 111, memory 112, communication interface 113, and user interface 114 have the same configuration as the processor 81, memory 82, communication interface 83, and user interface 84 shown in Figure 6, respectively.

[0134] The present invention has described a configuration in which the control device projects a third image (projected image 70) represented by the generated third image data using a projection device 10, but the invention is not limited to this configuration. For example, the control device may output the generated third image data to a display capable of displaying images. As a result, the third image represented by the third image data (for example, an image similar to the projected image 70) is displayed on the display. The display may be the display of a display device including the control device of the present invention, or it may be an external display capable of communicating with the control device of the present invention.

[0135] <Other examples of specific content> While augmented reality content was described as an example of specific content that can be associated with the projected image 70, the specific content that can be associated with the projected image 70 is not limited to augmented reality content. For example, the specific content that can be associated with the projected image 70 may be content related to things other than sight, such as sound or smell.

[0136] This specification contains at least the following:

[0137] (1) A control device comprising a processor, The above processor is A third image data is generated based on the first image data and the second image data. Output the above third image data, The third image represented by the third image data described above, the second image based on the second image data described above, is an image that indicates that specific content is associated with the third image, and the third image surrounds the first image based on the first image data described above. Control device.

[0138] (2) (1) The control device described above, The second image described above is in contact with at least a portion of the outer periphery of the first image described above, and does not include the internal region of the first image described above. Control device.

[0139] (3) A control device as described in (1) or (2), The above processor outputs the above third image data to the projection unit. Control device.

[0140] (4) A control device according to any one of (1) to (3), The above processor is Based on the information of the display area where the above third image is displayed, the above first image data, and the above second image data, the above third image data is generated. Control device.

[0141] (5) A control device according to any one of items (1) to (4), The second image above is an image determined based on the color of the first image above. Control device.

[0142] (6) A control device according to any one of (1) to (5), The second image above is a frame image that surrounds at least a portion of the first image above. Control device.

[0143] (7) A control device according to any one of items (1) to (6), The first image above is an image associated with the specific content described above. Control device.

[0144] (8) A control device according to any one of items (1) to (7), The second image above is an image associated with the specific content mentioned above. Control device.

[0145] (9) A control device according to any one of items (1) to (8), The above specific content includes augmented reality content. Control device.

[0146] (10) (9) The control device described above, The above augmented reality content includes a first augmented reality content that is played when the first object included in the first image is within the imaging field of view, and a second augmented reality content that is played regardless of whether or not the first object is within the imaging field of view. Control device.

[0147] (11) A control device according to any one of items (1) to (10), The second image above includes an identification image, The above identification image is information that can identify the relative position of the above identification image in the above third image. Control device.

[0148] (12) A control device comprising a processor, The processor is, A third image data is generated based on the first image data and the second image data. Output the above third image data, The third image represented by the third image data described above, specifically the second image based on the second image data described above, is an image that indicates the position of the image of the first object to which specific content is associated, which is included in the first image based on the first image data described above. Control device.

[0149] (13) (12) The control device described above, The image of the first object described above is the image that moves in the third image described above. The second image above is an image that moves in accordance with the movement of the first object image. Control device.

[0150] (14) A control device as described in (12) or (13), The above processor is Based on the correspondence information between the elapsed time of playback of the first image and at least one of the position and size of the first object in the first image, the third image data representing the third image including the second image is generated. Control device.

[0151] (15) A control device according to any one of items (12) to (14), The above processor is The system generates a third image data representing the third image, including the second image, based on the information of the feature region of the display area where the third image is displayed, and the position of the image of the first object in the third image. Control device.

[0152] (16) A control device according to any one of (12) to (15), The second image above is a frame image that surrounds at least a portion of the image of the first object. Control device.

[0153] (17) The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the above third image data, The third image represented by the third image data described above, the second image based on the second image data described above, is an image that indicates that specific content is associated with the third image, and the third image surrounds the first image based on the first image data described above. Control method.

[0154] (18) The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the above third image data, The third image represented by the third image data described above, specifically the second image based on the second image data described above, is an image that indicates the position of the image of the first object to which specific content is associated, which is included in the first image based on the first image data described above. Control method.

[0155] (19) The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the above third image data. A control program for executing a process, The third image represented by the third image data described above, the second image based on the second image data described above, is an image that indicates that specific content is associated with the third image, and the third image surrounds the first image based on the first image data described above. Control program.

[0156] (20) The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the above third image data. A control program for executing a process, The third image represented by the third image data described above, specifically the second image based on the second image data described above, is an image that indicates the position of the image of the first object to which specific content is associated, which is included in the first image based on the first image data described above. Control program.

[0157] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to these examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any way without departing from the spirit of the invention.

[0158] This application is based on a Japanese patent application (Patent Application No. 2021-194119) filed on November 30, 2021, the contents of which are incorporated by reference within this application. [Explanation of Symbols]

[0159] 1 Projection section 2. Operation reception unit 2A,3A Hollow part 2a,2b,3a,3c,15a opening 4. Control device 4a Storage medium 5 Communications Department 6. Object to be projected 6a Projection area 10 Projection device 11 Projection Range 12 Optical Modulation Unit 15 cabinets 21 Light source 22 Optical Modulation Section 23 Projection optical system 24 Control circuits 31 Second optical system 32,122 Reflective material 33 Third optical system 34 lenses 50,50A Projection target image 51,51A Car images 60,60A frame image 61 Identification Information 70 Projection Images 80,110 Information terminals 81,111 processors 82,112 memory 83,113 Communication Interfaces 84,114 User Interfaces 85 Imaging Module 85a Imaging area 86 Display section 89,119 buses 90 superimposed images 91 Glossy Images 92 Smoke Images 93 Meteor images 94 Arrow Image 100 Projection Systems 101 Main body 102 First Member 103 Second Member 104 Projection direction changing mechanism 105 First Shift Mechanism 106 Optical Units 121 1st optical system 171,172 windows G1 Image

Claims

1. A control device comprising a processor, The aforementioned processor, A third image data is generated based on the first image data and the second image data. Output the above-mentioned third image data, The third image represented by the third image data, the second image based on the second image data, is an image that indicates that specific content, separate from the content of the third image, provided based on the captured image including the first image or the second image, is associated with the third image, and the third image surrounds the first image based on the first image data. Control device.

2. A control device according to claim 1, The second image is in contact with at least a portion of the outer periphery of the first image and does not include the internal region of the first image. Control device.

3. A control device according to claim 1, The processor outputs the third image data to the projection unit. Control device.

4. A control device according to claim 1, The aforementioned processor, The third image data is generated based on the information of the display area on which the third image is displayed, the first image data, and the second image data. Control device.

5. A control device according to claim 1, The second image is an image determined based on the color of the first image. Control device.

6. A control device according to claim 1, The second image is a frame image that surrounds at least a part of the first image. Control device.

7. A control device according to claim 1, The first image is an image to which the specific content is associated. Control device.

8. A control device according to claim 1, The second image is an image to which the specific content is associated. Control device.

9. A control device according to claim 1, The aforementioned specific content includes augmented reality content. Control device.

10. A control device according to claim 9, The augmented reality content includes a first augmented reality content that is played when the first object included in the first image is included in the imaging field of view, and a second augmented reality content that is played regardless of whether the first object is included in the imaging field of view or not. Control device.

11. A control device according to any one of claims 1 to 10, The second image includes an identification image, The aforementioned identification image is information that can identify the relative position of the identification image in the third image. Control device.

12. A control device comprising a processor, The aforementioned processor, A third image data is generated based on the first image data and the second image data. Output the above-mentioned third image data, The third image represented by the third image data, specifically the second image based on the second image data, is an image indicating the position of an image of a first object to which specific content, provided based on an image capture including the first or second image, is associated, which is different from the content of the third image and is included in the first image based on the first image data among the third images. Control device.

13. A control device according to claim 12, The image of the first object is an image that moves in the third image, The second image is an image that moves in accordance with the movement of the first object image. Control device.

14. A control device according to claim 12, The aforementioned processor, Based on correspondence information between the elapsed time of playback of the first image and at least one of the position and size of the image of the first object in the first image, the third image data representing the third image including the second image is generated. Control device.

15. A control device according to claim 12, The aforementioned processor, The third image data is generated, which includes the second image and is based on information of a specific region for determining the display mode of the second image within the display area where the third image is displayed, and the position of the image of the first object in the first image. Control device.

16. A control device according to any one of claims 12 to 15, The second image is a frame image that surrounds at least a portion of the image of the first object. Control device.

17. The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the above-mentioned third image data, The third image represented by the third image data, the second image based on the second image data, is an image that indicates that specific content, separate from the content of the third image, provided based on the captured image including the first image or the second image, is associated with the third image, and the third image surrounds the first image based on the first image data. Control method.

18. The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the above-mentioned third image data, The third image represented by the third image data, specifically the second image based on the second image data, is an image indicating the position of an image of a first object to which specific content, provided based on an image capture including the first or second image, is associated, which is different from the content of the third image and is included in the first image based on the first image data among the third images. Control method.

19. The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the aforementioned third image data. A control program for executing a process, The third image represented by the third image data, the second image based on the second image data, is an image that indicates that specific content, separate from the content of the third image, provided based on the captured image including the first image or the second image, is associated with the third image, and the third image surrounds the first image based on the first image data. Control program.

20. The processor of the control unit, A third image data is generated based on the first image data and the second image data. Output the aforementioned third image data. A control program for executing a process, The third image represented by the third image data, specifically the second image based on the second image data, is an image indicating the position of an image of a first object to which specific content, provided based on an image capture including the first or second image, is associated, which is different from the content of the third image and is included in the first image based on the first image data among the third images. Control program.

Images