Image display system, image display method, control device, and computer program
The image display system uses a three-dimensional virtual space with dual cameras to synchronize user interactions, enhancing immersion and engagement by allowing manipulation and feedback, addressing the lack of immersion and engagement in existing systems.
Patent Information
- Application Number
- JP2023072804
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-04-26
AI Technical Summary
Existing image display systems fail to provide users with a sense of immersion and maintain their interest over time, as the displayed object images perform predetermined operations and lack interactive engagement.
An image display system that utilizes a three-dimensional virtual space, incorporating a display device and an operation terminal, with dual virtual cameras (bird's-eye and object-view) to synchronize user interactions, allowing users to manipulate objects and receive feedback through the terminal, enhancing immersion and engagement.
The system provides users with an immersive experience by synchronizing object operations across devices, allowing for interactive manipulation and feedback, thereby maintaining user interest and engagement.
Smart Images

Figure 2024158020000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image display system for displaying an image of a three-dimensional virtual space on a display device and an operation terminal, and also to an image display method, a control device, and a computer program. [Background technology]
[0002] The applicant of this application has proposed an image display system that can utilize coloring books and paintings to demonstrate the user's creativity and expressiveness (Patent Document 1). In the image display system of Patent Document 1, when a coloring book or painting is scanned and captured as image data, an image of the object depicted therein is generated, and the object image performs a predetermined action on a display screen such as a large screen. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2017-037614 Summary of the Invention [Problem to be solved by the invention]
[0004] In the image display system of Patent Document 1, after an object image is displayed on the screen, the object image basically only performs a predetermined action according to a program. A user who creates a coloring book or the like that is the basis of the object image can view his / her object image from the outside, but there is a problem that it is difficult to feel immersed in the work just by viewing it from the outside. In addition, since the object image in this system only performs a predetermined action, there is also a problem that the user is likely to become bored.
[0005] SUMMARY OF THE PRESENT EMBODIMENTS Accordingly, a main object of the present invention is to provide an image display system that gives a user a sense of immersion and can maintain the user's interest for a long period of time. [Means for solving the problem]
[0006] The inventor of the present invention has intensively studied means for solving the problems of the above-mentioned conventional inventions, and has found that by displaying an object image in a three-dimensional virtual space on both a display screen such as a large screen and a display screen of a terminal held by the user, and allowing the object image to be operated by the terminal, it is possible to give the user a sense of immersion in the work and keep the user's interest for a long time. Based on the above findings, the inventor has come to the conclusion that the problems of the conventional technology can be solved, and has completed the present invention. The invention will be specifically described below.
[0007] A first aspect of the present invention relates to an image display system. The image display system according to the present invention includes a display device, an operation terminal, and a control device. The display device may be a device that projects an image on a screen by a projector, or may be a device that includes a display element such as a liquid crystal display or an organic EL display. The operation terminal is a terminal that is mainly held by a user, and may be a mobile information terminal such as a smartphone or a tablet terminal, or may be a laptop or desktop computer. The control device is connected to the display device and the operation terminal via a wired or wireless network. The control device may be connected to the display device and the operation terminal via a communication line such as the Internet, or may be connected to the display device and the operation terminal via a wired or wireless local area network (LAN). The control device may be configured by one computer (server device), or may be configured by distributing functions to multiple computers.
[0008] Here, the control device has functional blocks such as a generation unit, a camera control unit, a first display control unit, a second display control unit, and an object control unit. These functional blocks are realized by a processor of a computer constituting the control device. The generation unit generates a three-dimensional virtual space including a playable object. The playable object can be operated by an operation terminal owned by a user. In addition to the playable object, a non-playable object may exist in the three-dimensional virtual space. The camera control unit controls an overhead camera and an object viewpoint camera in the three-dimensional virtual space. The overhead camera and the object viewpoint camera are virtual cameras existing in the three-dimensional virtual space. The overhead camera is controlled to mainly photograph the three-dimensional virtual space without depending on a specific object (particularly the playable object). On the other hand, the object viewpoint camera is controlled to photograph the three-dimensional virtual space from a first-person viewpoint from the viewpoint of the playable object, or to photograph the three-dimensional virtual space from a third-person viewpoint so as to follow the playable object. The first display control unit causes the display device to display an image from the overhead camera. The second display control unit causes the operation terminal to display an image from the object viewpoint camera. More specifically, the second display control unit causes the operation terminal that operates the operation object to display an image from an object viewpoint camera subordinate to the operation object. The object control unit controls the operation of the operation object in the three-dimensional virtual space according to an operation command from the operation terminal.
[0009] As described above, in the present invention, two types of virtual cameras, an overhead camera and an object viewpoint camera, are provided in a certain three-dimensional virtual space, and an image from the overhead camera is displayed on a display device, and an image from the object viewpoint camera is displayed on an operation terminal. In addition, an operation object in the three-dimensional virtual space can be freely operated by the operation terminal. This synchronizes the operation of the operation object operated by the operation terminal with the operation of the operation object displayed on the display device. In other words, when an operation object is operated by the operation terminal, the operation object displayed on the display device also performs the same operation. Therefore, the user can be given a sense of immersion as if he or she had become the operation object and entered the three-dimensional virtual space displayed by the display device. In addition, by allowing the user to freely operate the operation object, the user's interest can be long-lasting without getting bored.
[0010] The system according to the present invention preferably further comprises an image capture device that scans a physical medium on which an object is drawn and captures image data. In this case, the generation unit of the control device generates an operation object in a three-dimensional virtual space based on the image data captured by the image capture device. For example, when a user colors an object using a coloring book or the like, the object becomes an image and is captured in the three-dimensional virtual space, and functions as an operation object. Then, the user can freely operate the operation object that he or she has colored in the three-dimensional virtual space via the operation terminal. This can give the user a greater sense of immersion.
[0011] The system according to the present invention preferably further includes a detection device capable of detecting a contact position on the display screen of the display device. In this case, the control device may further include a terminal control unit. The terminal control unit generates a reaction command based on detection information by the detection device and transmits the reaction command to the operation terminal. In this case, the operation terminal preferably has an output unit that performs output according to the reaction command. Examples of the output unit are output devices such as a display, a vibrator, and a flashlight. For example, when the terminal control unit determines that the contact position detected by the detection device corresponds to an operation object displayed on the display screen, it generates a reaction command for the operation terminal that is operating the operation object. For example, when an operation object operated by an operation terminal of a certain user is touched by another user (viewer) on the display screen of the display device, some kind of display is performed on the display screen of the operation terminal, a vibrator in the operation terminal vibrates, a predetermined sound is output from the speaker of the operation terminal, or a flashlight of the operation terminal is turned on. This gives the user operating the operation object the feeling that the operation object displayed on his / her own operation terminal is synchronized with the operation object displayed on the display screen of the display device, providing an even greater sense of immersion.
[0012] In the system according to the present invention, the operation terminal preferably has a photographing unit. In this case, the operation terminal transmits the photographed image by the photographing unit to the control device. The control device preferably further has an image analysis unit and an object management unit. The image analysis unit analyzes the photographed image received from the operation terminal and judges whether or not the photographed image contains an operation object. When the image analysis unit judges that the photographed image contains an operation object, the object management unit grants the operation authority of the operation object to the operation terminal that photographed the operation object. This allows the operation terminal to operate the operation object contained in the photographed image. It is also possible to display a list of operable objects on the operation terminal and have the user select the operation object he or she wants to operate, but if there are a large number of selectable operation objects, it becomes difficult to find the object he or she wants to operate from the list. On the other hand, when a user likes an operation object displayed on the display screen of the display device, the user can obtain the operation authority of the operation object simply by photographing it, which can significantly improve the user's convenience.
[0013] A second aspect of the present invention is a computer program for an operation terminal. The computer program according to the second aspect causes an information communication terminal to function as an operation terminal in the image display system according to the first aspect. This program may be downloadable via the Internet, or may be non-temporarily recorded on a recording medium such as a CD-ROM.
[0014] A third aspect of the present invention relates to a control device connected to a display device and an operation terminal via a wired or wireless network. The control device includes a generation unit that generates a three-dimensional virtual space including a control object, a camera control unit that controls an overhead camera in the three-dimensional virtual space and the line of sight of the control object or an object viewpoint camera that follows the overhead camera, a first display control unit that causes the display device to display an image from the overhead camera, a second display control unit that causes the operation terminal to display an image from the object viewpoint camera, and an object control unit that controls the movement of the control object in the three-dimensional virtual space according to an operation command from the operation terminal.
[0015] A fourth aspect of the present invention is a program for a computer. The program according to the fourth aspect causes a computer to function as the control device according to the third aspect. This program may be downloadable via the Internet, or may be non-temporarily recorded on a recording medium such as a CD-ROM.
[0016] A fifth aspect of the present invention relates to an image display method executed by a system including a display device, an operation terminal, and a control device connected to the display device and the operation terminal via a wired or wireless network. In the method according to the present invention, the control device generates a three-dimensional virtual space including a control object (generation step). The control device also controls an overhead camera and the line of sight of the operation object or an object viewpoint camera that follows the line of sight (camera control step). The control device also causes the display device to display an image from the overhead camera (first display control step). The control device also causes the operation terminal to display an image from the object viewpoint camera (second control step). The control device also controls the movement of the operation object in the three-dimensional virtual space in accordance with an operation command from the operation terminal (object control step). Effect of the Invention
[0017] According to the present invention, an image display system is provided that gives a user a sense of immersion and can maintain the user's interest for a long period of time. [Brief description of the drawings]
[0018] [Figure 1] FIG. 1 shows an overview of an image display system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of an image display system. [Diagram 3] FIG. 3 shows the main features of the image display system. [Figure 4] FIG. 4 is a sequence diagram showing an example of a display process of an operation object. [Diagram 5] FIG. 5 is a sequence diagram showing an example of a reaction process of the operating terminal. [Figure 6] FIG. 6 is a sequence diagram showing an example of a process for granting an operating authority for an operation object to an operating terminal. [Figure 7] FIG. 7 shows an example of an object-view camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, and includes appropriate modifications of the embodiments described below within the scope obvious to those skilled in the art.
[0020] FIG. 1 is a schematic diagram of an image display system 100 according to the present invention. As shown in FIG. 1, an operation object PO that can be operated by a user exists in a three-dimensional virtual space. A unique virtual object viewpoint camera VC is attached to this operation object PO, and an image (including a still image and a video) taken by this object viewpoint camera VC is displayed on a display screen of an operation terminal 40 such as a smartphone owned by a user. A user can operate the operation object PO in the three-dimensional virtual space by using the operation terminal 40 while checking the image displayed on the operation terminal 40. Note that, in the three-dimensional virtual space, there exist a plurality of operation objects PO each having a unique object viewpoint camera VC, and each operation object PO is operated by a different user. In addition to the operation object PO, a non-operation object NO also exists in the three-dimensional virtual space.
[0021] On the other hand, in the three-dimensional virtual space, in addition to the object viewpoint camera VC associated with the operation object PO, there exists a virtual overhead camera OC that overlooks this virtual space. This overhead camera OC is not subordinate to an individual operation object PO or non-operation object NO, but is controlled to capture a plurality of operation objects PO and a plurality of non-operation objects NO as a whole. For example, in the example shown in FIG. 1, a dome-shaped space is provided in the three-dimensional virtual space, and the operation object PO and the non-operation object NO are provided in this dome-shaped space. In this case, the overhead camera OC photographs the operation object PO and the non-operation object NO that exist in this dome-shaped space while moving around the periphery of the dome-shaped space or moving in the sky above the dome-shaped space. The overhead camera OC can also enter the dome-shaped space to capture the operation object PO and the like.
[0022] The image captured by the overhead camera OC is output to the display device 30. In the example shown in FIG. 1, the display device 30 includes a projector 31 and a large screen 32. The projector 31 projects the image captured by the overhead camera OC onto the screen 32. At this time, the overhead camera OC and the object viewpoint camera VC may capture the same operation object PO in the same three-dimensional virtual space. In this case, when the user operates the operation object PO using the operation terminal 40, the operation of the operation object PO is displayed on the operation terminal 40 and at the same time, the operation of the operation object PO is also displayed on the display screen of the display device 30. As a result, it can be said that the operation object PO displayed on the operation terminal 40 and the operation object PO displayed on the display screen of the display device 30 are synchronized.
[0023] Also, an object drawn or colored by a user on a physical medium such as printing paper can be captured as image data by a scanner and used as an operational object PO in a three-dimensional virtual space. That is, the present system 100 includes an image capture device 20, and an operational object PO is generated based on an image captured by the image capture device 20. For such a mechanism, refer to Patent Document 1 (JP Patent Publication No. 2017-037614). This allows the user to have the experience of manipulating an object that he or she has drawn.
[0024] Also, the user (viewer) can touch the operation object PO displayed on the display screen (screen 32, etc.) of the display device 30. The touch position on the display screen is detected by the detection device 60, and it is determined whether the viewer has touched the operation object PO. When the viewer touches the operation object PO, the operation object PO may perform a predetermined action in the three-dimensional virtual space. Also, when the viewer touches the operation object PO, contact information may be transmitted to the operation terminal 40 that is operating the operation object PO, and the operation terminal 40 may output a predetermined reaction.
[0025] Fig. 2 is a block diagram showing an example of a functional configuration of an image display system 100 according to the present invention. As shown in Fig. 2, the image display system 100 includes, as main devices, a control device 10, an image capturing device 20, a display device 30, an operation terminal 40, and a detection device 60.
[0026] The control device 10 is a computer that is responsible for the overall control of the image display system 100, and mainly generates a three-dimensional virtual space and draws images to be displayed on the display device 30 and the operation terminal 40. The control device 10 is connected to each of the image capture device 20, the display device 30, the operation terminal 40, and the detection device 60 by wire or wirelessly. In this embodiment, the control device 10 is connected to the image capture device 20, the display device 30, and the detection device 60 by a local area network (LAN). On the other hand, the control device 10 is connected to the operation terminal 40 wirelessly via the Internet. However, in an environment where communication with an external base station is poor, the control device 10 and the operation terminal 40 can also be connected by wireless LAN.
[0027] The control device 10 mainly includes a central control unit 11, a storage unit 12, and a communication unit 13. The central control unit 11 is composed of, for example, a processor and a memory. Examples of the processor are a known CPU or GPU. The processor performs a predetermined arithmetic processing and image processing according to the programs and data stored in the memory, and executes various control processing while writing the results of the processing in the working space of the memory. The memory is composed of, for example, a volatile memory such as a RAM (Random Access Memory), and is used for the arithmetic processing by the above-mentioned processor. The storage unit 12 is an element (storage) for storing data mainly used for processing by the central control unit 11. The storage unit 12 may also store a program used for processing by the processor of the central control unit 11. The storage unit 12 is composed of a non-volatile memory such as a ROM (Read Only Memory) or a flash memory, and an HDD (Hard Disk Drive). The communication unit 13 is an element for transmitting and receiving data to and from the operation terminal 40 via the Internet, and may be any element capable of transmitting and receiving data by wire or wirelessly. When performing wireless communication, the communication unit 13 can be a communication module conforming to a known wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).
[0028] In this embodiment, the processing executed by the central control unit 11 is explained as functional blocks. As shown in FIG. 1, the central control unit 11 includes a generation unit 11a, a camera control unit 11b, an object management unit 11d, an object management unit 11d, a drawing processing unit 11e, a first display control unit 11f, a second display control unit 11g, a contact position determination unit 11h, a terminal control unit 11i, and an image analysis unit 11j.
[0029] The generation unit 11a mainly performs calculations to generate a three-dimensional virtual space. This three-dimensional virtual space includes operation objects and non-operation objects as shown in Fig. 1. Therefore, the generation unit 11a also performs calculations to generate operation objects and non-operation objects. In addition, as shown in Fig. 1, when a dome-shaped space in which each object is placed is provided in the three-dimensional virtual space, the generation unit 11a performs calculations to generate this dome-shaped space.
[0030] The camera control unit 11b controls the object viewpoint camera and the overhead camera in the three-dimensional virtual space. For example, when the object viewpoint camera is for photographing the operation object from a third-person viewpoint, the camera control unit 11b controls the object viewpoint camera so that the operation object always enters the shooting range of the object viewpoint camera while making the object viewpoint camera follow the movement of the operation object. Also, when the object viewpoint camera is from the first-person viewpoint of the operation object, the camera control unit 11b controls the object viewpoint camera so that the object viewpoint camera always photographs the three-dimensional virtual space from the line of sight of the operation object. Also, regarding the overhead camera, the camera control unit 11b may control the overhead camera to move on a predetermined trajectory in the three-dimensional virtual space, for example. As described above, when a dome-shaped space is formed in the three-dimensional virtual space, the camera control unit 11b may photograph the operation object and the non-operation object present in the dome-shaped space while making the overhead camera move around the periphery of the dome-shaped space or move above the dome-shaped space.
[0031] The object control unit 11c controls the operation objects and non-operation objects that exist in the three-dimensional virtual space. The object control unit 11c mainly controls the operation of the operation objects according to the operation command from the operation terminal 40. The object control unit 11c also controls the operation of the operation objects or non-operation objects based on the detection information from the detection device 60. In addition, the operation rules of the operation objects and non-operation objects are stored in the memory or the storage unit 12. For example, the object control unit 11c may control the operation of each object according to a predetermined operation rule, such as making each object perform a collision action when the operation objects come into contact with each other in the three-dimensional virtual space, or making the non-operation object perform an avoidance action when the operation object is about to come into contact with the non-operation object. Also, when the operation object is in a state where it has not received an operation command from the operation terminal 40, the object control unit 11c may operate the operation object according to a predetermined operation program.
[0032] The object management unit 11d manages the operation objects and non-operation objects that exist in the three-dimensional virtual space. Specifically, the object management unit 11d performs a process of granting the operation authority of the operation object to the operation terminal 40. For example, a unique identification information (ID) is assigned to the operation object, and the object management unit 11d associates the identification information (ID) of the operation object with the identification information (IP address, etc.) of the operation terminal 40 that has been given the authority to operate the operation object, and records the association in the database in the storage unit 12. This allows the database to manage which operation terminal 40 is operating which operation object. In addition, the object management unit 11d identifies the operation object and the non-operation object that currently exist in the three-dimensional virtual space. In addition, the object management unit 11d may manage objects that can exist simultaneously in the three-dimensional virtual space. For example, if an upper limit is set on the number of control objects and non-control objects that can appear in the three-dimensional virtual space at one time, when a new control object or non-control object is generated in the three-dimensional virtual space, the object management unit 11d deletes each object in order from the oldest to manage the number of each object currently existing in the three-dimensional virtual space so that it is within the upper limit.
[0033] The drawing processor 11e performs drawing processing of the three-dimensional virtual space and each object (operation object, non-operation object) photographed by the object viewpoint camera and the overhead camera. Specifically, the drawing processor 11e converts the three-dimensional virtual space expressed in the world coordinate system (X, Y, Z) into a screen coordinate system (U, V) from the camera positions of the object viewpoint camera and the overhead camera to generate a two-dimensional image. The two-dimensional image from the camera position of the overhead camera is an image to be displayed on the display device 30. Moreover, the two-dimensional image from the camera position of the object viewpoint camera is an image to be displayed on the operation terminal 40. Moreover, the drawing processor 11e may process the three-dimensional virtual space into an image suitable for display on the display device 30 or the operation terminal 40 by using a known 3DCG processing technique, instead of converting the three-dimensional virtual space directly into a two-dimensional image.
[0034] The first display control unit 11f outputs a two-dimensional image from the position of the overhead camera to the display device 30 via a predetermined I / O interface, and causes the two-dimensional image to be displayed on the display device 30. On the other hand, the second display control unit 11g transmits a two-dimensional image from the position of the object viewpoint camera to the operation terminal 40 via the communication unit 13, and causes the operation terminal 40 to display the two-dimensional image.
[0035] The contact position determination unit 11h determines the contact position of the user (viewer) or the like on the display screen of the display device 30 based on the detection information by the detection device 60. For example, the contact position determination unit 11h constantly grasps the correspondence between the xy coordinates on the display screen of the display device 30 and the xy coordinates in the two-dimensional image displayed on the display device 30. Therefore, when the detection device 60 detects the position (xy coordinates) where the viewer touches the display screen of the display device 30, the contact position determination unit 11h specifies the position (xy coordinates) where the viewer touches the two-dimensional image displayed on the display device 30. Thereby, for example, the contact position determination unit 11h can determine whether the viewer touches an operation object or a non-operation object on the display screen of the display device 30.
[0036] The terminal control unit 11i generates a reaction command related to the control of the output unit (display unit 48, vibrator 49, speaker 51, flashlight 52, etc.) of the operation terminal 40. The reaction command generated by the terminal control unit 11i is transmitted to the operation terminal 40 via the communication unit 13. In this embodiment, for example, when an operation object being operated by the operation terminal 40 of a certain user is touched by another user (viewer) on the display screen of the display device 30, a reaction command is generated for the operation terminal 40.
[0037] The image analysis unit 11j mainly analyzes an image captured by the photographing unit 45 of the operation terminal 40. In this embodiment, the image analysis unit 11j judges whether or not an operation object is included in the captured image of the operation terminal 40. If an operation object is included in the captured image of the operation terminal 40, the image analysis unit 11j performs individual identification of the operation object. Specifically, the image analysis unit 11j may identify the operation object from characteristic points such as the shape and color of the operation object included in the captured image. Also, when the operation object is generated, a marker (digital watermark) for individual identification may be embedded in advance, and the marker of the operation object included in the captured image may be analyzed to perform individual identification of the operation object.
[0038] The image capture device 20 is a device for capturing image data by scanning a physical medium such as a printed sheet on which an object is drawn. If a marker for identifying an object is drawn on the physical medium, the image capture device 20 scans the surface of the physical medium on which the object and the marker are drawn and converts the image data into image data. A known scanner or camera can be used as the image capture device 20. The image capture device 20 converts the scanned image into data in a known format such as GIF or JPEG and outputs the data to the control device 10.
[0039] Further, the generating unit 11a of the control device 10 can generate an operation object based on the image data captured by the image capturing device 20. Specifically, the generating unit 11a performs an analysis to identify the type of object included in the image data. For example, as in Patent Document 1, when a marker corresponding to an object is attached to the surface of a physical medium (coloring paper), the generating unit 11a can analyze the marker included in the image data to identify the type of object. Furthermore, the generating unit 11a performs a drawing process to extract an area in which an object is drawn from the image data and generate an object image from which unnecessary parts such as the background are removed. For example, when a marker is printed at each of the four corners of the surface of the physical medium (coloring paper), the generating unit 11a grasps the positions of the four markers included in the image data, defines a rectangular area with the four markers as vertices, and removes the outside of the rectangular area. This makes it possible to cut out a rectangular image in which an object is drawn from the image data. After that, the generating unit 11a extracts the outlined part of the object and the colored part from this rectangular area to generate an image of the operation object. For example, the generating unit 11a may generate an image of the operation object by extracting the outline of the object by applying a mask corresponding to the type of object to the rectangular image, or may generate an image of the operation object by performing chromakey compositing processing to extract colored parts from the rectangular image and make white parts (colorless parts) transparent. In addition, the generating unit 11a assigns a predetermined operation program corresponding to the type of object to the image of the operation object generated as described above. This allows the operation object to operate according to the predetermined operation program even before it is operated by the user. After the operation by the user is started, the operation object will execute an operation by prioritizing the operation command by the user over the operation program.
[0040] The display device 30 displays a two-dimensional image from the viewpoint of the overhead camera under the control of the control device 10. In this embodiment, the display device 30 includes a projector 31 and a screen 32. The projector 31 receives an image drawn by the control device 10 and projects the image onto the surface of the screen 32. Any known projector 31 and screen 32 may be used. The screen 32 may be replaced by a flat wall or the like. In addition, a large monitor, a liquid crystal display, an organic EL display, or the like may also be used as the display device 30. However, it is preferable to use the projector 31 and the screen 32 as the display device 30 because an image can be displayed on a large screen easily and at low cost.
[0041] The operation terminal 40 is a terminal device held by a user who operates an operation object. An example of the operation terminal 40 is a mobile information terminal such as a smartphone or a tablet terminal. The operation terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, a photographing unit 45, a gyro sensor 46, an acceleration sensor 47, a display unit 48, a vibrator 49, a speaker 51, a flashlight 52, and the like. In addition, the operation terminal 40 may include known modules that are included in general smartphones and tablet terminals.
[0042] The control unit 41 performs overall control of other elements included in the operation terminal 40. The control unit 41 is composed of a known processor such as a CPU and a known memory such as a RAM. The processor of the control unit 41 expands an application program stored in the storage unit 42 into the memory and controls other elements according to the application program. The storage unit 42 is an element for storing data used in processing by the control unit 41. Specifically, the storage unit 42 stores an application program for causing a mobile information terminal such as a general smartphone to function as the operation terminal 40 in this system. The application program stored in the storage unit 42 is read by the control unit 41, and processing according to the program is executed. Note that the storage unit 42 may store a plurality of other application programs. The storage unit 42 of the operation terminal 40 can be realized by a non-volatile memory such as an HDD or an SDD. The communication unit 43 is an element for communicating with the control device 10 through a communication line such as the Internet.
[0043] The input unit 44 is an element for inputting operation information by a user to the operation terminal 40. As the input unit 44, a known input device such as a touch panel, a mouse, a keyboard, or a stylus pen may be used. The input unit 44 may be physically separable from the operation terminal 40, in which case the input unit 44 is connected to the operation terminal 40 by a short-range wireless communication standard such as Bluetooth (registered trademark). Also, a touch panel display may be configured by arranging a touch panel (input unit 44) on the front of a display (display unit 48).
[0044] The photographing unit 45 is an element for acquiring image data of still images or videos. A known digital camera built into the operation terminal 40 may be used as the photographing unit 45. The image data acquired by the photographing unit 45 is sent to the control unit 41, where it is subjected to a predetermined arithmetic processing, and then stored in the storage unit 42. The camera is realized by, for example, a lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a CCD image sensor unit or a CMOS image sensor unit, a digital signal processor (DSP) that reads the amount of charge from the photoelectric conversion element and generates image data, an IC memory, and the like.
[0045] The gyro sensor 46 is a sensor for continuously measuring the angular velocity of the operation terminal 40. An example of the gyro sensor 46 is a three-axis vibration gyro sensor that vibrates an element to detect the angular velocity from the Coriolis force applied to the element. The acceleration sensor 47 is a sensor for continuously measuring the acceleration of the operation terminal 40.
[0046] The acceleration sensor 47 detects three-dimensional inertial motion (translational motion in three orthogonal axial directions), and a three-axis acceleration sensor can be used. The measurement method of the acceleration sensor 47 may be any known method such as a frequency change type, a piezoelectric type, a piezoresistance type, or a capacitance type.
[0047] The display unit 48 is an element for displaying a predetermined image (still image and video), and is configured with a known display device such as a liquid crystal display or an organic EL display. The vibrator 49 is an element for vibrating the operation terminal 40. The speaker 51 is an element for outputting sound. The flashlight 52 is an element for generating light, and is configured with a known light-emitting element such as an LED. The display unit 48, the vibrator 49, the speaker 51, and the flashlight 52 function as output units of the operation terminal 40. The control unit 41 of the operation terminal 40 may drive these output units in response to a reaction command generated by the terminal control unit 11i of the control device 10 described above.
[0048] As shown in FIG. 1, the detection device 60 detects the presence of a person near the display screen (screen 32) of the display device 30 and the contact position on the display screen (screen 32). As the detection device 60, various known sensors such as a range sensor, an infrared sensor, an optical sensor, an ultrasonic sensor, a capacitance sensor, a pressure sensor, a sound sensor, and a temperature sensor can be adopted. As the detection device 60, it is also possible to adopt a plurality of sensors having different functions, such as a combination of an infrared sensor and a pressure sensor. In particular, the detection device 60 preferably adopts a range sensor and detects the contact position (coordinate position on the screen 32) when a user (viewer) touches the screen 32. The range sensor is a two-dimensional scanning type optical distance measuring sensor that emits an inspection light (laser) along the screen 32 and measures the distance and angle to the contact position by the viewer by scanning the inspection light. Information detected by the range sensor is transmitted to the control device 10. If the position coordinates of the range sensor in real space are known, the position coordinates of the object in real space can be detected by measuring the distance and angle to the object touched by the light emitted from the range sensor. The detection information by such a detection device 60 is sent to the control device 10 in real time and is used, for example, to control the operation of an operational object or a non-operational object.
[0049] FIG. 3 is a schematic diagram showing main features of the image display system 100 according to the present embodiment. The first feature is that the operation of the operation object PO displayed on the operation terminal 40 and operated by the user is synchronized on the display screen of the display device 30. The second feature is that when the user touches the operation object PO on the display screen of the display device 30, the operation terminal 40 that is operating the operation object PO reacts by vibrating, for example. The third feature is that when an operation object PO that has not yet been operated is displayed on the display screen of the display device 30, if the user photographs the operation object PO with the camera of the operation terminal 40, the operation object PO can be operated on the operation terminal 40. The following describes the sequence for realizing each of these features. FIG. 4 shows the sequence for the first feature, FIG. 5 shows the sequence for the second feature, and FIG. 6 shows the sequence for the third feature.
[0050] 4 shows an example of the first feature described above, that is, a process for synchronizing the image display of an operation object and the image display of an image. First, the user selects an operation object that the user wishes to operate via the operation terminal 40 (step S1-1). At this time, for example, a list of the operation objects that can be operated is displayed on the display unit 48 of the operation terminal 40. The user may select one operation object that the user wishes to operate from the list. The operation terminal 40 transmits to the control device 10 identification information (ID) unique to the operation object selected by the user and identification information (IP address, etc.) unique to this operation terminal 40.
[0051] Next, the control device 10 grants the operation authority for the operation object selected by the user to the operation terminal 40 of the user (step S1-2). Specifically, the object management unit 11d of the control device 10 associates the identification information (ID) unique to the operation object received from the operation terminal 40 with the identification information (IP address, etc.) unique to this operation terminal 40, and registers them in the database of the storage unit 12. Thus, when the control device 10 receives an operation command for the operation object from the operation terminal 40, it can determine that the command is an operation command for the operation object associated with the operation terminal 40 in the database. Note that, when the operation authority for the operation terminal 40 is to be released, the identification information of the operation terminal 40 may be deleted from the database of the storage unit 12.
[0052] Next, the camera control unit 11b of the control device 10 assigns an object viewpoint camera to the operation object selected by the user, and controls the object viewpoint camera to capture the three-dimensional virtual space from a predetermined viewpoint (step S1-3). The viewpoint and the width of the field of view of the object viewpoint camera can be set for each type of operation object.
[0053] FIG. 7 shows an example of the correspondence between the operation object PO and the object viewpoint camera VC. In FIG. 7(a), a third-person object viewpoint camera VC is assigned to an airplane-shaped operation object PO. In this case, the camera control unit 11b controls the object viewpoint camera VC to be positioned diagonally above and behind the operation object PO and to capture the operation object PO. In FIG. 7(b), a first-person object viewpoint camera VC is assigned to an airplane-shaped operation object PO. In this case, the camera control unit 11b controls the object viewpoint camera VC to be positioned in front of the operation object PO and to capture the image in front of the operation object PO. In the first-person viewpoint, the operation object PO does not enter the field of view of the object viewpoint camera VC. The user may be allowed to arbitrarily select whether the object viewpoint camera VC is the third-person viewpoint (FIG. 7(a)) or the first-person viewpoint (FIG. 7(b)).
[0054] Also, in FIG. 7(c), a wide-angle object-viewpoint camera VC from a first-person viewpoint is assigned to a butterfly-shaped operation object PO. The viewing angle of a normal object-viewpoint camera VC is generally set to 40 to 50 degrees, but as shown in FIG. 7(c), the viewing angle of the object-viewpoint camera VC can be set to 90 to 360 degrees, more specifically, 180 to 340 degrees. When such a wide-angle object-viewpoint camera VC is set, the image displayed on the display screen of the operation terminal 40 is distorted at both the left and right ends of the screen. Also, in FIG. 7(d), a zoomable object-viewpoint camera VC from a first-person viewpoint is assigned to a bird-shaped operation object PO. For example, when the user of the operation terminal 40 touches the display screen, the field of view of the object-viewpoint camera VC is zoomed around the touch position, and when the user touches the display screen again, the zoom is released. Although not shown, it is also possible to assign multiple object-viewpoint cameras VC with different viewing directions to one operation object PO (three or more are also possible). When two or more object viewpoint cameras VC are set in this way, images from two or more viewpoints are displayed simultaneously on the display screen of the operation terminal 40.
[0055] Next, the drawing processing unit 11e of the control device 10 draws a two-dimensional image from the viewpoint of the object viewpoint camera (step S1-4). That is, the drawing processing unit 11e converts the three-dimensional virtual space expressed in the world coordinate system (X, Y, Z) into the screen coordinate system (U, V) from the object viewpoint camera to generate a two-dimensional image. After that, the second display control unit 11g of the control device 10 transmits the two-dimensional image generated here to the operation terminal 40 via the communication unit 13 (step S1-5).
[0056] Next, the operation terminal 40 displays the two-dimensional image received from the control device 10 on the display unit 48 (step S1-6). Since this two-dimensional image is from the viewpoint of an object viewpoint camera associated with the operation object, the user can operate the operation object while viewing this two-dimensional image.
[0057] The user operates the operation object using the operation terminal 40 (step S1-7). The method of operating the operation object is not particularly limited, and a known method can be adopted. For example, if the operation terminal 40 has a touch panel display, a cursor may be displayed on the touch panel display, and the user may touch the cursor to operate the operation object. If the operation terminal 40 has a gyro sensor 46 or an acceleration sensor 47, the acceleration and angular velocity detected by the user when the operation terminal 40 is tilted by the user may be used as operation information for the operation object. The control unit 41 of the operation terminal 40 generates an operation command for the operation object based on the information thus input by the user, and transmits the command to the control device 10.
[0058] Next, the object control unit 11c of the control device 10 controls the movement of the control object in the three-dimensional virtual space based on the operation command received from the operation terminal 40 (step S1-8). For example, the object control unit 11c may cause the control object to perform tilting, panning, forward movement, backward movement, acceleration, deceleration, or other actions (attack, jump, etc.) in response to the operation command. In addition, the camera control unit 11b controls the object viewpoint camera to follow the control object in accordance with the movement of the control object. Note that a series of processes related to the operation of this control object (steps S1-3 to S1-8) are repeated.
[0059] On the other hand, the camera control unit 11b of the control device 10 controls the overhead camera in parallel with the control of the object viewpoint camera described above (step S1-9). The overhead camera is for capturing images of a three-dimensional virtual space without depending on a specific object. As described above, the camera control unit 11b may control the overhead camera to move on a predetermined track in the three-dimensional virtual space, for example. The drawing processing unit 11e of the control device 10 draws a two-dimensional image from the viewpoint of the overhead camera (step S1-10). That is, the drawing processing unit 11e converts the three-dimensional virtual space expressed in the world coordinate system (X, Y, Z) into the screen coordinate system (U, V) from the overhead camera to generate a two-dimensional image. After that, the first display control unit 11f of the control device 10 outputs the two-dimensional image generated here to the projector 31 of the display device 30 via a predetermined I / O interface (step S1-11).
[0060] Next, the projector 31 of the display device 30 displays the two-dimensional image input from the control device 10 by projecting it onto the screen 32 (step S1-12). At this time, an object being operated by the operation terminal 40 may appear in the field of view of the three-dimensional virtual space captured by the overhead camera. In that case, the object being displayed by the display device 30 moves according to the operation of the operation terminal 40. In other words, the object being displayed and operated by the operation terminal 40 and the object being displayed by the display device 30 are the same in the three-dimensional virtual space, so it can be said that these objects are synchronized with each other. Note that a series of processes (steps S1-9 to S1-12) for displaying the image from the overhead camera on the display device 30 are repeated.
[0061] 5 shows an example of the second feature mentioned above, that is, a process in which the operation terminal 40 reacts when an operation object displayed on the display device 30 is touched. First, the detection device 60 detects the touch position of the user (viewer) on the screen 32 of the display device 30 (step S2-1). The detection device 60 detects the x and y coordinates of the touch position of the viewer on the plane of the screen 32, and outputs this information to the control device 10.
[0062] Next, the contact position determination unit 11h of the control device 10 determines whether the viewer has touched the operation object displayed on the screen 32 of the display device 30 based on the detection information input from the detection device 60 (step S2-2). Then, when the contact position determination unit 11h determines that the viewer has touched the operation object, the object control unit 11c controls the operation object so that the operation object executes a predetermined action in the three-dimensional virtual space (step S2-3). The action of the operation object at this time is not particularly limited, but it is possible to vibrate the operation object, generate a predetermined effect around the operation object, accelerate or decelerate the operation object, or make the operation object bounce. Note that the action performed by the operation object in response to the viewer's contact may differ depending on the type of the operation object. In addition, at this time, the contact position determination unit 11h can also determine whether the viewer has touched a non-operation object. When it is determined that the viewer has touched a non-operation object, the object control unit 11c may control the operation of the operation object so that the non-operation object executes a predetermined action in the three-dimensional virtual space.
[0063] Next, the terminal control unit 11i of the control device 10 specifies the operation terminal 40 that is operating the operation object that is determined by the contact position determination unit 11h to have been contacted by the viewer (step S2-4). As described above, the storage unit 12 of the control device 10 stores the operation objects currently existing in the three-dimensional virtual space in association with the operation terminals 40 that are operating them. The terminal control unit 11i can specify the operation terminal 40 that is operating the operation object that the viewer has contacted by referring to this database. After that, the terminal control unit 11i generates a reaction command for the operation terminal 40 specified in this manner and transmits this command (step S2-5). The reaction command is basically a command related to the drive control of the output unit provided in the operation terminal 40. The reaction command is not particularly limited, and may be, for example, a command to display something on the display screen of the display unit 48 of the operation terminal 40, a command to vibrate the vibrator 49 of the operation terminal 40, a command to output a predetermined sound from the speaker 51 of the operation terminal 40, or a command to turn on the flashlight 52 of the operation terminal 40. The reaction command transmitted to the operation terminal 40 may be different depending on the type of the operation object. After that, when the control unit 41 of the operation terminal 40 receives a reaction command from the control device 10, the control unit 41 executes a reaction process according to the reaction command (step S2-6). Specifically, the control unit 41 may control output units such as the display unit 48, the vibrator 49, the speaker 51, and the flashlight 52 according to the reaction command.
[0064] Fig. 6 shows an example of the above-mentioned third characteristic point, that is, a process for photographing an operation object on the display screen of the display device 30 with the photographing unit 45 of the operation terminal 40, thereby enabling the operation object to be operated with the operation terminal 40. In particular, Fig. 6 shows an example of a sequence in which an object drawn or painted by a user is taken into a three-dimensional virtual space and becomes an operation object, and the user is then able to operate this operation object created by the user with his or her own operation terminal 40.
[0065] 6, a user scans a printing paper on which an object has been drawn or colored using the image capture device 20 (step S3-1). This causes an image of the printing paper to be captured by the image capture device 20. The image data of the printing paper captured by the image capture device 20 is output to the control device 10 via a specified I / O interface (step S3-2).
[0066] Next, the generating unit 11a of the control device 10 generates an operation object based on the image data input from the image capture device 20 (step S3-3). The process of generating an operation object is as described above. The generating unit 11a also gives a predetermined operation program to the generated operation object to make it appear in the three-dimensional virtual space. As a result, the operation object created by the user performs a predetermined operation according to the operation program in the three-dimensional virtual space. Also, as described above, an overhead camera exists in the three-dimensional virtual space, and an image from this overhead camera is output in real time to the projector 31 of the display device 30 (step S3-4). Since the operation object created by the user operates in the three-dimensional virtual space, it may be photographed by the overhead camera, in which case the operation object created by the user will be projected on the screen 32 by the projector 31 of the display device 30.
[0067] Next, the user finds the operation object that he / she created from among the images displayed on the screen 32, and photographs it with the photographing unit 45 of his / her operation terminal 40 (step S3-5). The control unit 41 of the operation terminal 40 stores the image photographed by the photographing unit 45 in the storage unit 42, and transmits it to the control device 10 via the communication unit 43 (step S3-6).
[0068] Next, the image analysis unit 11j of the control device 10 analyzes the captured image received from the operation terminal 40 via the communication unit 13, and determines whether or not the captured image includes an unoperated operation object (step S3-7). If the image analysis unit 11j determines that the captured image includes an operation object, it performs individual identification of the operation object based on features such as the shape and color of the operation object in the captured image (step S3-8).
[0069] Next, the object management unit 11d of the control device 10 grants the operation authority of the operation object identified by the image analysis unit 11j to the operation terminal 40 that captured the image of the operation object (step S3-9). Specifically, the identification information (ID) of the operation object and the identification information (IP address, etc.) of the operation terminal 40 are registered in the database of the storage unit 12. This allows the user to operate the operation object that he or she created using the operation terminal 40. The subsequent process is as described above. When the user inputs operation information of the object to the operation terminal 40, the operation command is transmitted from the operation terminal 40 to the control device 10 (step S3-10), and the control device 10 controls the operation of the operation object in the three-dimensional virtual space based on the operation command (step S3-11).
[0070] In this way, even if there are many operation objects in the three-dimensional virtual space and it is difficult for the user to find the operation object he / she created from the list, the user can easily operate the operation object by taking a picture of the operation object he / she created with the camera. Also, it is possible to provide the user with a game of finding the operation object he / she created from the objects displayed on the screen 32.
[0071] In the above, in order to express the contents of the present invention, the present specification has described the embodiments of the present invention with reference to the drawings. However, the present invention is not limited to the above-mentioned embodiments, and includes modifications and improvements that are obvious to a person skilled in the art based on the matters described in the present specification. [Explanation of symbols]
[0072] 10...Control device 11...Central control unit 11a... Generation unit 11b... Camera control unit 11c... object control unit 11d... object management unit 11e: drawing processing unit 11f: first display control unit 11g...Second display control section 11h...Touch position determination section 11i... Terminal control unit 11j... Image analysis unit 12...Memory section 13...Communication section 20...Image capture device 30...Display device 31…Projector 32…Screen 40: Operation terminal 41: Control unit 42: Memory section 43: Communication section 44...input section 45...photographing section 46...Gyro sensor 47...Acceleration sensor 48...Display unit 49...Vibrator 51...Speaker 52...Flashlight 60...detection device 100...image display system
Claims
1. An image display system including a display device, an operation terminal, and a control device connected to the display device and the operation terminal via a wired or wireless network, The control device includes: A generating unit that generates a three-dimensional virtual space including an operation object; a camera control unit that controls an overhead camera in the three-dimensional virtual space and the operation object's line of sight or an object viewpoint camera that follows the operation object; a first display control unit that causes the display device to display an image captured by the overhead camera; a second display control unit that causes the operation terminal to display an image from the object viewpoint camera; An object control unit that controls the action of the operation object in the three-dimensional virtual space in accordance with an operation command from the operation terminal. system.
2. an image capture device for scanning the physical medium on which the object is depicted to capture image data; The generation unit generates the operation object in the three-dimensional virtual space based on the image data. The system of claim 1 .
3. The display device further includes a detection device capable of detecting a contact position on a display screen of the display device, The control device further includes a terminal control unit that generates a reaction command based on the detection information by the detection device and transmits the reaction command to the operation terminal; The operation terminal has an output unit that performs an output in response to the reaction command. The system of claim 1 .
4. When the terminal control unit determines that the contact position detected by the detection device corresponds to the operation object displayed on the display screen, the terminal control unit generates the reaction command to the operation terminal that is operating the operation object. The system of claim 3.
5. The operation terminal has an image capturing unit, and transmits an image captured by the image capturing unit to the control device; The control device includes: an image analysis unit that analyzes the captured image and determines whether the captured image includes the operation object; The image capturing device further includes an object management unit that, when it is determined that the captured image includes the operation object, grants an operation authority for the operation object to the operation terminal that captured the image of the operation object. The system of claim 1 .
6. A computer program for causing an information communication terminal to function as an operation terminal in the system according to claim 1.
7. A control device connected to a display device and an operation terminal via a wired or wireless network, A generating unit that generates a three-dimensional virtual space including an operation object; a camera control unit that controls an overhead camera in the three-dimensional virtual space and the operation object's line of sight or an object viewpoint camera that follows the operation object; a first display control unit that causes the display device to display an image captured by the overhead camera; a second display control unit that causes the operation terminal to display an image from the object viewpoint camera; An object control unit that controls the action of the operation object in the three-dimensional virtual space in accordance with an operation command from the operation terminal. Control device.
8. A computer program for causing a computer to function as the control device according to claim 7.
9. An image display method executed by a system including a display device, an operation terminal, and a control device connected to the display device and the operation terminal via a wired or wireless network, A step of generating a three-dimensional virtual space including an operation object by the control device; a step of the control device controlling an overhead camera and the operation object's line of sight or an object viewpoint camera following the line of sight in the three-dimensional virtual space; a step of causing the control device to display an image from the overhead camera on the display device; a step of causing the control device to display an image from the object viewpoint camera on the operation terminal; The control device controls a movement of the operation object in the three-dimensional virtual space in accordance with an operation command from the operation terminal. Image display method.
Citation Information
Patent Citations
Video display system
JP2002073246A
Interactive experience system and method, and controller therefor
JP2014514621A
Display control device, display control program, display system and display control method
JP2018045663A
Multi-user game system with character-based generation of projection view
US20160279509A1
Painting image display system
JP2017037614A
