A program for providing a sense of presence, a device for providing a sense of presence, a method for providing a sense of presence, and a recording medium.

Abstract

This disclosure provides a program that can deliver an immersive experience. [Solution] The present disclosure's program for providing a sense of presence includes an information acquisition procedure, an extraction procedure, a layout image generation procedure, and an image output procedure. The information acquisition procedure acquires a photographic image of the seating arrangement of a subject in a shooting area. The extraction procedure extracts unit images containing the subject in predetermined units from the photographic image. The layout image generation procedure generates a layout image by arranging the unit images according to predetermined rules. The image output procedure outputs the layout image to a display device worn by the subject in the display area.

Description

Technical Field

[0001] The present disclosure relates to a presence providing program, a presence providing device, a presence providing method, and a recording medium.

Background Art

[0002] In Patent Document 1, a lecture system including a communication lecture server that distributes lecture content for viewing a lecture conducted in a virtual space to a viewing terminal is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Online viewing such as the online lecture described in Patent Document 1 is performed by a single viewer, and thus may give the viewer an impression of lacking presence.

[0005] Therefore, an object of the present disclosure is to provide a presence providing program, a presence providing device, a presence providing method, and a recording medium that can provide an environment with presence.

Means for Solving the Problems

[0006] In order to achieve the above object, the presence providing program of the present disclosure includes an information acquisition procedure, an extraction procedure, an arrangement image generation procedure, and an image output procedure, the information acquisition procedure acquires a captured image obtained by capturing the seating status of a person to be captured in a shooting area, the extraction procedure extracts unit images each including the person to be captured in a predetermined unit from the captured image, The aforementioned arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output procedure involves outputting the arranged image to a display device worn by the person to be viewed in the display area. This is a program that causes a computer to execute each of the aforementioned steps.

[0007] The presence-providing device disclosed herein is It includes an information acquisition unit, an extraction unit, a layout image generation unit, and an image output unit, The aforementioned information acquisition unit acquires images capturing the seating status of the subject in the shooting area. The extraction unit extracts unit images containing the subject of photography in predetermined units from the captured images. The arrangement image generation unit generates an arrangement image in which the unit images are arranged according to a predetermined rule, The image output unit outputs the arranged image to a display device worn by the person to be viewed in the display area. It is a device.

[0008] The method of providing a sense of realism in this disclosure is: This process includes an information acquisition step, an extraction step, a layout image generation step, and an image output step. The aforementioned information acquisition step involves acquiring images of the seating arrangement of the subject in the shooting area. The extraction step involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation step generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output step outputs the arranged image to a display device worn by the person to be viewed in the display area. This method involves each of the aforementioned steps being performed by a computer.

[0009] The recording medium disclosed herein is This includes information acquisition procedures, extraction procedures, placement image generation procedures, and image output procedures. The aforementioned information acquisition procedure involves acquiring images of the seating arrangement of the subject in the shooting area, The extraction procedure extracts unit images included in the subject to be photographed from the photographed image in a predetermined unit, The arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to a predetermined rule, The image output procedure outputs the arrangement image to a display device worn by the display target person in the display area. It is a computer-readable recording medium on which a program for causing a computer to execute each of the above procedures is recorded.

Advantages of the Invention

[0010] According to the present disclosure, an environment with a sense of presence can be provided.

Brief Description of the Drawings

[0011] [Figure 1] FIG. 1 is a block diagram showing a configuration example of a presence providing device of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the presence providing device of the present disclosure. [Figure 3] FIG. 3 is a flowchart showing an example of a procedure by the presence providing program of the present disclosure. [Figure 4] FIG. 4 is a flowchart showing another example of a procedure by the presence providing program of the present disclosure. [Figure 5] FIG. 5 is a block diagram showing another example of the configuration of the presence providing device of the present disclosure. [Figure 6] FIG. 6 is a flowchart showing another example of a procedure by the presence providing program of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram showing an example of a usage form of the presence providing device of the present disclosure. [Figure 8] FIG. 8(A) is a schematic diagram showing an example of the arrangement image of the present disclosure. FIG. 8(B) is a schematic diagram showing an example in which the presence providing device of the present disclosure outputs the arrangement image to a display device.

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The present disclosure is not limited to the following embodiments. In the following figures, the same parts are denoted by the same reference numerals. Also, unless otherwise specified, the descriptions of the respective embodiments can be mutually referred to, and the configurations of the respective embodiments can be combined unless otherwise specified. In the present disclosure, each drawing may apply to one or more embodiments.

[0013] [Embodiment 1] The sense-of-presence providing program of the present disclosure is a program for causing a computer to execute an information acquisition procedure, an extraction procedure, an arrangement image generation procedure, and an image output procedure. The sense-of-presence providing program of the present disclosure can also be said to be a program for causing a computer to function as an information acquisition procedure, an extraction procedure, an arrangement image generation procedure, and an image output procedure. Also, the sense-of-presence providing program of the present disclosure can also be said to be, for example, a program for causing a computer to execute each step of the sense-of-presence providing method described later.

[0014] The information acquisition procedure acquires a captured image of the seating situation of the person to be photographed in the photographing area, the extraction procedure extracts unit images in which the person to be photographed is included in a predetermined unit from the captured image, the arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to a predetermined rule, and the image output procedure outputs the arrangement image to a display device worn by the person to be displayed in the display area.

[0015] Each of the aforementioned procedures can be reinterpreted, for example, as a "process." The presence-providing program of this disclosure may also be recorded on, for example, a computer-readable storage medium. The storage medium is, for example, a non-transitory computer-readable storage medium. The storage medium is not particularly limited and includes, for example, random access memory (RAM), read-only memory (ROM), hard disk (HD), flash memory (e.g., SSD (Solid State Drive), USB flash memory, SD / SDHC card, etc.), optical disc (e.g., CD-R / CD-RW, DVD-R / DVD-RW, BD-R / BD-RE, etc.), magneto-optical disk (MO), floppy disk (FD), etc. The presence-providing program of this disclosure (for example, also called a programming product or program product) may also be delivered, for example, from an external computer. The "delivery" may be, for example, delivered via a communication network or delivered via a wired device. The presence-providing program of this disclosure may be installed and executed on the device to which it is delivered, or it may be executed without being installed. An information processing device capable of executing the presence-providing program of this disclosure may be, for example, the presence-providing device of this disclosure.

[0016] Next, an example of the configuration of the presence-providing device of this disclosure will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the configuration of the presence-providing device 10 of this disclosure (hereinafter also referred to as "the device 10"). As shown in Figure 1, the device 10 includes an information acquisition unit 11, an extraction unit 12, a placement image generation unit 13, and an image output unit 14. The device 10 may also include, for example, an input unit, an output unit, a display unit, and / or a storage unit, although these are not shown. The information acquisition unit 11, the extraction unit 12, the placement image generation unit 13, and the image output unit 14 can each execute, for example, the information acquisition procedure, the extraction procedure, the placement image generation procedure, and the image output procedure in the presence-providing program of this disclosure.

[0017] The device 10 may be, for example, a single device including the aforementioned parts, or it may be a device in which each of the aforementioned parts can be connected via a communication network. Furthermore, the device 10 can be connected to an external device described later via the communication network. The communication network is not particularly limited and can use a known network, for example, it may be wired or wireless. Examples of the communication network include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, LPWA, etc. The wireless communication may be in the form of direct communication between devices (Ad Hoc communication), infrastructure communication, indirect communication via an access point, etc. The device 10 may be, for example, incorporated into a server as a system. Furthermore, the device 10 may be, for example, a personal computer (PC, e.g., desktop or notebook), smartphone, tablet terminal, etc., on which the program of this disclosure is installed. The device 10 may also be in the form of cloud computing or edge computing, for example, in which at least one of the aforementioned parts is on a server and the other aforementioned parts are on a terminal.

[0018] Figure 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, memory 102, bus 103, storage device 104, input device 105, output device 106, communication device 107, etc. Each part of the device 10 is interconnected via the bus 103 through its respective interface (I / F).

[0019] The central processing unit 101 operates in coordination with other components via controllers (system controller, I / O controller, etc.) and is responsible for the overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program disclosed herein (a program that provides a sense of presence) and other programs, and also reads and writes various types of information. Specifically, for example, the central processing unit 101 functions as an information acquisition unit 11, an extraction unit 12, a layout image generation unit 13, and an image output unit 14. The device 10 may also include other computing devices such as a CPU, GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), or a combination thereof as computing devices.

[0020] Bus 103 can also be connected to external devices, for example. Examples of such external devices include external storage devices (external databases, etc.), printers, external input devices, external display devices, and external imaging devices. The device 10 can be connected to an external network (the aforementioned communication network) via a communication device 107 connected to bus 103, for example, and can also be connected to other devices via the external network.

[0021] Memory 102 may be, for example, main memory. When the central processing unit 101 performs processing, memory 102 reads various operational programs, such as the program of this disclosure, stored in the storage device 104 (described later), and the central processing unit 101 receives data from memory 102 and executes the program. The main memory may be, for example, RAM (random access memory). Alternatively, memory 102 may be, for example, ROM (read-only memory).

[0022] The storage device 104 is also called an auxiliary storage device, for example, in relation to the main memory (primary memory). As described above, the storage device 104 stores an operating program including the program of this disclosure. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited and may be internal or external, for example, an HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD). If the device 10 includes the storage unit, for example, the storage device 104 functions as the storage unit. The storage unit can record, for example, various types of information, which will be described later.

[0023] In this device 10, the memory 102 and storage device 104 can also store various types of information, such as log information, information obtained from an external database (not shown) or external devices, information generated by this device 10, and information used by this device 10 when executing processing. In this case, the memory 102 and storage device 104 may store, for example, the user information of this device as described above. At least some of the information may be stored on an external server other than the memory 102 and storage device 104, or it may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0024] The device 10 further includes, for example, an input device 105 and an output device 106. The input device 105 may include, for example, a pointing device such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; an audio input means such as a microphone; and so on. The output device 106 may include, for example, a display device such as an LED display or liquid crystal display; an audio output device such as a speaker; a printer; and so on. In this embodiment 1, the input device 105 and the output device 106 are configured separately, but the input device 105 and the output device 106 may be configured as an integrated unit, such as a touch panel display.

[0025] An example of processing performed by the presence-providing program of this disclosure will be explained in more detail with reference to Figure 3. Figure 3 is a flowchart illustrating an example of each step of the presence-providing program of this disclosure. The presence-providing program of this disclosure can be implemented, for example, using the apparatus 10 of this disclosure shown in Figure 1 or Figure 2. However, the presence-providing program of this disclosure is not limited to a program that uses the apparatus 10 of this disclosure.

[0026] The information acquisition unit 11 acquires images capturing the seating status of the subject in the shooting area (S11, information acquisition procedure).

[0027] The aforementioned shooting area is, for example, an area where at least one seat is arranged and at least one subject to be photographed is seated in the seat. The seat may be, for example, a desk, a chair, or a combination thereof. The aforementioned shooting area may be, for example, an area where a shooting device is installed. The shooting device may be, for example, a camera. The number of shooting devices may be, for example, one or two or more per shooting area. The shooting direction of the shooting devices may be the same direction between the shooting devices or different directions between the shooting devices if there are two or more shooting devices. The shooting direction of the shooting devices may be, for example, forward, backward, left, and right if there are four shooting devices. The aforementioned shooting area may be, for example, a classroom, lecture hall, or presentation hall. The number of aforementioned shooting areas may be, for example, one or two or more. The aforementioned shooting areas may be, for example, areas in the same facility or areas in different facilities if there are two or more shooting areas. The aforementioned shooting area may be, for example, an area within the same facility as the display area described later, or an area within a different facility from the display area described later. The subjects of the shooting may include, for example, students, pupils, or course participants.

[0028] This disclosure may, for example, set at least one of a plurality of areas as a display area, and set an area other than the area set as the display area as the shooting area. For example, if there are three areas A to C, if area A is set as the display area, areas B and C may be set as the shooting areas; if area B is set as the display area, areas A and C may be set as the shooting areas; if area C is set as the display area, areas A and B may be set as the shooting areas; and combinations of these may be set simultaneously. In this way, this disclosure may, for example, set interrelationships between the display area and the shooting area for a plurality of areas.

[0029] The captured image is, for example, an image of the seated position of the subject in the shooting area. The captured image may be, for example, a still image or a video. The number of captured images may be, for example, one or two or more. The shooting direction of the captured image may be the same direction between images or different directions between images if there are two or more captured images. The shooting direction of the captured image may be, for example, forward, backward, left, and right if there are four captured images.

[0030] The information acquisition unit 11 may, for example, acquire the captured image from the input device 105 provided by the device 10, or from an input device provided by a device other than the device 10. The information acquisition unit 11 may, for example, acquire the captured image from the memory 102 or storage device 104 provided by the device 10, or from the memory or storage device provided by a device other than the device 10. The information acquisition unit 11 may, for example, acquire the captured image from the imaging device installed in the imaging area.

[0031] The extraction unit 12 extracts unit images containing the subject of photography in predetermined units from the captured images (S12, extraction procedure).

[0032] The unit image is, for example, an image in which the subject being photographed is included in a predetermined unit. The unit image may be, for example, an image that includes only the subject being photographed, or an image that includes the subject being photographed and the equipment for the subject to sit on. The equipment for sitting on may be, for example, an explanation of the seat. The predetermined unit may be, for example, a seat unit, a row of seats unit, etc. The seat unit may be, for example, a unit corresponding to one subject being photographed. The row of seats unit may be, for example, a unit corresponding to a row of seats composed of multiple subjects being photographed. The unit image may be, for example, a seat-by-seat image, a row-by-seat image, etc., as described later.

[0033] The extraction unit 12 may, for example, extract seat-specific images in which the subject of photography is included on a seat-by-seat basis, or seat-row-specific images in which the subject of photography is included on a seat-row basis, as the unit images.

[0034] The extraction unit 12 may, for example, perform the extraction process of the unit image from the captured image by detecting a person for the subject included in the captured image. If the unit image is a seat-specific image, the extraction unit 12 may, for example, extract an image corresponding to the subject detected by the person detection as the seat-specific image. If the unit image is a seat-row-specific image, the extraction unit 12 may, for example, identify a seat row composed of multiple subjects detected by the person detection, and extract an image corresponding to the multiple subjects included in the identified seat row as the seat-row-specific image. The direction of the seat row may, for example, be left-right or up-down in the captured image. The extraction unit 12 may, for example, perform the person detection using a known method for person detection. Examples of known methods for person detection include using the OpenCV person detection library.

[0035] The arrangement image generation unit 13 generates an arrangement image in which the unit images are arranged according to predetermined rules (S13, arrangement image generation procedure).

[0036] The aforementioned arrangement image is, for example, an image obtained by arranging the unit images according to predetermined rules. Examples of predetermined rules include rules relating to a predetermined shape, rules relating to a predetermined side, and so on.

[0037] The arrangement image generation unit 13 may, for example, generate an image as the arrangement image when the unit images are extracted as seat-specific images by the extraction procedure, by arranging the seat-specific images in a predetermined shape.

[0038] The arrangement image generation unit 13 may, for example, perform the arrangement image generation process as the arrangement of the seat-specific images for vacant seats assigned to a predetermined shape when the unit image has been extracted as a seat-specific image by the extraction procedure. The predetermined shape may be, for example, a polygon, a circle, a semicircle, or a combination thereof. The vacant seats may be assigned, for example, along the sides of the predetermined shape or along the circumference of the predetermined shape. The arrangement image generation unit 13 may, for example, randomly arrange the seat-specific images for vacant seats assigned to the predetermined shape, or arrange them according to the selection of the person to be displayed in the display area.

[0039] The arrangement image generation unit 13 may, for example, generate an image as the arrangement image when the unit images are extracted as row-specific images by the extraction procedure, by arranging the row-specific images on a predetermined side.

[0040] The arrangement image generation unit 13 may, for example, perform the arrangement image generation process as the distribution of the seat row images to a predetermined side when the unit image is extracted as a seat row image by the extraction procedure. The predetermined side is, for example, one of the two sides of the display area divided in half based on the seat rows composed of the people to be displayed in the display area. Examples of combinations of the predetermined sides include the right and left sides, the front and back sides, etc. Specifically, the arrangement image generation unit 13 may, for example, divide the total number of seat rows related to the seat row images by 2. If there is no remainder from the division, the arrangement image generation unit 13 may, for example, place a number of seat row images corresponding to the quotient of the division on each of the two sides. The arrangement image generation unit 13 may, for example, if a remainder is obtained from the division, place the number of seat row images corresponding to the quotient on one of the two sides, and place the number of seat row images corresponding to the number obtained by subtracting the quotient from the total on the other side. The arrangement image generation unit 13 may, for example, distribute the seat row images to the predetermined side and then randomly arrange the seat row images on the predetermined side, or arrange them according to the selection of the person to be displayed in the display area.

[0041] The image output unit 14 outputs the arranged image to a display device worn by the person to be displayed in the display area (S14, image output procedure).

[0042] The image output unit 14 may, for example, output the arranged image to the display device as an image relating to augmented reality or a mixed reality.

[0043] The display device is, for example, a device worn by the person to be displayed in the display area. The display device is, for example, a head-mounted display (HMD). The type of HMD is, for example, glasses type, goggle type, etc. The glasses type HMD is, for example, smart glasses. The smart glasses are, for example, AR glasses, MR glasses, etc.

[0044] The display device includes, for example, an image acquisition unit, an image display unit, and a display screen. The display device is connected to the device 10, for example, via a wired or wireless communication network. When the display device acquires the arrangement image from the device 10, for example, it may perform the following processes: The image acquisition unit acquires the arrangement image from the device 10. The image display unit displays the arrangement image on the display screen provided by the display device. In this case, the image display unit may superimpose the arrangement image onto an image of real space.

[0045] The method for providing a sense of presence as disclosed herein (hereinafter also referred to as the Method of Disclosure) is a method that is implemented by, for example, replacing each "procedure" in the Program of Disclosure with "process". Specifically, the Method of Disclosure includes, for example, an information acquisition process, an extraction process, a layout image generation process, and an image output process. The information acquisition process acquires a photographic image of the seating situation of a subject in a shooting area. The extraction process extracts unit images containing the subject in predetermined units from the photographic image. The layout image generation process generates a layout image by arranging the unit images according to predetermined rules. The image output process outputs the layout image to a display device worn by a subject in a display area. The Method of Disclosure can be implemented, for example, using the Apparatus 10 of the Disclosure shown in Figure 1 or Figure 2. However, the Method of Disclosure is not limited to, for example, a method using the Apparatus 10 of the Disclosure. The Method of Disclosure can be implemented by referring to, for example, the descriptions in the Program and Apparatus of the Disclosure.

[0046] As described above, the presence-providing program of this disclosure can be used to provide a sense of presence by: acquiring images of the seating situation of subjects in a shooting area through an information acquisition procedure; extracting unit images containing the subjects in predetermined units from the acquired images through an extraction procedure; generating a layout image by arranging the unit images according to predetermined rules through a layout image generation procedure; and outputting the layout image to a display device worn by a subject in a display area through an image output procedure. Therefore, according to this disclosure, a sense of presence can be provided.

[0047] [Embodiment 2] This disclosure describes another example of a program that provides a sense of presence.

[0048] Another example of processing performed by the presence-providing program of this disclosure will be explained in more detail with reference to Figure 4. Figure 4 is a flowchart illustrating an example of each step of the presence-providing program of this disclosure. The presence-providing program of this disclosure can be implemented, for example, using the apparatus 10 of this disclosure shown in Figure 1. Furthermore, the presence-providing program of this disclosure can be implemented, for example, using the hardware configuration of the presence-providing apparatus 10 shown in Figure 2. However, the presence-providing program of this disclosure is not limited to, for example, a program utilizing the apparatus 10 of this disclosure.

[0049] The information acquisition unit 11 further acquires, for example, a speaker image (S11A, information acquisition procedure). That is, the information acquisition unit 11 acquires, for example, the captured image and the speaker image.

[0050] The speaker image is, for example, an image relating to the speaker. The speaker image may be, for example, an image containing only the speaker, or an image containing the speaker and tools related to the speaker. Examples of tools related to the speaker include a teacher's desk, a lectern, a blackboard, a whiteboard, or a combination thereof. The speaker may be, for example, a teacher or lecturer. The speaker image may be, for example, an image taken in the shooting area, or an image taken in an area other than the shooting area. The information acquisition unit 11 may, for example, acquire a speaker image taken of the speaker's speaking situation, extract a speaker image containing the speaker from the speaker image, and acquire the extracted speaker image as the speaker image. Examples of the speaking situation include a class setting or a lecture setting.

[0051] The information acquisition unit 11 may, for example, acquire the speaker image from the input device 105 provided by the device 10, or from an input device provided by a device other than the device 10. The information acquisition unit 11 may, for example, acquire the speaker image from the memory 102 or storage device 104 provided by the device 10, or from the memory or storage device provided by a device other than the device 10. The information acquisition unit 11 may, for example, acquire the speaker image from the imaging device installed in the imaging area.

[0052] Steps S12 to S13 may be performed, for example, in the same manner as steps S12 to S13 in Embodiment 1.

[0053] The image output unit 14 outputs, for example, the arrangement image and the speaker image to the display device (S14A, image output procedure).

[0054] The display device may, for example, perform the following processes when acquiring the arrangement image and the speaker image from the device 10. The image acquisition unit acquires the arrangement image and the speaker image from the device 10. The image display unit displays the arrangement image and the speaker image on the display screen provided by the display device. In this case, the image display unit may superimpose the arrangement image and the speaker image onto an image of the real space. The image display unit may display the speaker image at the position in the display area where speaking takes place. Examples of positions where speaking takes place include the position where a teacher's desk is placed, the position where a lectern is placed, the position where a blackboard is placed, the position where a whiteboard is placed, and so on.

[0055] The method for providing a sense of presence as disclosed herein (hereinafter also referred to as the Method of Disclosure) is a method that is implemented by, for example, replacing each "procedure" in the Program of Disclosure with "process". Specifically, in the Method of Disclosure, for example, the information acquisition process further acquires a speaker image, and the image output process outputs the arrangement image and the speaker image to the display device. The Method of Disclosure can be implemented, for example, using the Apparatus 10 of the Disclosure shown in Figure 1. However, the Method of Disclosure is not limited to, for example, a method using the Apparatus 10 of the Disclosure. The Method of Disclosure can be implemented by referring to, for example, the descriptions in the Program of the Disclosure and the Apparatus of the Disclosure.

[0056] According to the presence-providing program of this disclosure, for example, speaker images can be further acquired through an information acquisition procedure, and the arrangement image and the speaker images can be output to the display device through an image output procedure. Therefore, according to this disclosure, for example, it is possible to view the speaker's speech, such as the teacher's lesson or the lecturer's presentation, in a realistic environment.

[0057] [Embodiment 3] This disclosure describes another example of a program that provides a sense of presence.

[0058] Figure 5 is a block diagram showing the configuration of another example of the presence-providing device 10A. As shown in Figure 5, the presence-providing device 10A includes a stereophonic sound processing unit 15 and a sound output unit 16 in addition to the configuration of the presence-providing device 10 of Embodiment 1. The hardware configuration of the presence-providing device 10A is the same as that of the presence-providing device 10 in Figure 2, except that the central processing unit 101 has the configuration of the presence-providing device 10A in Figure 5 instead of the configuration of the presence-providing device 10 in Figure 1. The embodiments of this disclosure can be described by referring to, for example, the descriptions of other embodiments.

[0059] Another example of processing by the presence-providing program of this disclosure will be explained in more detail with reference to Figure 6. Figure 6 is a flowchart showing an example of each step of the presence-providing program of this disclosure. The presence-providing program of this disclosure can be implemented, for example, using the apparatus 10A of this disclosure shown in Figure 5. Furthermore, the presence-providing program of this disclosure can be implemented, for example, in the hardware configuration of the presence-providing device 10 of Figure 2, by using a hardware configuration in which the central processing unit 101 has the configuration of the presence-providing device 10A of Figure 5 instead of the configuration of the presence-providing device 10 of Figure 1. Note that the presence-providing program of this disclosure is not limited to a program using the apparatus 10A of this disclosure. The processing of the stereophonic sound processing unit 15 and the sound output unit 16 will be explained below. The processing of the stereophonic sound processing unit 15 and the sound output unit 16 can be appropriately inserted at any position in the flowchart of Figure 3 described in Embodiment 1, for example, but as shown in Figure 6, it is preferable that the processing of the stereophonic sound processing unit 15 and the sound output unit 16 be executed in parallel with, for example, S12 to S14.

[0060] The information acquisition unit 11 further acquires, for example, the voice of the subject being photographed, which is recorded in the shooting area (S11B, information acquisition procedure). That is, the information acquisition unit 11 acquires, for example, the captured image and the voice of the subject being photographed.

[0061] The information acquisition unit 11 may, for example, acquire the voice of the subject being filmed from a microphone device installed in the filming area. The number of microphone devices may be, for example, one or two or more per filming area. The direction of sound acquisition by the microphone devices may be the same direction between them or different directions between them, for example, if there are two or more microphone devices. The direction of sound acquisition by the microphone devices may be forward, backward, left, and right, for example, if there are four microphone devices.

[0062] The stereophonic sound processing unit 15 performs stereophonic sound processing on the voice of the subject being filmed, for example, based on the display position of the subject on the display screen of the display device (S15, stereophonic sound processing procedure).

[0063] The display position of the subject on the display screen of the display device is different from, for example, the position of the subject in the shooting area, which is the real space. For this reason, the voice of the subject needs to be converted into voice corresponding to the display position of the subject on the display screen of the display device, for example, by stereophonic sound processing. The stereophonic sound processing unit 15 may perform the stereophonic sound processing, for example, by adjusting the volume and the time it takes for the sound to travel for the voice of the subject. The stereophonic sound processing may be performed, for example, by known application software related to stereophonic sound processing.

[0064] The sound output unit 16 outputs, for example, the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to the sound output device in the display area (S16, sound output procedure).

[0065] The sound output unit 16 may, for example, output the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to a sound output device installed in the display area. In this case, the sound output device may be, for example, a loudspeaker. There may be one sound output device or two or more sound output devices per display area. The output direction of the sound output device may be the same direction between the sound output devices or different directions between the sound output devices if there are two or more sound output devices. The output direction of the sound output device may be forward, backward, left, and right if there are four sound output devices. The sound output unit 16 may also, for example, output the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to a sound output device worn by the subject being displayed. In this case, the sound output device may be, for example, headphones or earphones. The sound output device may be, for example, an accessory device of the display device.

[0066] Steps S12 to S14 may be performed, for example, in the same manner as steps S12 to S14 in Embodiment 1.

[0067] The method for providing a sense of presence as disclosed herein (hereinafter also referred to as the Method of Disclosure) is a method that is implemented by, for example, replacing each "procedure" in the Program of Disclosure with "process". Specifically, the Method of Disclosure further includes, for example, a stereophonic sound processing process and a sound output process, wherein the information acquisition process further acquires subject voice, which is the voice of the subject being filmed in the filming area; the stereophonic sound processing process performs stereophonic sound processing on the subject voice based on the display position of the subject on the display screen of the display device; and the sound output process outputs the subject voice, on which the stereophonic sound processing has been performed, to a sound output device in the display area. The Method of Disclosure can be implemented, for example, using the Apparatus 10A of the Disclosure shown in Figure 5. However, the Method of Disclosure is not limited to, for example, the method using Apparatus 10A of the Disclosure. The Method of Disclosure can be implemented by referring to, for example, the descriptions in the Program of the Disclosure and the Apparatus of the Disclosure.

[0068] According to the presence-providing program of this disclosure, for example, an information acquisition procedure can capture the voice of the subject being filmed in the filming area, and then the subject's voice can be further acquired. A stereophonic sound processing procedure can be performed on the subject's voice based on the display position of the subject on the display screen of the display device, and the subject's voice, on which the stereophonic sound processing has been performed, can be output to a sound output device in the display area. Therefore, according to this disclosure, for example, a presence-providing environment can be provided in terms of sound.

[0069] [Embodiment 4] An example of how the device described herein can be used will be explained below with reference to Figures 7 and 8. In the following description, the device described herein includes, for example, a device that includes a presence-providing program and a group classification program. However, this disclosure is not limited in any way to the following description.

[0070] As shown in Figure 7, this disclosure shows an example in which classroom A is set as a display area, and classrooms B and C are set as shooting areas. However, this disclosure may set at least one of classrooms A through C as a display area, and classrooms other than those set as display areas may be set as shooting areas. In this way, this disclosure may set up a relationship between display areas and shooting areas for classrooms A through C. This device is connected to shooting device A installed in classroom A, shooting device B installed in classroom B, and shooting device C installed in classroom C, respectively, via a wireless communication network. Students are seated in classrooms A, B, and C. The student in classroom A is wearing AR glasses on their head. If classrooms B and C are also set as display areas, students in classrooms B and C may also wear AR glasses on their heads. Shooting devices A through C acquire images of the seating arrangement of students in the classrooms and output the captured images to this device. This device acquires the captured images from each shooting device. This device extracts images of each seat row from the captured image as unit images. In Figure 7, the images of each seat row for classroom B are shown as rectangular frames drawn with dashed lines, and the images of each seat row for classroom C are shown as rectangular frames drawn with solid lines. That is, there are 2 seat rows in the images of each seat row for classroom B, and 4 seat rows in the images of each seat row for classroom C. This device generates an image as a layout image in which the images of each seat row for classroom B and classroom C are arranged on a predetermined side. In Figure 7, the total number of seat rows in the images of each seat row for classroom B and classroom C is calculated as 6 by adding the number of seat rows in classroom B and classroom C. This device divides the total number of seat rows, 6, by 2 and calculates 3 as the quotient of the division. As shown in Figure 8(A), the device distributes six seat row images corresponding to classrooms B and C, three to the left and three to the right, based on the seat rows composed of multiple students in classroom A.As shown in Figure 8(A), the device distributes two of the six row-of-seat images corresponding to classroom B and one of the row-of-seat images corresponding to classroom C to the left side, and distributes three of the six row-of-seat images corresponding to classroom C to the right side. After distributing the six row-of-seat images corresponding to classroom B and classroom C to the left and right sides, the device generates an image with the row-of-seat images randomly arranged on each side as a layout image. The device outputs the layout image to the AR glasses worn by the students in classroom A via a wireless communication network. Figure 8(B) shows an example of an image of a student seated at the back of the row of seats in classroom A as seen through the display screen of the AR glasses. As shown in Figure 8(B), a student seated at the back of the row in classroom A can see the students of classroom A who are actually seated in front of them through the display screen, see the students of classroom B superimposed on the display screen as augmented reality images to the left and slightly in front, and see the students of classroom C superimposed on the display screen as augmented reality images to the right and slightly in front. In this way, AR glasses superimpose placed images onto images of the real space on their display screen. Therefore, according to this disclosure, it is possible to provide an immersive environment.

[0071] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0072] <Note> Some or all of the above embodiments may be described as follows, but are not limited to the following: (Note 1) This includes information acquisition procedures, extraction procedures, placement image generation procedures, and image output procedures. The aforementioned information acquisition procedure involves acquiring images of the seating arrangement of the subject in the shooting area, The extraction procedure involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output procedure involves outputting the arranged image to a display device worn by the person to be viewed in the display area. A program that provides a sense of presence to cause a computer to perform each of the aforementioned steps. (Note 2) The extraction procedure extracts seat-specific images in which the subject being photographed is included on a seat-by-seat basis, or seat-row-specific images in which the subject being photographed is included on a seat-row basis, as the unit images. The program for providing an immersive experience as described in Appendix 1. (Note 3) The aforementioned arrangement image generation procedure is a realism-providing program as described in Appendix 2, which generates an image as the arrangement image by arranging the individual seat images in a predetermined shape when the unit images are extracted as individual seat images by the extraction procedure. (Note 4) The aforementioned arrangement image generation procedure is a realism-providing program as described in Appendix 2, wherein, when the unit image is extracted as a row-by-row seat image by the extraction procedure, an image is generated as the arrangement image by arranging the row-by-row seat images on a predetermined side. (Note 5) The image output procedure outputs the arranged image to the display device as an image relating to augmented reality or mixed reality. A program that provides an immersive experience, as described in any of the appendices 1 through 4. (Note 6) The aforementioned information acquisition procedure further acquires the speaker's image, The image output procedure involves outputting the arrangement image and the speaker image to the display device. A program that provides an immersive experience, as described in any of the appendices 1 through 5. (Note 7) Further including a 3D audio processing procedure and a sound output procedure, The information acquisition procedure further acquires the voice of the subject being filmed, which is obtained by recording the voice of the subject being filmed in the filming area. The aforementioned stereophonic sound processing procedure performs stereophonic sound processing on the voice of the subject being filmed based on the display position of the subject being filmed on the display screen of the display device, The sound output procedure involves outputting the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to the sound output device in the display area. A program that provides an immersive experience, as described in any of the appendices 1 through 6. (Note 8) It includes an information acquisition unit, an extraction unit, a layout image generation unit, and an image output unit, The aforementioned information acquisition unit acquires images capturing the seating status of the subject in the shooting area. The extraction unit extracts unit images containing the subject of photography in predetermined units from the captured images. The arrangement image generation unit generates an arrangement image in which the unit images are arranged according to a predetermined rule, The image output unit outputs the arranged image to a display device worn by the person to be viewed in the display area. A device that provides a sense of presence. (Note 9) The extraction unit extracts seat-specific images in which the subject being photographed is included on a seat-by-seat basis, or seat-row-specific images in which the subject being photographed is included on a seat-row basis, as the unit images. The immersive experience device described in Appendix 8. (Note 10) The presence-providing device according to Appendix 9, wherein the arrangement image generation unit generates an image in which the seat-specific images are arranged in a predetermined shape, when the unit image is extracted as a seat-specific image by the extraction unit, and this image is then used as the arrangement image. (Note 11) The presence-providing device as described in Appendix 9, wherein the arrangement image generation unit generates an image in which the seat row images are arranged on a predetermined side when the unit image is extracted as a seat row image by the extraction unit. (Note 12) The image output unit outputs the arranged image to the display device as an image relating to augmented reality or a mixed reality. A device that provides a sense of presence as described in any of the appendices 8 to 11. (Note 13) The aforementioned information acquisition unit further acquires the speaker's image, The image output unit outputs the arrangement image and the speaker image to the display device. A device that provides a sense of presence as described in any of the appendices 8 to 12. (Note 14) It further includes a stereophonic sound processing unit and a sound output unit, The information acquisition unit further acquires the voice of the subject being filmed, which is the voice of the subject being filmed in the filming area. The stereophonic sound processing unit performs stereophonic sound processing on the voice of the subject being filmed based on the display position of the subject on the display screen of the display device. The sound output unit outputs the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to the sound output device in the display area. A device that provides a sense of presence as described in any of the appendices 8 to 13. (Note 15) This process includes an information acquisition step, an extraction step, a layout image generation step, and an image output step. The aforementioned information acquisition step involves acquiring images of the seating arrangement of the subject in the shooting area. The extraction step involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation step generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output step outputs the arranged image to a display device worn by the person to be viewed in the display area. A method for providing a sense of presence, in which each of the aforementioned steps is performed by a computer. (Note 16) The extraction step extracts seat-specific images in which the subject being photographed is included on a seat-by-seat basis, or seat-row-specific images in which the subject being photographed is included on a seat-row basis, as the unit images. The method for providing a sense of presence as described in Appendix 15. (Note 17) The arrangement image generation step, when the unit images are extracted as seat-specific images by the extraction step, generates an image in which the seat-specific images are arranged in a predetermined shape as the arrangement image, as described in Appendix 16, the method for providing a sense of presence. (Note 18) The arrangement image generation step, when the unit image is extracted as a row-by-row seat image by the extraction step, generates an image in which the row-by-row seat images are arranged on a predetermined side as the arrangement image, as described in Appendix 16. (Note 19) The image output step outputs the arranged image to the display device as an image relating to augmented reality or mixed reality. A method of providing a sense of presence as described in any of Appendix 15 to 18. (Note 20) The aforementioned information acquisition process further acquires speaker images, The image output step involves outputting the arrangement image and the speaker image to the display device. A method of providing a sense of presence as described in any of the appendices 15 to 19. (Note 21) The process further includes a 3D audio processing step and a sound output step, The information acquisition step further acquires the voice of the subject being filmed, which is obtained by recording the voice of the subject being filmed in the filming area. The stereophonic sound processing step performs stereophonic sound processing on the voice of the subject being filmed based on the display position of the subject being filmed on the display screen of the display device. The sound output step involves outputting the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to the sound output device in the display area. A method of providing a sense of presence as described in any of the appendices 15 to 20. (Note 22) This includes information acquisition procedures, extraction procedures, placement image generation procedures, and image output procedures. The aforementioned information acquisition procedure involves acquiring images of the seating arrangement of the subject in the shooting area, The extraction procedure involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output procedure involves outputting the arranged image to a display device worn by the person to be viewed in the display area. A computer-readable recording medium containing a program that causes a computer to perform each of the aforementioned steps. (Note 23) The extraction procedure extracts seat-specific images in which the subject being photographed is included on a seat-by-seat basis, or seat-row-specific images in which the subject being photographed is included on a seat-row basis, as the unit images. Recording medium as described in Appendix 22. (Note 24) The recording medium described in Appendix 23, wherein, when the unit image is extracted as the seat-specific image by the extraction procedure, the recording medium generates an image in which the seat-specific images are arranged in a predetermined shape as the arrangement image. (Note 25) The recording medium described in Appendix 23, wherein, when the unit image is extracted as a row-by-row seat image by the extraction procedure, the recording medium generates an image in which the row-by-row seat images are arranged on a predetermined side as the arrangement image. (Note 26) The image output procedure outputs the arranged image to the display device as an image relating to augmented reality or mixed reality. A recording medium as described in any of the appendices 22 to 25. (Note 27) The aforementioned information acquisition procedure further acquires the speaker's image, The image output procedure involves outputting the arrangement image and the speaker image to the display device. A recording medium as described in any of the appendices 22 to 26. (Note 28) Further including a 3D audio processing procedure and a sound output procedure, The information acquisition procedure further acquires the voice of the subject being filmed, which is obtained by recording the voice of the subject being filmed in the filming area. The aforementioned stereophonic sound processing procedure performs stereophonic sound processing on the voice of the subject being filmed based on the display position of the subject being filmed on the display screen of the display device, The sound output procedure involves outputting the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to the sound output device in the display area. A recording medium as described in any of appendices 22 to 27. [Industrial applicability]

[0073] This disclosure enables the provision of an immersive environment. Therefore, this disclosure can be widely and usefully utilized in various fields, such as education. [Explanation of Symbols]

[0074] 10, 10A presence providing device 11 Information acquisition department 12 Extraction part 13. Arrangement Image Generation Unit 14 Image output section 15. Spatial Sound Processing Unit 16. Sound output section 101 Central Processing Unit 102 memory 103 Bus 104 Storage device 105 Input device 106 Output device 107 Communication devices

Claims

1. This includes information acquisition procedures, extraction procedures, placement image generation procedures, and image output procedures. The aforementioned information acquisition procedure involves acquiring images of the seating arrangement of the subject in the shooting area, The extraction procedure involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output procedure involves outputting the arranged image to a display device worn by the person to be viewed in the display area. A program that provides a sense of presence to cause a computer to perform each of the aforementioned steps.

2. The extraction procedure extracts seat-specific images in which the subject being photographed is included on a seat-by-seat basis, or seat-row-specific images in which the subject being photographed is included on a seat-row basis, as the unit images. The program for providing a sense of presence according to claim 1.

3. The program for providing a sense of presence according to claim 2, wherein the arrangement image generation procedure generates an image in which the seat-specific images are arranged in a predetermined shape, when the unit image is extracted as a seat-specific image by the extraction procedure.

4. The arrangement image generation procedure is a program that provides a sense of presence according to claim 2, wherein, when the unit image is extracted as a row-by-row seat image by the extraction procedure, an image is generated in which the row-by-row seat images are arranged on a predetermined side, and this image is then generated as the arrangement image.

5. The image output procedure outputs the arranged image to the display device as an image relating to augmented reality or mixed reality. A program for providing a sense of presence according to any one of claims 1 to 4.

6. The aforementioned information acquisition procedure further acquires the speaker's image, The image output procedure involves outputting the arrangement image and the speaker image to the display device. A program for providing a sense of presence according to any one of claims 1 to 4.

7. Further including a 3D audio processing procedure and a sound output procedure, The information acquisition procedure further acquires the voice of the subject being filmed, which is obtained by recording the voice of the subject being filmed in the filming area. The aforementioned stereophonic sound processing procedure performs stereophonic sound processing on the voice of the subject being filmed based on the display position of the subject being filmed on the display screen of the display device, The sound output procedure involves outputting the voice of the subject being filmed, on which the stereophonic sound processing has been performed, to the sound output device in the display area. A program for providing a sense of presence according to any one of claims 1 to 4.

8. It includes an information acquisition unit, an extraction unit, a layout image generation unit, and an image output unit, The aforementioned information acquisition unit acquires images capturing the seating status of the subject in the shooting area. The extraction unit extracts unit images containing the subject of photography in predetermined units from the captured images. The arrangement image generation unit generates an arrangement image in which the unit images are arranged according to a predetermined rule, The image output unit outputs the arranged image to a display device worn by the person to be viewed in the display area. A device that provides a sense of presence.

9. This process includes an information acquisition step, an extraction step, a layout image generation step, and an image output step. The aforementioned information acquisition step involves acquiring images of the seating arrangement of the subject in the shooting area. The extraction step involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation step generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output step outputs the arranged image to a display device worn by the person to be viewed in the display area. A method for providing a sense of presence, in which each of the aforementioned steps is performed by a computer.

10. This includes information acquisition procedures, extraction procedures, placement image generation procedures, and image output procedures. The aforementioned information acquisition procedure involves acquiring images of the seating arrangement of the subject in the shooting area, The extraction procedure involves extracting unit images from the captured images in which the subject being photographed is included in a predetermined unit, The aforementioned arrangement image generation procedure generates an arrangement image in which the unit images are arranged according to predetermined rules, The image output procedure involves outputting the arranged image to a display device worn by the person to be viewed in the display area. A computer-readable recording medium containing a program that causes a computer to perform each of the aforementioned steps.

Images