Output control device, output control system, and control method

Abstract

The present application provides output control device, output control system, and control method. The image and the sound with the sense of presence are provided to the passenger of the moving body. The output control device (1A, 1B, and 1C) has: position direction determination part (11), which determines the real position and the real orientation; reading part (12), which reads virtual space data (31); virtual position decision part (131), which decides the virtual position and the virtual orientation according to the real position and the real orientation; display control part (14), which makes the display device (4) display the image of the partial area or the partial object of the virtual space based on the virtual position and the virtual orientation and the setting position of the display device (4); sound output control part (15), which controls the position of the sound image formed by the output sound from the loudspeaker (5) based on the virtual orientation and the relative position relationship between the virtual position and the sound source body.

Description

Technical Field

[0001] This invention relates to an output control device, an output control system, and a control method. Background Technology

[0002] There is an existing technique that controls the position of the sound field formed by the sound output from a loudspeaker, i.e., controls the position of the "audio image".

[0003] (Existing technical literature)

[0004] Patent Document 1: Japanese Patent Application Publication No. 2015-16787 Summary of the Invention

[0005] (The problem the invention aims to solve)

[0006] In recent years, technologies such as AR (Augmented Reality) or MR (Mixed Reality) have been seeking a technology that can provide users with a more immersive visual and auditory experience.

[0007] One aspect of the present invention is to provide passengers of a mobile vehicle with immersive visual and auditory experiences.

[0008] (Technical means used to solve the problem)

[0009] To address the aforementioned issues, one aspect of the present invention provides an output control device that controls a display device and a speaker disposed within the internal space of a moving body. The output control device includes: a position and orientation determination unit that determines the actual position of the moving body in real space and the orientation of the moving body in real space; a reading unit that reads virtual space data from a storage device, wherein the virtual space data defines a virtual space formed by placing a 3D sound source body as a virtual sound source on a 3D mapping; and a virtual position determination unit that determines the virtual position of the moving body based on the actual position of the moving body and the orientation of the moving body in real space. The virtual position in the virtual space and the orientation of the moving body in the virtual space; a display control unit, which, based on the virtual position of the moving body and the orientation of the moving body in the virtual space, and the setting position of the display device, causes the display device to display an image of a portion of the virtual space or an image of a portion of an object; and an audio output control unit, which causes the speaker to output sound, and the audio output control unit controls the position of the sound image formed by the sound output from the speaker based on the orientation of the moving body in the virtual space, and the relative positional relationship between the virtual position of the moving body and the sound source body.

[0010] To address the aforementioned issues, one aspect of the present invention provides a control method for controlling a display device and a speaker disposed within the internal space of a moving body. The method includes: a position and orientation determination step, which determines the actual position of the moving body in real space and its orientation in real space; a reading step, which reads virtual space data from a storage device, wherein the virtual space data defines a virtual space formed by placing a 3D sound source body as a virtual sound source on a 3D mapping; and a virtual position determination step, which determines the virtual position of the moving body based on its actual position and its orientation in real space. The virtual position in space and the orientation of the moving body in the virtual space; a display control step, which, based on the virtual position of the moving body and the orientation of the moving body in the virtual space, and the setting position of the display device, causes the display device to display an image of a portion of the virtual space or an image of a portion of an object; and an audio output control step, which causes the speaker to output sound, wherein, based on the orientation of the moving body in the virtual space and the relative positional relationship between the virtual position of the moving body and the sound source, the position of the sound image formed by the sound output from the speaker is controlled.

[0011] Through the aforementioned configuration and processing, the position and orientation of the moving object in the virtual space can be determined based on its position and orientation in the real space. Furthermore, the display device displays an image corresponding to the position and orientation of the moving object in the virtual space. Additionally, when a sound source object positioned in the virtual space is used as a virtual sound source, a sound image corresponding to the sound source object can be formed at a position corresponding to the positional relationship between the moving object and the sound source object in the virtual space and their orientation.

[0012] Therefore, it is possible to give riders of mobile vehicles existing in real space the following experience: the originally virtual sound source, or sound source entity, seems to be a real sound source emitting sound. That is, it is possible to provide riders of mobile vehicles with immersive images and sounds.

[0013] In the output control device, the position and direction determination unit can periodically update the real position of the moving body and the orientation of the moving body in the real space; the virtual position determination unit can update the virtual position and the orientation of the moving body in the virtual space based on the updated real position and the updated orientation of the moving body in the real space; the display control unit can display the updated image of the partial area or the updated image of the partial object based on the updated virtual position and the updated orientation of the moving body in the virtual space; and the audio output control unit can update the position of the sound image based on the updated virtual position and the updated orientation of the moving body in the virtual space.

[0014] According to the aforementioned scheme, when the actual position and orientation of the moving object change, the output control device can determine the changed actual position and the changed orientation in the real space. The virtual position and orientation of the moving object in the virtual space are determined based on its actual position and orientation in the real space. Furthermore, the positions of the image or object displayed by the display device, and the sound image formed by the sound output from the speaker, are determined based on the virtual position and orientation of the moving object in the virtual space.

[0015] Therefore, according to the above scheme, when the actual position and actual orientation of the moving object change, it is possible to realize image display and sound output that reflect the change.

[0016] In the output control device, the audio output control unit can control the speaker in such a way that the farther the distance between the virtual position in the virtual space and the audio source is, the smaller the output volume of the audio corresponding to the audio source.

[0017] According to the proposed solution, the positional relationship between the moving object and the sound source can be represented by the volume level. Therefore, it allows passengers riding the moving object to experience a more immersive audio experience.

[0018] In the output control device, the virtual position determination unit can determine the virtual position in such a way that the positional relationship between any reference point in the virtual space and the virtual position is approximately the same as the positional relationship between any reference point in the real space and the real position. Alternatively, it can determine the image of the partial area or the image of the partial object that the display control unit wants the display device to display in such a way that the virtual space and the real space are displayed at the same scale.

[0019] According to the proposed scheme, passengers of the mobile vehicle can be provided with a visual effect that makes the real space appear to merge with the virtual space.

[0020] To address the aforementioned issues, one aspect of the control system of the present invention includes: the output control device; a positioning device that measures data of the actual position of the moving body; a storage device; a display device; and a speaker. The position orientation determination unit of the output control device determines the actual position of the moving body and the orientation of the moving body in real space based on the actual position data acquired by the positioning device. This solution achieves the same effect as the output control device.

[0021] (The effect of the invention)

[0022] According to one aspect of the invention, immersive visuals and audio can be provided to passengers of a mobile vehicle. Attached Figure Description

[0023] Figure 1 An example of the operation of the output control system of Embodiment 1 of the present invention is shown.

[0024] Figure 2 This is a block diagram of the main structure of the various devices that constitute the output control system.

[0025] Figure 3 A specific example of the virtual space of Embodiment 1 of the present invention is shown.

[0026] Figure 4 An example of the arrangement of the display device and speaker in a mobile body according to Embodiment 1 of the present invention is shown.

[0027] Figure 5 This is a flowchart of the processing flow in the output control device of Embodiment 1 of the present invention.

[0028] Figure 6 It shows the changes in the relative positional relationship between the moving object and the sound source in the virtual space, as well as the changes in the position of the sound image in the real space.

[0029] Figure 7 This is a main structural block diagram of various devices constituting the output control system of Embodiment 2 of the present invention.

[0030] Figure 8 An example of the vibrator configuration in the output control system of Embodiment 2 of the present invention is shown.

[0031] Figure 9 This is a main structural block diagram of the various devices constituting the output control system of Embodiment 3 of the present invention.

[0032] <Explanation of Figure Markers>

[0033] 100, 200, 300 Output Control System

[0034] Output control devices 1A, 1B, and 1C

[0035] 2. Positioning device

[0036] 3. Storage device

[0037] 4 Display devices

[0038] 5 speakers

[0039] 6. Vibrator

[0040] 7 cameras

[0041] 8 microphones

[0042] 9. Moving bodies

[0043] 11. Position and Direction Determination Unit

[0044] 12 Reading Department

[0045] 13 Space Processing Department

[0046] 131 Virtual Location Decision Department

[0047] 132 Sound Source Decision Department

[0048] 133, 134 Display Image Production Department

[0049] 14 Display Control Unit

[0050] 15 Audio Output Control Unit

[0051] 151 Playback Department

[0052] 152 Audio-visual Control Department

[0053] 153 Allocation Decision Department

[0054] 16 Vibration Control Department

[0055] 31 Virtual Space Data

[0056] 32. Audio Source Setting Information

[0057] 33 Audio Data

[0058] 34 Map Data Detailed Implementation

[0059] [Implementation Method 1]

[0060] The output control system of this embodiment controls the output of images and sounds within the internal space of the mobile body. The output control system provides the rider of the mobile body with images and sounds of a virtual space linked to the real world. In other words, the output control system 100 implements MR (Mixed Reality) within the internal space of the mobile body.

[0061] The following is for reference Figures 1-6 This specification provides a detailed description of the structure and operation of the various devices constituting the output control system of this embodiment. In this specification, "image" includes both still images and moving images. Furthermore, "sound" in this specification is not limited to human voice, but includes all sounds such as sound effects, ambient sounds, and background music (BGM).

[0062] Examples of System Actions

[0063] First, a brief overview of the operation of the output control system 100 will be provided. Figure 1 This is an example of the operation of the output control system 100 in this embodiment. Figure 1 This is a schematic diagram showing the interior space of the mobile body 9, where the output control system 100 is installed, viewed from behind the passenger P1 who is stationary inside the mobile body 9. (See diagram below.) Figure 1 As shown, the interior space of the mobile body 9 contains a seat S1, a control panel S2, a display device 4, and a speaker 5. However, for the output control system 100, the seat S1 and control panel S2 are not essential structures. Furthermore, the number of display devices 4 and speakers 5 is not limited. Figure 1 The example shown.

[0064] (The interior space of moving body 9)

[0065] by Figure 1 Looking at the interior space of the movable body 9, the display device 4 is installed on at least two walls, excluding the front of the operator's seat and the back of the seat S1. Additionally, Figure 1 In this case, the front of the control seat is not the display device 4 but a window. Figure 1 In this example, the wall structure behind seat S1 is not limited. A passenger P1, residing inside the mobile body 9, sits in a seat S1. Hereinafter, the "passenger residing inside the mobile body 9" will be simply referred to as the "passenger". The output control system 100 causes each display device 4 to display images depicting the scenery of the virtual space. Additionally, the output control system 100 causes each speaker 5 to output sound for creating the virtual space.

[0066] (Virtual space)

[0067] In this specification, "virtual space" refers to a 3D spatial model composed of a 3D mapping within a defined range and various 3D objects placed on that 3D mapping. The output control system 100 determines the position and orientation of the moving body 9 in the virtual space based on its position and orientation in the real space; details of this process will be described later. For ease of explanation, the position in the real space will be referred to as the "real position." The position in the virtual space will be referred to as the "virtual position." The orientation in the real space will be referred to as the "real orientation," and the orientation in the virtual space will be referred to as the "virtual orientation." The output control system 100 controls the image display on the display device 4 and the sound output on the speaker 5 based on the determined virtual position and virtual orientation of the moving body 9.

[0068] This specification does not adhere to the distinction between real and virtual space. For example, virtual space can be a space model that faithfully replicates the real space in which the mobile vehicle 9 actually travels, or it can be a space model that simulates spaces in which the mobile vehicle 9 cannot travel in reality, such as the seabed or the air. In addition, maps and objects that simulate objects or phenomena that exist in real space, as well as maps and objects that simulate objects or phenomena that do not exist in real space, can be mixed and integrated into virtual space.

[0069] (Display control and audio output control)

[0070] The output control system 100 generates an image depicting a virtual landscape, which is envisioned as being viewable from the virtual position and orientation of the moving body 9. The output control system 100 then displays the generated image on the display device 4. For example, the area of ​​the virtual space to be depicted by the output control system 100 can be determined based on the virtual position and orientation of the moving body 9, the position and orientation of the display device 4 in real space, and the size and shape of the display device 4.

[0071] In this embodiment, unless otherwise specified, the 3D mapping of the virtual space and the 3D objects in the virtual space are both mappings and objects that can be depicted as scenery (i.e., non-transparent mappings and objects). However, at least a portion of the 3D mapping of the virtual space may also be a transparent mapping that is not displayed on the display device 4. In addition, at least a portion of the 3D objects in the virtual space may also be transparent objects that are not displayed on the display device 4.

[0072] Additionally, the output control system 100 causes the speaker 5 to output sound used to create a virtual space. The output control system 100 determines a hypothetical sound that the passenger can hear under the assumption that the passenger is in a virtual position and virtual orientation of the moving body 9, and causes the speaker 5 to output the sound in a manner that reproduces the auditory experience corresponding to the virtual position and virtual orientation.

[0073] More specifically, the output control system 100 controls the position of the sound image formed by the output sound by controlling the loudspeaker 5. For example, the output control system 100 controls the loudspeaker 5 so that the following two orientations are consistent: the orientation of the sound image formation position in the internal space of the moving body 9; and the orientation relative to the sound source body from the virtual position and virtual orientation of the moving body 9. Figure 1 In the example, in order to represent the sound emitted by the mammoth (i.e., the object in the virtual space) displayed on the display device 4, the output control system 100 controls the speaker 5 to form a sound image near the display position of the mammoth.

[0074] (The movement of the moving object 9 and the changes in its image and sound)

[0075] The output control system 100 continuously determines the actual position and orientation of the moving body 9, and updates the virtual position and orientation of the moving body 9 accordingly. Furthermore, based on the updated virtual position and orientation, the output control system 100 continuously updates the displayed image on the display device 4, the output sound of the speaker 5, and the position of the sound image. Thus, the output control system 100 can change the displayed content of each display device 4 and the sound output of each speaker 5 in real time according to the movement of the moving body 9.

[0076] For example, when the mobile vehicle 9 is in motion, the output control system 100 can display, for instance, the scenery of a virtual space that changes as the mobile vehicle 9 moves. Additionally, the output control system 100 can represent, for example, changes in the position of the sound image: the change in the relative position between the mobile vehicle 9 and an object assumed to be a sound source, and the change in their orientation. Thus, the output control system 100 allows the passenger P1 to experience images and stereo sound as if the mobile vehicle 9 is moving in a virtual space.

[0077] Structure of the main parts

[0078] Next, the structure of the various devices constituting the output control system 100 will be described. Figure 2 This is a main structural block diagram of the various devices constituting the output control system 100 of this embodiment. The output control system 100 includes an output control device 1A, a positioning device 2, a storage device 3, a display device 4, and a speaker 5. At least the output control device 1A, the display device 4, and the speaker 5 are disposed in the internal space of the movable body 9. In addition, in the example of this embodiment, the positioning device 2 and the storage device 3 are also disposed in the internal space of the movable body 9. Furthermore, the storage device 3 may also be included in the output control device 1A.

[0079] (Positioning device 2)

[0080] The positioning device 2 collects information related to the actual position of the moving body 9. The positioning device 2 outputs the measured information (also called positioning data) to the output control device 1A. The specific construction of the positioning device 2 is not particularly limited. For example, the positioning device 2 can be a LiDAR (Light Detection and Ranging) system used to detect structures around the moving body 9. In this case, the positioning data represents the size and shape of the structures around the moving body 9. Alternatively, the positioning device 2 can be a GPS receiver. In this case, the positioning data is information representing latitude and longitude received from GPS satellites.

[0081] (Storage device 3)

[0082] Storage device 3 is a device for storing various information required for the operation of output control device 1A. Storage device 3 contains at least virtual space data 31, sound source setting information 32, and sound data 33. In addition, storage device 3 may contain map data 34.

[0083] Virtual space data 31 is a collection of various data used to construct a virtual space. Virtual space data 31 includes, for example, mapping data of the virtual space and data of objects placed within the virtual space. Virtual space data 31 is pre-created and stored in storage device 3. In this embodiment, storage device 3 stores one type of virtual space data 31. However, storage device 3 may also store multiple types of virtual space data 31.

[0084] (Specific examples of virtual space)

[0085] Figure 3 A specific example of virtual space is shown. Figure 3 Although the virtual space is illustrated schematically using a 2D diagram, the actual virtual space is a 3D model. Furthermore, Figure 3 The location and orientation of the moving object 9 in the virtual space are also shown.

[0086] In a virtual space, there are one or more objects designated as virtual sound sources, or sound source entities. A "virtual sound source" refers to a sound source in a virtual space. Here, "sound source" does not mean that it actually emits sound, but rather that it is virtually constructed to emit sound in that space.

[0087] Figure 3 In the example, the mammoth (object B1), the building (object B2), and the dwelling (object B3) are all sound sources. Therefore, a sound source can be a stationary object or an object with regular or irregular movement. Besides... Figure 3Beyond these examples, if the virtual space is a space that creates roads and cities, then the sound source can be an object whose position does not change, such as a railway crossing, or an object whose position changes, such as a tram, a car, or a person.

[0088] Furthermore, the sound source can be a depictable object, i.e., an object that can be displayed on the display device 4, or it can be a transparent object that cannot be depicted. For example... Figure 3 The mammoth shown as object B1 can be an object that can be displayed on display device 4 and is also a sound source. Additionally, for example, Figure 3 The mammoth shown as object B1 can also be an object that can be displayed on display device 4, but it is not a sound source. Furthermore, the sound source of sounds related to object B1, such as the sound of the mammoth or footsteps, can also be set as a transparent object that is different from object B1.

[0089] The sound source setting information 32 specifies which objects within the virtual space data 31 are sound source entities. The sound source setting information 32 shows the allocation scheme for the audio data 33 assigned to the sound source entities. More specifically, the sound source setting information 32 is information that stores the identification information of the objects serving as sound source entities in correspondence with the identification information of the audio data 33. The object identification information, for example, refers to the ID assigned to each object in the virtual space data 31. The audio data 33 identification information, for example, refers to the filename of the audio data 33.

[0090] The audio source setting information 32 may include information indicating the volume of the audio data 33. For example, the audio source setting information 32 may be information that stores the identification information of the audio data 33 and the information indicating the set volume of the audio data 33 in correspondence with the identification information of the audio source. The set volume refers to the playback volume of the audio data 33 when the distance from the virtual position of the moving body 9 to the audio source is zero. The set volume may be set to a value from 1 to 100, for example. In this case, 1 may be the minimum volume and 100 may be the maximum volume.

[0091] Audio data 33 is the audio data to be output from speaker 5. After being read by output control device 1A, audio data 33 undergoes playback processing and effects rendering in audio output control unit 15 (described later) before being output from speaker 5. If the audio source setting information 32 includes the set volume of audio data 33, the audio output control unit 15 can attenuate the volume of audio data 33 based on the distance from the virtual position of the moving body 9 to the audio source body in virtual space.

[0092] Map data 34 includes information such as terrain, road shapes, and landmark structures in the area where the mobile body 9 can travel. There are no particular limitations on the method for acquiring map data 34 in the output control system 100. For example, if the mobile body 9 is equipped with a car navigation system, the storage device 3 can be pre-connected to the car navigation system to download map data 34.

[0093] Map data 34 can be used when determining the actual location of the moving body 9 based on the positioning data. However, if the actual location of the moving body 9 can be determined directly from the positioning data, then the storage device 3 may not need to store map data 34. For example, if the positioning device 2 is a GPS receiver, then the positioning data represents latitude and longitude, so map data 34 is not necessary information.

[0094] (Output control device 1A)

[0095] Output control device 1A is a device that uniformly controls display device 4 and speaker 5. Output control device 1A generates an image based on the actual position and orientation of the moving body 9 and provides it to display device 4. Furthermore, output control device 1A determines the sound to be output from speaker 5 based on the actual position and orientation of the moving body 9, and determines the manner in which the sound is output from speaker 5. Here, "sound output manner" refers, for example, to the position of the sound image formed by the sound. Output control device 1A provides the sound to be output to speaker 5 and controls the sound output of speaker 5 in a manner that allows the sound image to be formed at the position determined by output control device 1A.

[0096] (Detailed configuration of output control device 1A)

[0097] More specifically, the output control device 1A includes a position and orientation determination unit 11, a reading unit 12, a spatial processing unit 13, a display control unit 14, and an audio output control unit 15.

[0098] The position and orientation determination unit 11 determines the actual position and actual orientation based on the positioning data acquired from the positioning device 2. The position and orientation determination unit 11 provides the determined actual position information and the determined actual orientation information to the space processing unit 13.

[0099] The position and direction determination unit 11 determines the actual position as follows: If the positioning device 2 is a LiDAR, the position and direction determination unit 11 compares the positioning data from the LiDAR with the map data 34 stored in the storage device 3 to determine the actual position of the moving body 9. On the other hand, if the positioning device 2 is a GPS receiver and the positioning data already shows the actual position of the moving body 9 (e.g., latitude and longitude), the position and direction determination unit 11 can directly set the obtained positioning data as information representing the actual position.

[0100] The position and orientation determination unit 11 determines the actual orientation as follows: If the positioning device 2 is a LiDAR, the positioning data used to determine the actual position is compared with the map data 34 to simultaneously determine the actual orientation. If the positioning device 2 is a device other than a LiDAR, the position and orientation determination unit 11 first determines the actual position multiple times based on the positioning data, compares these actual positions, and thus determines the direction of movement of the actual position. The position and orientation determination unit 11 determines the direction of movement of the actual position of the moving body 9 as the direction of travel of the moving body 9, that is, the actual orientation of the moving body 9. If the actual position determined multiple times does not change, that is, if the moving body 9 does not move, the position and orientation determination unit 11 can directly use the previously determined actual orientation.

[0101] The reading unit 12 reads virtual space data 31 from the storage device 3 and provides it to the space processing unit 13. There is no particular limitation on the timing of the reading unit 12 reading the virtual space data 31 and providing it to the space processing unit 13. For example, the reading unit 12 may read the virtual space data 31 and provide it to the space processing unit only when it receives a request from the space processing unit, or it may read the virtual space data 31 and provide it to the space processing unit 13 every time the virtual space data 31 is updated. Furthermore, if the storage device 3 stores multiple types of virtual space data 31, the reading unit 12 can decide which type of virtual space data 31 to read.

[0102] The spatial processing unit 13 uses virtual spatial data 31 to perform various processes. More specifically, the spatial processing unit 13 includes a virtual position determination unit 131, a sound source determination unit 132, and a display image production unit 133.

[0103] The virtual position determination unit 131 determines the virtual position and virtual orientation of the moving body 9 based on the actual position and orientation provided by the position and orientation determination unit 11. The virtual position determination unit 131 provides information indicating the virtual position and information indicating the virtual orientation to the audio output control unit 15, the audio source determination unit 132, and the display image production unit 133.

[0104] The specific method by which the virtual position determination unit 131 determines the virtual position and virtual orientation can be appropriately set according to the type and characteristics of the virtual space data 31. For example, the virtual position determination unit 131 preferably predefines any point in the real space and any point in the virtual space as reference points, and determines the virtual position in such a way that the positional relationship between the reference points in the virtual space and the virtual position approximates the positional relationship between the reference points in the real space and the real position. The method of defining the aforementioned reference points is not particularly limited. For example, points represented by specific latitude and longitude on a map can be used as reference points, or the real position and virtual position initially determined by the position orientation determination unit 11 and the virtual position determination unit 131 can be used as subsequent reference points.

[0105] In the image production unit 133 described later, if the virtual space and the real space need to be depicted at the same scale, the virtual position determination unit 131 can determine the virtual position of the moving body 9 in a manner that the virtual position of the moving body 9 is substantially consistent with the real position of the moving body 9. Furthermore, the virtual position determination unit 131 can determine the virtual orientation of the moving body 9 in a manner that the virtual orientation of the moving body 9 is substantially consistent with the real orientation of the moving body 9. Thus, the position of the moving body 9 in the real space can be linked with the position of the moving body 9 in the virtual space.

[0106] The sound source determination unit 132 selects a sound source that is assumed to be audible to the passenger when the passenger is in a virtual position (and virtual orientation) of the moving body 9. In other words, the sound source determination unit 132 determines the sound source corresponding to the audio data 33 to which audio output needs to be applied. For example, the sound source determination unit 132 selects a sound source located within a predetermined range from the virtual position of the moving body 9 in the virtual space. The sound source determination unit 132 determines the identification information of the selected sound source based on the virtual space data 31. The sound source determination unit 132 provides the determined identification information of the sound source to the audio output control unit 15.

[0107] The sound source determination unit 132 can select a sound source based on the set volume of the audio data 33 assigned to the sound source and the distance from the virtual position of the moving body 9 to the sound source. For example, the sound source determination unit 132 can perform the following steps 1 to 4 to select the sound source.

[0108] Step 1: Calculate the distance from the virtual position of the moving body 9 to each sound source body.

[0109] Step 2: Based on the sound source setting information 32, determine the set volume of each sound data 33 corresponding to each sound source.

[0110] Step 3: Based on the distance calculated in Step 1 and the set volume, determine the playback volume of each sound source. For example, in the sound source determination unit 132, the greater the distance calculated in Step 1, the more the set volume determined in Step 2 can be attenuated. If the distance is zero, the set volume can be directly used as the playback volume.

[0111] Step 4: Select the audio source with a playback volume above the threshold as "the audio source corresponding to the audio data 33 that needs to be output".

[0112] The distance from the virtual position of the moving body 9 to each sound source can be an absolute distance or a relative distance. Furthermore, steps 1 and 2 can be executed out of order. Additionally, the threshold in step 4 can be set appropriately. Furthermore, the sound source determination unit 132 can transmit the playback volume determined in step 3 to the audio output control unit 15.

[0113] Through the above steps, for example from a virtual space perspective, even if a sound source is far away from the moving body 9, as long as the playback volume of the sound data 33 assigned to that sound source is above the threshold, the sound data 33 can be used as an output object. Therefore, for example, sounds such as fireworks and sirens coming from a distance can be reproduced more accurately.

[0114] The display image creation unit 133 creates an image to be displayed on the display device 4 based on the virtual position and virtual orientation of the moving body 9. The display image creation unit 133 provides the created image to the display control unit 14. The display image creation unit 133 can instruct the display control unit 14 on the timing of displaying the provided image. Furthermore, if there are multiple display devices 4, the display image creation unit 133 can instruct the display control unit 14 on which display device 4 the image should be displayed when providing the image.

[0115] For example, the display image production unit 133 sets up a virtual camera in the virtual space, each having a field of view corresponding to each display device 4, and depicts the scene captured by the virtual camera as an image displayed on the display device 4 corresponding to the virtual camera. The position, orientation, and shape and area of ​​the field of view of the virtual camera are determined based on the following (1) to (3).

[0116] (1) The virtual position and virtual orientation of the moving object 9;

[0117] (2) The position and orientation of each display device 4 in real space;

[0118] (3) The size and shape of each display device 4.

[0119] The output control system 100 may include an in-vehicle camera and / or in-vehicle sensors that can detect the position of a passenger's head through imaging or measurement. Additionally, the output control device 1A can communicate with the in-vehicle camera and / or in-vehicle sensors. The spatial processing unit 13 of the output control device 1A can determine the passenger's head position based on images captured by the in-vehicle camera and / or measurement results from the in-vehicle sensors.

[0120] If the space processing unit 13 needs to determine the position of the passenger's head, the image display unit 133 can also determine the area to be cropped from the captured image based on the position of the passenger's head, in addition to the above (1) to (3). Thus, the display device 4 can display an image that matches the height of the passenger.

[0121] Specifically, exemplify Figure 1 Let's illustrate the image display with an example. Set the direction of movement of the moving body 9 as "forward," and the direction opposite to the direction of movement as "backward," so that in... Figure 1 In the actual space shown, the control seat S2 of the mobile body 9 is positioned in front. Furthermore, display devices 4 are installed substantially vertically on the right and rear walls of the internal space of the mobile body 9. In this case, the display image production unit 133 creates an image depicting the scenery to the right of the mobile body 9 as viewed from virtual space for the display device 4 installed on the right wall of the internal space. Additionally, the display image production unit 133 creates an image depicting the scenery behind the mobile body 9 as viewed from virtual space for the display device 4 installed on the rear wall of the internal space.

[0122] Therefore, as Figure 1 As shown, the two display devices 4 display the scenery extending from the right side to the rear of the mobile body 9 as viewed in virtual space. That is, from the perspective of the passenger P1, the real scenery visible from the window of the control seat can be seen, and the scenery of the virtual space can also be seen from each display device 4. The "scenery of the virtual space" mentioned here includes not only the map, but also objects. For example, if the 3D map is a transparent map that is partially or completely not depicted, the display image production unit 133 can create an image depicting a portion of the objects in the virtual space and output it to the display control unit 14.

[0123] The display image production unit 133 can determine the magnification of the virtual camera by displaying the virtual space and the real space at the same scale. In other words, the display image production unit 133 can depict the scenery of the virtual space on the display device 4 by enlarging or shrinking it in a way that displays the virtual space and the real space at the same scale. As a result, a visual effect that blends the real space and the virtual space can be provided to the rider of the mobile vehicle. In addition, since the scenery in the real space and the scenery in the virtual space move at the same speed, the rider can be prevented from experiencing so-called "3D motion sickness" even when the mobile vehicle 9 is in motion.

[0124] The display control unit 14 causes the display device 4 to display the image provided by the display image production unit 133 of the space processing unit 13. If the display image production unit 133 instructs each display device 4 on the timing of image display, the display control unit 14, in accordance with the instruction, causes each display device 4 to display the image at the specified timing. Furthermore, if there are multiple display devices 4, the display control unit 14, in accordance with the instruction of the space processing unit 13, controls the corresponding display device 4 to display the corresponding image.

[0125] The audio output control unit 15 performs various processes related to audio output and sound image formation. More specifically, the audio output control unit 15 includes a playback unit 151, a sound image control unit 152, and a distribution decision unit 153.

[0126] The playback unit 151 plays the audio data 33 assigned to the audio source, which is determined by the audio source determination unit 132. More specifically, firstly, the playback unit 151 uses the identification information of the audio source determined by the audio source determination unit 132 as the search key to retrieve the audio source setting information 32 in the storage device 3, thereby determining the audio data 33 assigned to that audio source. Next, the playback unit 151 reads from the storage device 3 and plays the determined audio data 33. While playing the audio, the playback unit 151 sequentially provides the audio to the sound image control unit 152. The playback unit 151 may also sequentially provide the played audio to the allocation determination unit 153.

[0127] If the sound source determination unit 132 determines multiple sound sources, that is, if there are multiple audio data 33 to be played, the playback unit 151 plays each audio data 33 individually while providing each audio data 33 individually to the sound image control unit 152 (and the allocation determination unit 153). In addition, if the audio output control unit 15 receives information indicating the playback volume of the audio data 33 from the sound source determination unit 132, the playback unit 151 can play each audio data 33 at the determined playback volume.

[0128] The image control unit 152 is used to determine the image formation position of the audio to be output. If multiple audio signals are provided simultaneously from the playback unit 151, the image control unit 152 can individually determine the image formation position of each audio signal. The image control unit 152 can determine the image formation position in real time. That is, the image control unit 152 can control the image formation position of the audio data 33 to be output in real time.

[0129] Specifically, the audio-visual control unit 152 determines the position of the audio-visual image based at least on the following (1) and (2).

[0130] (1) The virtual orientation of the moving body 9 is determined by the virtual position determination unit 131;

[0131] (2) The relative positional relationship between the virtual position of the moving body 9 and the sound source body.

[0132] The sound image control unit 152 can also correct the position of the sound image based on the working allocation scheme for the loudspeaker 5 determined by the allocation decision unit 153 described later.

[0133] The allocation decision unit 153 determines in real time which speaker 5 will output the sound sequentially provided by the playback unit 151 (i.e., which speaker 5 will be allocated the played sound). If only one speaker 5 is installed in the mobile body 9, the sound output control unit 15 may not include the allocation decision unit 153. Alternatively, if multiple sounds are provided simultaneously by the playback unit 151, the allocation decision unit 153 can individually determine how to allocate each sound to the speaker 5. Furthermore, the allocation decision unit 153 can feed back the determined allocation scheme for the speaker 5 to the sound image control unit 152.

[0134] Therefore, when the playback of the audio data 33 of the output target, the position of the sound image formation, and the working allocation scheme for each speaker 5 are determined in the audio output control unit 15, the audio output control unit 15 instructs each speaker 5 to output audio according to the aforementioned allocation scheme. At this time, the audio output control unit 15 can also simultaneously instruct the actual output volume and output mode of each speaker 5. Thus, the audio output control unit 15 can use the speakers 5 to form a sound image at the position determined by the sound image control unit 152.

[0135] (Display device 4 and speaker 5)

[0136] Display device 4 is a display device that displays images under the control of output control device 1A. The type of display device 4 is not particularly limited. For example, display device 4 can be a transparent display device. Speaker 5 is an audio output device that outputs sound under the control of output control device 1A. The type of speaker 5 is not particularly limited; speaker 5 can be an audio output device that achieves sound localization, i.e., forms a sound image, by working in concert with one or more speakers. Furthermore, speaker 5 can be an audio output device that changes the position of the sound image formation by adjusting the direction and volume of the sound output.

[0137] Figure 4 An example of the arrangement of the display device 4 and speaker 5 in the mobile body 9 is shown. Figure 4 This is a view of the movable body 9 from the ceiling. For example, the display device 4 and speaker 5 are shown... Figure 4 As shown, it is set on the wall of the mobile body 9 in a manner that surrounds the seat S1 where the passenger P1 sits.

[0138] More concrete examples can be given, such as in the interior space of the mobile body 9, where the display device 4 is installed on one or more of three walls, excluding the wall in the direction of travel, in a manner substantially parallel to the corresponding wall. The speaker 5 is installed, for example, on one or more walls and / or the ceiling surface in the interior space of the mobile body 9. The installation locations of the display device 4 and the speaker 5 are not limited to these examples. For example, several of the display device 4 and / or speaker 5 may also be installed on the floor surface of the mobile body 9.

[0139] The location, size, and shape of the display device 4 can be freely determined without affecting the operation of the movable body 9. For example, if the movable body 9 is manually operated, then... Figure 1 and Figure 4 As shown, the front of the control seat S2 can be set as a window, and the display device 4 can be set on other walls of the interior space. Alternatively, for example, if the mobile body 9 is fully automated, there is no control seat S2, so the display device 4 can be set on all walls of the interior space of the mobile body 9. However, in order to realize the aforementioned MR, it is preferable to set the display device 4 in a way that allows a view of the real world from at least a portion of the windows of the mobile body 9.

[0140] The installation height and orientation of the display device 4 are not particularly limited. However, the height and orientation of the display device 4 are preferably configured to allow the passenger P1 to easily view the image projected on the display device 4. Similarly, the installation height and orientation of the speakers 5 are not particularly limited, but the height and orientation of the speakers 5 can be configured to form a sound image in an appropriate position through the coordinated operation of each speaker 5. Although Figure 1 and Figure 4In the example, the display device 4 and the speaker 5 are separate devices, but the display device 4 and the speaker 5 can also be a single integrated device.

[0141] Processing flow

[0142] Figure 5 This is a flowchart of the processing flow in the output control device 1A. Figure 5 There is no particular limitation on when the processing shown can begin. For example, the output control device 1A can begin when the engine of the moving body 9 is running. Figure 5 The processing is as follows. Furthermore, when a passenger or operator of the mobile body 9 performs a prescribed input operation via an input device (not shown) provided with the mobile body 9, the output control device 1A can also be triggered to start... Figure 5 The processing.

[0143] First, the reading unit 12 reads virtual space data 31 from the storage device 3 (S11). The reading unit 12 then provides the read virtual space data 31 to the space processing unit 13.

[0144] Next, the position and orientation determination unit 11 determines the actual position and actual orientation of the moving body 9 based on the positioning data obtained from the positioning device 2 (S12). The position and orientation determination unit 11 provides the determined actual position and actual orientation to the space processing unit 13.

[0145] The virtual position determination unit 131 of the spatial processing unit 13 determines the virtual position and virtual orientation based on the provided real position and real orientation (S13). The virtual position determination unit 131 provides the determined virtual position and virtual orientation to the audio output control unit 15, the audio source determination unit 132, and the display image production unit 133.

[0146] Then, the display control processes shown in S21 to S23 and the audio output control processes shown in S31 to S33 are executed out of order or in parallel.

[0147] The display image production unit 133 determines the setting point and orientation of the virtual camera in the virtual space specified by the virtual space data 31 (S21). The number of virtual cameras is set according to the number of display devices 4 and the orientation of the virtual cameras is determined.

[0148] Next, the display image creation unit 133 depicts the areas in the virtual space reflected in each virtual camera, thereby creating images to be displayed on each display device 4 (S22). The display image creation unit 133 provides the created images to the display control unit 14. The display control unit 14 causes the display device 4 to display the images provided by the display image creation unit 133 (S23).

[0149] On the other hand, the sound source determination unit 132 determines the sound source body based on the virtual position and virtual orientation provided by the virtual position determination unit 131, and the sound source body outputs the sound it has been assigned (S31). The sound source determination unit 132 provides the identification information of the determined sound source body to the sound output control unit 15.

[0150] The playback unit 151 of the audio output control unit 15 determines the audio data 33 to be played based on the provided identification information and plays the audio data. While playing the audio, the playback unit 151 sequentially provides the audio to the audio-visual control unit 152 (and the allocation determination unit 153).

[0151] The sound image control unit 152 determines the position of the sound image of the output sound based on the virtual orientation of the moving body 9, the virtual position of the moving body 9, and the relative positional relationship between the moving body 9 and the sound source body (S32). The sound image control unit 152 transmits the determined sound image formation position to the allocation determination unit 153. The allocation determination unit 153 determines the speaker 5 used to output the sound being played by the playback unit 151. That is, the allocation determination unit 153 allocates the played sound to one or more speakers 5. The sound output control unit 15 controls each speaker 5 according to the allocation scheme for the speakers 5 determined by the allocation determination unit 153, so that the sound image is formed at the position determined by the sound image control unit 152 (S33).

[0152] When the display control processes shown in S21 to S23 and the audio output control processes shown in S31 to S33 end, or during the execution of these processes, the output control device 1A executes the S12 process again. That is, the position and orientation determination unit 11 performs the S12 process periodically. Furthermore, based on the actual position and actual orientation updated in S12, the processes after S12 are executed again. That is, if the actual position and actual orientation are updated, the virtual position and virtual orientation are also updated, thereby updating the virtual space area to be displayed by the display device 4, and updating the position of the sound image formed by the sound output from the speaker 5.

[0153] Therefore, through repeated practice Figure 5 Following S12, the output control device 1A can control the display of the display device 4 and the audio output of the speaker 5 in real time based on the ever-changing real position and orientation of the moving body 9 while it is moving.

[0154] Location of Virtual Sound Sources and Sound Image Location

[0155] Figure 6 This diagram illustrates the changing relative positions of the moving object 9 in virtual space and the sound source, as well as the changing positions of the sound image in real space. The diagram shows the various parts of the moving object 9 and... Figure 3 The same applies, so I won't repeat it further. Also, although... Figure 6 In the example, two speakers 5 are used to form a sound image, but a sound can also be output from one or more speakers 5 to form a sound image of that sound.

[0156] Figure 6 For ease of explanation, the positional relationship between the moving body 9 and the sound source body shown is from a virtual space perspective, while the internal space of the moving body 9 represents a phenomenon occurring in real space. Furthermore, Figure 6 In this context, we assume that time flows by transitioning from the top column graph to the middle column graph and from the middle column graph to the bottom column graph.

[0157] like Figure 6 As shown in the top and middle columns, suppose the sound source, as seen from the moving object 9 in virtual space, has moved backward. In this case, the sound image also moves backward. Additionally, as... Figure 6 As shown in the middle and lower columns, assume that the sound source has moved to the right and rear relative to the moving body 9. In this case, the sound image also moves to the right and rear.

[0158] As previously stated, display device 4 displays an image including the sound source. Therefore, as Figure 6 As shown, by changing the position of the sound image, its position can be altered in a way that links it to the display position of the sound source. Therefore, this allows the passenger, P1, to experience a more immersive audio experience.

[0159] In addition, the audio output control unit 15 can control the output volume of the speaker 5 in such a way that the greater the distance between the moving body 9 in the virtual space and the sound source body, the smaller the audio output corresponding to the sound source body.

[0160] For example, if the audio source setting information 32 includes the set volume of the audio data 33, and the audio source determination unit 132 determines the playback volume of each audio data 33, then the farther the distance from the virtual position of the moving body 9 to the audio source body, the smaller the playback volume of the audio data 33 by the playback unit 151. Therefore, the audio output control unit 15 can reduce the actual output volume of the speaker 5 without changing the volume amplifier level of the speaker 5.

[0161] Additionally, for example, the audio output control unit 15 can obtain information from the audio source determination unit 132 indicating the distance from the virtual position of the moving body 9 to the audio source body. Furthermore, each speaker 5 can be controlled in such a way that the greater the distance, the smaller the output volume of the audio corresponding to the audio source body.

[0162] Therefore, the positional relationship between the moving object 9 in the real space and the sound source in the virtual space can be expressed not only by the location of the sound image but also by the volume of the sound. This allows passengers to experience a more immersive audio experience.

[0163] Furthermore, the audio-visual control unit 152 can render the sound output from the speaker 5 in such a way that the farther the distance between the moving body 9 and the sound source in the virtual space, the lower the pitch of the sound corresponding to the sound source. This allows, for example, the reproduction of sound effects such as an ambulance siren. Therefore, passengers can experience a more immersive audio experience.

[0164] [Implementation Method 2]

[0165] The output control device of the present invention can control the operation of the oscillator according to the audio output. Hereinafter, refer to... Figure 7 and Figure 8 The second embodiment of the present invention will now be described. For ease of explanation, components that have the same function as those described in the previous embodiments will be given the same reference numerals and will not be described again. The same applies to the following embodiments.

[0166] Figure 7 This is a main structural block diagram of the various devices constituting the output control system 200 of this embodiment. The output control system 200 differs from the output control system 100 in that it includes the vibrator 6 and the output control device 1B. Furthermore, the output control system 200 may, like the output control system 100, include an in-vehicle camera and / or in-vehicle sensors capable of detecting the occupant's head position. Additionally, the output control device 1B may be connected and communicate with the in-vehicle camera and / or in-vehicle sensors.

[0167] The vibrator 6 has multiple vibrators. The output control system 200 may have multiple vibrators 6. Figure 8 An example of the configuration of the vibrator 6 in the output control system 200 is shown. For example... Figure 8 As shown, the vibrator 6 may be installed, for example, on the back and seat surface of the seat S1, and on the floor surface that the feet of the passenger P1 contact when the passenger P1 is sitting on the seat S1.

[0168] The output control device 1B differs from the output control device 1A of Embodiment 1 in that it includes a vibration control unit 16. In the output control device 1B, when the speaker 5 is operated, the audio output control unit 15 outputs an operation instruction corresponding to the instruction given to the speaker 5 to the vibration control unit 16.

[0169] The vibration control unit 16 controls the operation of the vibrator 6 according to the instructions of the audio output control unit 15. For example, the vibration control unit 16 can control the vibrator located at a certain position in a certain vibrator to vibrate in a certain way. In addition, the vibration mode mentioned here refers to the vibration intensity, vibration interval, vibration and the timing of stopping, etc.

[0170] The content and timing of the instructions given by the audio output control unit 15 to the vibration control unit 16, as well as the control measures performed by the vibration control unit 16 on the vibrator 6, are not particularly limited. For example, the instructions given by the audio output control unit 15 to the vibration control unit 16 may include information indicating the output volume of each speaker 5. Furthermore, the vibration control unit 16 may determine the vibration position and / or intensity of the vibrator 6 based on the output volume of each speaker 5.

[0171] For example, if Figure 8 If the volume of the three speakers 5 on the left side of passenger P1 is set to be greater than the volume of the two speakers 5 on the front and three speakers 5 on the right side of passenger P1, then the vibration control unit 16 can make the vibration intensity on the left side of the vibrator 6 on the back of seat S1 stronger than the vibration intensity on the right side of the vibrator 6.

[0172] Through the above processing, as the audio data 33 is output, a vibrational stimulus that is linked to the output mode of the audio data 33 can be provided to the passenger P1. For example, if the audio data 33 is a deep bass or an impact sound, the output and vibration of the audio data 33 are provided to the passenger, thereby more accurately reproducing the phenomenon experienced by the person when the sound actually occurs. Therefore, a more immersive audio experience can be provided to the passenger P1.

[0173] [Implementation Method 3]

[0174] Although the scheme of displaying an image depicting a virtual space across the entire screen area of ​​the display device 4 has been described using output control systems 100 and 200 as examples, the output control system of the present invention can also overlay a portion of a real-world image onto the virtual space or objects contained therein, thereby creating an image and displaying it on the display device 4. In other words, the output control system of the present invention can provide passengers with a virtual experience in an AR (Augmented Reality) space.

[0175] Furthermore, although the scheme of the speaker 5 playing pre-stored audio data 33 was described using output control systems 100 and 200 as examples, the output control system of the present invention can also cause the speaker 5 to output: the audio currently being emitted in the real world, and the audio data 33 after adjusting the position of the audio image. Hereinafter, refer to... Figure 9 The third embodiment of the present invention will now be described.

[0176] Structure of the main parts

[0177] Figure 9 This is a main structural block diagram of the various devices constituting the output control system 300 of this embodiment. The output control system 300 differs from the output control systems 100 and 200 in that it includes a camera 7, a microphone 8, and an output control device 1C. Furthermore, the microphone 8 is not a necessary component in the output control system 300. Additionally, if the positioning device 2 is a camera, then the positioning device 2 can also function as the camera 7.

[0178] Additionally, the output control system 300 may include an in-vehicle camera and / or in-vehicle sensors capable of detecting the position of the passenger's head. Furthermore, the output control device 1C may be connected and communicate with the in-vehicle camera and / or in-vehicle sensors.

[0179] Camera 7 is used to capture images of the real space surrounding the moving body 9. There may be one or more cameras 7, preferably one or more cameras 7 can be used to capture 360° images of the surroundings of the moving body 9. Camera 7 provides the captured images to the output control device 1C.

[0180] Microphone 8 is used to acquire sounds occurring in the real space surrounding the moving body 9. There may be one or more microphones 8, preferably one or more microphones 8 to acquire sounds around the moving body 9 in a 360° radius. The microphones 8 provide the acquired sound data to the output control device 1C. In this embodiment, to distinguish it from sound data 33, the "sounds occurring in the real space surrounding the moving body 9" will be referred to as "ambient sound." However, ambient sound in this specification is not simply the sound of objects touching each other, but may include various sounds such as human speech.

[0181] The output control device 1C acquires captured images from the camera 7. Additionally, the output control device 1C acquires ambient sound from the microphone 8. The spatial processing unit 13 of the output control device 1C includes a display image production unit 134. The audio output control unit 15 of the output control device 1C includes an allocation decision unit 153.

[0182] The display image creation unit 134 can combine the image captured by the camera 7 with an image depicting a portion of a virtual space or an object to create an image to be displayed on each display device 4. Hereinafter, unless otherwise specified, the display image creation unit 134 performs the same processing as the display image creation unit 133 described in embodiments 1 and 2.

[0183] The allocation decision unit 153 of the audio output control unit 15 can determine the allocation scheme of the audio data 33, and also the allocation scheme of the ambient sound acquired from the microphone 8. If the output control system 300 does not include the microphone 8, the allocation decision unit 153 can also perform the same processing as the allocation decision unit 153 in embodiments 1 and 2.

[0184] In addition, if the output control device 1C is connected to the aforementioned in-vehicle camera and / or in-vehicle sensor, the space processing unit 13 of the output control device 1C can also determine the position of the passenger's head based on the image captured by the in-vehicle camera and / or the measurement results of the in-vehicle sensor.

[0185] (Image creation)

[0186] The display image production unit 134 extracts the area to be displayed on each display device 4 from the captured image obtained by the camera 7. The display image production unit 134 preferably determines the area to be extracted from the captured image based on the following (1) to (3).

[0187] (1) The actual position and orientation (i.e., direction of travel) of the moving object 9;

[0188] (2) The position and orientation of each display device 4 in real space;

[0189] (3) The size and shape of each display device 4.

[0190] If the spatial processing unit 13 can determine the position of the passenger's head, the display image production unit 134 can also determine the area to be cropped from the captured image based on the position of the passenger's head, in addition to the above (1) to (3).

[0191] (A composite of real-world scenery and virtual landscapes)

[0192] The display image production unit 134 composites an image cropped from a captured image with an image of a portion of a virtual space or an object, thereby creating an image to be displayed on the display device 4. In this way, the display image production unit 134 can composite a landscape in real space with a landscape or object in virtual space. In this compositing scheme, the method of compositing the image cropped from the captured image with the image in virtual space is not particularly limited. Furthermore, the display area ratio of the image cropped from the captured image and the virtual space image, as well as their layout, are not particularly limited.

[0193] For example, the image production unit 134 can crop out an area from a captured image that can be seen through the window from the interior space of the moving body 9 when the display device 4 is simulated as a window. Furthermore, the cropped image, i.e., a real-world landscape image, can be composited with an image of an object in virtual space and displayed.

[0194] Through the above processing, the output control system 300 can make the passenger feel as if a part of the virtual space (in the specific example above, an object) has appeared in the real world. That is, the output control system 300 can enable the passenger to experience augmented reality (AR). Furthermore, if the output control system 300 includes a microphone 8, it can output ambient sounds from the real world as well as sounds used to create the virtual space. Therefore, the output control system 300 can enable the passenger to experience augmented reality (AR).

[0195] (A variation of implementation method 3)

[0196] Figure 9 As an example, an output control system 300 is shown that incorporates the characteristic scheme of this embodiment into the output control system 100 of embodiment 1. However, the output control system 300 may also include the scheme of the output control system 200 of embodiment 2, and perform the processing described in embodiment 2 together.

[0197] [Implementation Method 4]

[0198] Although the embodiments described above show that the positioning device 2 is a device mounted on the moving body 9, the positioning device 2 in the output control systems 100, 200, and 300 may also be a device that is independently set up from the moving body 9.

[0199] For example, the positioning device 2 described in embodiments 1 to 3 can also be an external camera installed near the road. The external camera is used to capture images of the moving body 9 traveling on the road. The external camera has a communication function and can communicate with the output control devices 1A, 1B, or 1C. In addition, the output control devices 1A, 1B, or 1C have a communication unit. The output control devices 1A, 1B, or 1C communicate with the external camera near the road through the communication unit, thereby acquiring the images captured by the external camera. Furthermore, the position and orientation determination unit 11 determines the actual position and actual orientation of the moving body 9 based on the images captured by the external camera and the map data 34.

[0200] Alternatively, for example, the positioning device 2 can be a communication base station existing in the real space. In this case, the output control devices 1A, 1B, or 1C have a communication unit and can communicate simultaneously with at least three of the aforementioned communication base stations. Furthermore, the position orientation determination unit 11 determines the actual position of the mobile body 9 based on the relative positional relationship between the communication base station and the mobile body 9. In addition, as mentioned above, the position orientation determination unit 11 determines the actual position multiple times and determines the direction of change of this position as the traveling direction of the mobile body 9, that is, the actual orientation of the mobile body 9.

[0201] With the above solution, even if the positioning device 2 is not installed in the mobile body 9, the actual location of the mobile body 9 can be determined using information obtained from, for example, surveillance cameras or base stations. Therefore, the number of components installed in the mobile body 9 can be reduced.

[0202] [Examples implemented using software]

[0203] The control blocks of output control devices 1A, 1B, and 1C can be implemented by logic circuits (hardware) formed in integrated circuits (IC chips), or by software.

[0204] When implemented via software, the output control devices 1A, 1B, and 1C include a computer capable of executing software program commands to achieve each function. This computer, for example, includes one or more processors and a storage medium storing the program in a computer-readable manner. Furthermore, the objective of the present invention is achieved by the processor reading and executing the program from the storage medium. The processor can be, for example, a CPU (Central Processing Unit). The storage medium can be, for example, a "non-temporary physical medium" such as ROM (Read Only Memory), as well as storage tapes, storage disks, memory cards, semiconductor memories, programmable logic circuits, etc. Additionally, RAM (Random Access Memory) for deploying the program can also be included. Furthermore, the program can be provided to the computer via any transmission medium capable of transmitting the program (communication networks, broadcast waves, etc.). Moreover, even if the program is in the form of a data signal carried on a carrier wave and transmitted electronically, one aspect of the present invention can still be achieved.

[0205] This invention is not limited to the embodiments described above. Various modifications can be made within the scope shown in the specification. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included within the technical scope of this invention.

Claims

1. An output control device for controlling a display device and a plurality of speakers disposed on at least one wall surface in the interior space of a mobile body in which a passenger resides, characterized in that, have: The position and orientation determination unit determines the actual position of the moving body in real space and the orientation of the moving body in real space; The reading unit reads virtual space data from the storage device, wherein the virtual space data defines a virtual space formed by setting a 3D sound source body as a virtual sound source on a 3D mapping map; The virtual position determination unit determines the virtual position of the mobile body in the virtual space and the orientation of the mobile body in the virtual space based on the real position of the mobile body and the orientation of the mobile body in the real space. The display control unit, based on the virtual position of the moving body, the orientation of the moving body in the virtual space, and the placement position of the display device, causes the display device to display an image of a portion of the virtual space and an image of a portion of the sound source; and The audio output control unit causes the plurality of speakers to output audio, thereby forming a sound image in the internal space. The audio output control unit controls the sound image formation position based on the orientation of the moving body in the virtual space and the relative positional relationship between the virtual position of the moving body and the sound source body. This is done by linking the sound image formation position with the display position of the sound source body in the display device and ensuring that the orientation of the sound image formation position in the internal space is consistent with the orientation viewed from the virtual position and virtual orientation of the moving body relative to the sound source body. The location where the sound image is formed corresponds to the virtual position of the moving body and the orientation of the moving body in the virtual space.

2. The output control device according to claim 1, characterized in that, The position and orientation determination unit periodically updates the actual position of the moving body and its orientation in the real space. The virtual location determination unit updates the virtual location and the orientation of the moving body in the virtual space based on the updated real location and the updated orientation of the moving body in the real space. Based on the updated virtual position and the updated orientation of the moving body in the virtual space, the display control unit causes the display device to display the updated image of the partial area and the updated image of the partial sound source body. The audio output control unit updates the position of the sound image based on the updated virtual position and the updated orientation of the moving body in the virtual space.

3. The output control device according to claim 1 or 2, characterized in that, The audio output control unit controls the speaker in such a way that the farther the distance between the virtual position in the virtual space and the audio source, the smaller the output volume of the audio corresponding to the audio source.

4. The output control device according to claim 1 or 2, characterized in that, The virtual location determination unit determines the virtual location in such a way that the positional relationship between any reference point in the virtual space and the virtual location approximates the positional relationship between any reference point in the real space and the real location.

5. The output control device according to claim 1 or 2, characterized in that, The display control unit determines which portion of the image and the portion of the sound source body the display device should display, so that the virtual space and the real space can be displayed at the same scale.

6. An output control system, characterized in that, have: The output control device according to any one of claims 1 to 5; A positioning device that measures data of the actual position of the moving body; The storage device; The display device; as well as The speaker, The position and orientation determination unit of the output control device determines the actual position of the moving body and its orientation in real space based on the actual position data obtained by the positioning device.

7. A control method for controlling a display device and a plurality of speakers disposed on at least one wall surface in the interior space of a moving body in which a passenger resides, characterized in that, include: The position and orientation determination step determines the real position of the moving body in real space and the orientation of the moving body in real space; The reading step reads virtual space data from the storage device, wherein the virtual space data defines a virtual space formed by setting a 3D sound source body as a virtual sound source on a 3D mapping map; The virtual position determination step determines the virtual position of the mobile body in the virtual space and the orientation of the mobile body in the virtual space based on the real position of the mobile body and the orientation of the mobile body in the real space. The display control step, based on the virtual position of the moving body, the orientation of the moving body in the virtual space, and the setting position of the display device, causes the display device to display an image of a portion of the virtual space and an image of a portion of the sound source; and The audio output control step causes the plurality of speakers to output audio to form a sound image in the internal space. In the audio output control step, based on the orientation of the moving body in the virtual space and the relative positional relationship between the virtual position of the moving body and the sound source body, the sound image formation position is controlled in such a way that it is linked to the display position of the sound source body in the display device, and the orientation of the sound image formation position in the internal space is consistent with the orientation viewed from the virtual position and virtual orientation of the moving body relative to the sound source body. The location where the sound image is formed corresponds to the virtual position of the moving body and the orientation of the moving body in the virtual space.

Images