Information processing system, control method for information processing system, and program

Abstract

To provide an information processing system that gives a sense of realism as if a user is in a vehicle even when the user is outside the vehicle.SOLUTION: A video information acquisition unit 10 acquires video information related to video captured by an imaging device 3 which moves together with a vehicle 4. A behavior information acquisition unit 20 acquires behavior information related to the behavior of the vehicle 4 when the imaging device 3 captured the video. A display control unit 30 displays an image based on the video information on a user terminal 2 located outside the vehicle 4. An output sound control unit 50 uses the behavior information to control the sound to be output by the user terminal 2.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for displaying an image of the area in front of a vehicle on an external terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-213791 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the case of Patent Document 1 mentioned above, the vehicle information available to a user outside the vehicle is limited to images of the vehicle, audio inside the vehicle, and vehicle position information, making it difficult for the user outside the vehicle to get the feeling that they are actually riding in the vehicle.

[0005] One example of a problem to be solved by the present invention is to provide an information processing system that allows a user to feel as if he or she is inside a vehicle even when the user is outside the vehicle. [Means for solving the problem]

[0006] The invention described in claim 1 is a video information acquisition unit that acquires video information related to a video captured by an imaging device that moves together with the vehicle; a behavior information acquisition unit that acquires behavior information regarding a behavior of the vehicle when the imaging device captures the video; a display control unit that displays an image based on the video information on a user terminal located outside the vehicle; and an output sound control unit that uses the behavior information to control the sound to be output by the user terminal.

[0007] The invention described in claim 7 is The computer acquiring video information relating to a video captured by an imaging device moving together with the vehicle; acquiring behavior information regarding a behavior of the vehicle when the imaging device captures the video; Displaying an image based on the video information on a user terminal located outside the vehicle; The control method for an information processing system uses the behavior information to control a sound to be output from the user terminal.

[0008] The invention described in claim 8 is On the computer, acquiring video information relating to a video captured by an imaging device moving with the vehicle; acquiring behavior information regarding a behavior of the vehicle when the imaging device captures the video; a step of displaying an image based on the video information on a user terminal located outside the vehicle; The program is for executing a procedure for controlling the sound to be output by the user terminal using the behavior information. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram for explaining an information processing system according to a first embodiment. [Diagram 2] FIG. 13 is a diagram showing an image based on video information displayed on a user terminal. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a flow chart showing processing of the information processing system according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram illustrating an information processing system according to a second embodiment. [Figure 6]FIG. 11 is a flowchart showing processing of an information processing system according to a second embodiment. [Figure 7] FIG. 11 is a block diagram illustrating an information processing system according to a third embodiment. [Figure 8] FIG. 13 is a diagram showing an example 1 of behavior state information added to an image based on video information. [Figure 9] FIG. 11 is a diagram showing a second example of behavior state information added to an image based on video information. [Figure 10] FIG. 11 is a diagram showing a third example of behavior state information added to an image based on video information. [Figure 11] FIG. 11 is a diagram showing a fourth example of behavior state information added to an image based on video information. [Figure 12] FIG. 11 is a diagram showing a fifth example of behavior state information added to an image based on video information. [Figure 13] FIG. 13 is a diagram showing an example 1 of blurring processing applied to an image based on video information. [Figure 14] FIG. 11 is a diagram showing a second example of blurring processing applied to an image based on video information. [Figure 15] FIG. 13 is a diagram showing a state in which an image based on video information is rotated. [Figure 16] FIG. 11 is a flowchart showing processing of an information processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals and the description will be omitted as appropriate.

[0011] In the following description, each component of each device is represented as a functional block, not as a hardware configuration. Each component of each device is realized by any combination of hardware and software, centered on the CPU of any computer, memory, a program loaded into the memory, a storage medium such as a hard disk that stores the program, and a network connection interface. There are various variations in the realization method and device.

[0012] First Embodiment (Information processing system 100A) 1 is a block diagram for explaining an information processing system 100A according to the first embodiment. The information processing system 100A according to the first embodiment includes an information processing device 1A, a user terminal 2, and an imaging device 3. In the first embodiment, the information processing device 1A is provided outside a vehicle 4. The information processing device 1A may be a so-called cloud server.

[0013] The information processing device 1A is configured to be able to communicate with a user terminal 2 (described later) and an imaging device 3 (described later) via a communication network 101. The communication network 101 includes, for example, communication over a 4G or 5G line. The information processing device 1A is configured to include an external vehicle communication unit (not shown) for configuring the communication network 101.

[0014] (Imaging device 3) The vehicle 4 is equipped with an imaging device 3. In the first embodiment, the imaging device 3 is inside the vehicle 4. The imaging device 3 may be outside the vehicle 4. The imaging device 3 may be brought in from outside the vehicle 4, or may be attached to the vehicle 4 in advance. The imaging device 3 moves together with the vehicle 4.

[0015] In the first embodiment, the imaging device 3 captures an image of the front (traveling direction) of the vehicle 4. The imaging device 3 may also be able to capture an image of the width direction (lateral direction) of the vehicle 4 or the rear of the vehicle 4. The imaging device 3 generates video information related to the captured video. The imaging device 3 may include a sound collection unit (not shown), generate audio information related to the sound inside the vehicle 4 acquired by the sound collection unit, and transmit the audio information to the information processing device 1A. The imaging device 3 may also transmit behavior information (described later) related to the behavior of the vehicle 4 to the information processing device 1A.

[0016] The imaging device 3 is configured to be able to communicate with the information processing device 1A via a communication network 101. The communication network 101 includes, for example, communication over a 4G or 5G line. The imaging device 3 is configured to include an external vehicle communication unit (not shown) for configuring the communication network 101. Note that the imaging device 3 and the information processing device 1A may communicate with each other using WebRTC (Web Real-Time Communication).

[0017] (User terminal 2) The user terminal 2 is located outside the vehicle 4. The user terminal 2 is a terminal of a user outside the vehicle 4, and is a terminal owned, managed, carried, or used by the user outside the vehicle 4. The user terminal 2 includes any one of a smartphone, a tablet, and a PC (personal computer).

[0018] An application program for a user, which is used in the information processing system 100, is installed in the user terminal 2. The user uses the application by starting the application on the user terminal 2. The application will be described in detail later.

[0019] The user terminal 2 is configured to be able to communicate with the information processing device 1A via a communication network 101. The communication network 101 includes, for example, communication over a 4G or 5G line. The user terminal 2 is configured to include an external vehicle communication unit (not shown) for configuring the communication network 101. The user terminal 2 and the information processing device 1A may communicate with each other using WebRTC (Web Real-Time Communication).

[0020] (Video information acquisition unit 10) The information processing device 1A according to the first embodiment includes a video information acquisition unit 10, a behavior information acquisition unit 20, a display control unit 30, and a vibration control unit 40. The video information acquisition unit 10 acquires video information related to a video captured by an imaging device 3. In the first embodiment, the video information includes, for example, a video captured in the forward direction (traveling direction) of the vehicle 4.

[0021] (Display control unit 30) The display control unit 30 causes the user terminal 2 to display an image based on the video information. The "image based on the video information" may be a video. The "image based on the video information" may include a video captured by the imaging device 3 with a reduced frame rate.

[0022] FIG. 2 is a diagram showing an image 11 based on video information displayed on the user terminal 2. In FIG. 2, the imaging device 3 captures an image of the area ahead of the vehicle 4, and the display control unit 30 displays the video captured by the imaging device 3 on the display of the user terminal 2 in real time. Note that real time does not necessarily mean completely simultaneous, and there may be a time difference of several seconds to several tens of seconds. In FIG. 2, the display control unit 30 displays a video of the vehicle 4 traveling through a tunnel approaching a right curve on the user terminal 2. Reference numeral 12 in FIG. 2 indicates a one-lane road 12 on one side, and reference numeral 13 in FIG. 2 indicates a division line 13. Note that, in addition to the video captured by the imaging device 3, map information indicating the current position of the vehicle 4, speed information indicating the current traveling speed of the vehicle 4, and the like may be displayed simultaneously on the display of the user terminal 2.

[0023] When the user activates a user application from the user terminal 2, the user terminal 2 can exhibit a function of displaying an image 11 based on video information on the user terminal 2.

[0024] (Behavior information acquisition unit 20) Returning to FIG. 1, the behavior information acquisition unit 20 acquires behavior information transmitted from the vehicle 4. The behavior information is information related to the behavior of the vehicle 4 when the imaging device 3 captures a video. The behavior information is information related to the behavior of the vehicle 4 when the imaging device 3 captures a video. The behavior information is information related to the behavior of the vehicle 4 when an image 11 (see FIG. 2) based on the video information is displayed in real time on the user terminal 2.

[0025] The behavior information includes at least one of acceleration information related to the acceleration of the vehicle 4 and rotation speed information related to the rotation speed of the engine of the vehicle 4. The acceleration of the vehicle 4 includes at least one of the acceleration in the traveling direction of the vehicle 4, the acceleration in the width direction (lateral direction) of the vehicle 4, and the acceleration of the vehicle 4 in the vertical direction. The acceleration information may be generated by a three-axis acceleration sensor (not shown) mounted on the vehicle 4 or the imaging device 3. The engine rotation speed may also be generated by an engine rotation speed sensor (not shown) mounted on the vehicle 4. The imaging device 3 may transmit the acceleration information and the rotation speed information acquired via an OBD (On board diagnostics) interface of the vehicle 4 to the information processing device 1A, and the behavior information acquisition unit 20 may acquire the behavior information transmitted from the imaging device 3.

[0026] (Vibration control unit 40) The vibration control unit 40 uses the above behavior information to control the vibration of the user terminal 2. The vibration control unit 40 controls the vibration of the user terminal 2 by adjusting at least one of the interval at which the user terminal 2 vibrates and the duration of the vibration of the user terminal 2.

[0027] In the first embodiment, the vibration control unit 40 may control the vibration of the user terminal 2 so that the vibration interval of the user terminal 2 becomes shorter as the acceleration of the vehicle 4 increases or the engine speed of the vehicle 4 increases. The vibration control unit 40 may also control the vibration of the user terminal 2 so that the vibration duration of the user terminal 2 becomes longer as the acceleration of the vehicle 4 increases or the engine speed of the vehicle 4 increases. For example, when the engine speed is low, the vibration interval may be set to a longer pulse interval, and the pulse interval may be shortened as the engine speed increases.

[0028] (Hardware configuration example) 3 is a diagram showing an example of the hardware configuration of the information processing device 1 A. The information processing device 1 A has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0029] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0030] The processor 1020 is implemented by a central processing unit (CPU) or a graphics processing unit (GPU).

[0031] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0032] The storage device 1040 is an auxiliary storage device realized by a removable medium such as a hard disk drive (HDD), a solid state drive (SSD), a memory card, or a read only memory (ROM), and has a recording medium. The recording medium of the storage device 1040 stores program modules that realize each function of the information processing device 1A (for example, the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the vibration control unit 40). The processor 1020 loads each of these program modules onto the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also function as a memory unit (not shown) included in the information processing device 1A.

[0033] The input / output interface 1050 is an interface for connecting the information processing device 1A to various input / output devices.

[0034] The network interface 1060 is an interface for connecting the information processing device 1A to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may connect to the network wirelessly or by wire. The information processing device 1A may communicate with the user terminal 2 and the imaging device 3 via the network interface 1060.

[0035] (Operation example of the first embodiment) 4 is a flow diagram showing the processing of the information processing system 100 according to the first embodiment. First, in step S100, the video information acquisition unit 10 acquires video information captured by the imaging device 3. In the first embodiment, the imaging device 3 captures an image ahead of the vehicle 4, and the video information acquisition unit 10 acquires video information relating to the image ahead of the vehicle 4.

[0036] Next, in step S200, the display control unit 30 displays, in real time, an image 11 (see FIG. 2) based on video information captured by the imaging device 3 on the user terminal 2. This allows the user to share the view of the front of the vehicle 4 with the driver even if the user is outside the vehicle 4.

[0037] Next, in step S300, the behavior information acquisition unit 20 acquires behavior information. For example, the behavior information acquisition unit 20 acquires the acceleration of the vehicle 4 in the vertical direction or the engine speed as the behavior information transmitted from the vehicle 4.

[0038] In step S400, the behavior information acquisition unit 20 judges whether the behavior information acquired in step S300 exceeds a predetermined criterion. For example, when the acquired behavior information is the acceleration of the vehicle 4 in the vertical direction, the threshold value of the acceleration is 3 [m / s 2 ], and the acceleration of the vehicle 4 in the vertical direction as the behavior information acquired in step S300 is 7 [m / s 2 ], the behavior information acquisition unit 20 may determine that the behavior information acquired in step S300 has exceeded the predetermined standard. In addition, in a case where the acquired behavior information is the engine speed of the vehicle 4, the threshold value of the engine speed is 4000 [rpm], and the engine speed of the vehicle 4 as the behavior information acquired in step S300 is 4500 [rpm], the behavior information acquisition unit 20 may determine that the behavior information acquired in step S300 has exceeded the predetermined standard.

[0039] If the behavior information acquired in step S300 exceeds the predetermined standard (YES in step S400), the process proceeds to step S500. If the behavior information acquired in step S300 does not exceed the predetermined standard (NO in step S400), the process may end. Note that step S400 may be omitted.

[0040] In step S500, the vibration control unit 40 controls the vibration of the user terminal 2. The vibration control unit 40 may control the vibration function of the user terminal 2. The vibration control unit 40 may control at least one of the vibration interval when the user terminal 2 is vibrated intermittently and the vibration duration when the user terminal 2 is vibrated continuously, according to the magnitude of the acceleration of the vehicle 4 in the vertical direction. The vibration control unit 40 may shorten the vibration interval of the user terminal 2 or lengthen the vibration duration of the user terminal 2 as the acceleration of the vehicle 4 in the vertical direction increases. Alternatively, in step S500, the vibration control unit 40 may control at least one of the vibration interval when the user terminal 2 is vibrated intermittently and the vibration duration when the user terminal 2 is vibrated continuously, according to the engine speed of the vehicle 4. The vibration control unit 40 may shorten the vibration interval of the user terminal 2 or lengthen the vibration duration of the user terminal 2 as the engine speed of the vehicle 4 increases.

[0041] As described above, the information processing system 100 according to the first embodiment includes the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the vibration control unit 40. Since the user terminal 2 vibrates in accordance with the behavior (acceleration, engine speed) of the actual vehicle 4, the user can virtually experience the vibration of the vehicle 4 as if he or she were riding in the vehicle 4. Therefore, even when the user is outside the vehicle 4, the user can get a sense of realism as if he or she were actually riding in the vehicle 4.

[0042] Furthermore, the behavior information includes at least one of acceleration information related to the acceleration of the vehicle and rotation speed information related to the rotation speed of the engine of the vehicle, whereby information related to the vibration of the vehicle 4 can be easily acquired.

[0043] Furthermore, the acceleration of the vehicle 4 includes the acceleration of the vehicle 4 in the vertical direction. This allows the user to virtually experience the vibration caused by a step on the road, for example, when the vehicle 4 vibrates in the vertical direction.

[0044] Furthermore, the vibration control unit 40 controls the vibration of the user terminal 2 by adjusting at least one of the interval at which the user terminal 2 is vibrated and the duration of vibration of the user terminal 2. This makes it possible to effectively reproduce the vibration of an actual vehicle 4 on the user terminal 2.

[0045] <Second embodiment> 5 is a block diagram for explaining an information processing system 100B according to the second embodiment. The information processing device 1B according to the second embodiment does not include a vibration control unit 40, but includes an output sound control unit 50 and a vehicle model information acquisition unit 60. Functions other than the output sound control unit 50 and the vehicle model information acquisition unit 60 are similar to those of the first embodiment.

[0046] (Output sound control unit 50) The output sound control unit 50 uses the behavior information to control the sound to be output by the user terminal 2. In the second embodiment, the sound is an engine sound. The engine sound may be a pseudo engine sound that imitates an actual engine sound of the vehicle 4. The engine sound may be stored in advance in a storage unit (not shown) as an engine sound file. The output sound control unit 50 according to the second embodiment causes the user terminal 2 to output the engine sound as the sound.

[0047] The output sound control unit 50 changes the engine sound in accordance with at least one of acceleration information related to the acceleration in the traveling direction of the vehicle 4 and rotation speed information related to the rotation speed of the engine of the vehicle 4.

[0048] In the second embodiment, for example, the output sound control unit 50 may control the engine sound output by the user terminal 2 so that the playback speed of the engine sound output by the user terminal 2 increases as the engine speed of the vehicle 4 increases. Alternatively, the output sound control unit 50 may control the engine sound output by the user terminal 2 so that the playback speed of the engine sound output by the user terminal 2 increases as the acceleration in the traveling direction of the vehicle 4 increases, assuming that the engine speed of the vehicle 4 increases in a scene where the acceleration in the traveling direction of the vehicle 4 is large. By increasing the playback speed of the engine sound according to the engine speed of the vehicle 4 or the acceleration in the traveling direction of the vehicle 4, the high-speed engine sound of the actual vehicle 4 can be reproduced. Therefore, for example, the user can experience the sensation of the acceleration of the vehicle 4.

[0049] (Vehicle type information acquisition unit 60) The vehicle model information acquisition unit 60 acquires vehicle model information relating to the vehicle model of the vehicle 4. The vehicle model information includes information regarding the vehicle model (model) of the vehicle 4. The vehicle model information may include information regarding the vehicle classification, such as a regular car and a light vehicle. The vehicle model information may include information regarding the vehicle category, such as a sports car, a sedan, or a van. The vehicle model information may also include, for example, a specific manufacturer name (e.g., Ferrari (registered trademark), Lamborghini (registered trademark), etc.) or a product name (e.g., Ferrari 458).

[0050] The output sound control unit 50 controls the sound by using the vehicle model information. Specifically, for example, when the vehicle model information acquisition unit 60 acquires information related to a product name (e.g., Ferrari 458) as vehicle model information, the output sound control unit 50 selects a file of an engine sound corresponding to the product name and causes the user terminal 2 to output the engine sound.

[0051] The hardware configuration of the information processing device 1A according to the second embodiment is similar to that of the first embodiment.

[0052] (Operation example of the second embodiment) 6 is a flow chart showing the processing of the information processing system 100 according to the second embodiment. Steps S100 to S300 are the same as those in the first embodiment.

[0053] In step S350 after step S300, the vehicle model information acquisition unit 60 acquires vehicle model information. In the second embodiment, for example, it is assumed that the vehicle model information acquisition unit 60 acquires information on the product name of the vehicle 4 (e.g., Ferrari 458) as the vehicle model information.

[0054] In step S510 after step S350, the output sound control unit 50 uses the behavior information and the vehicle model information to control the sound to be output by the user terminal 2. In step S510, the output sound control unit 50 uses the product name (e.g., Ferrari 458) of the vehicle 4 as the vehicle model information acquired in step S350 to acquire an engine sound file corresponding to Ferrari 458 from a storage unit (not shown), and causes the user terminal 2 to output the engine sound stored in the sound file. Then, using rotation speed information related to the rotation speed of the engine of the vehicle 4 as the behavior information acquired in step S300, the output sound control unit 50 plays the above sound file at a playback speed according to the rotation speed of the engine of the vehicle 4. In the second embodiment, for example, the output sound control unit 50 may control the engine sound to be output by the user terminal 2 so that the playback speed of the engine sound to be output by the user terminal 2 becomes faster as the rotation speed of the engine of the vehicle 4 becomes higher.

[0055] Alternatively, in step S510, instead of controlling the playback speed of the engine audio file using rotation speed information related to the rotation speed of the engine of the vehicle 4 as the behavior information acquired in step S300, the output sound control unit 50 may control the playback speed of the engine audio file using acceleration information related to the acceleration in the traveling direction of the vehicle 4 as the behavior information acquired in step S300. That is, the output sound control unit 50 may control the engine sound to be output to the user terminal 2 such that the playback speed of the engine sound to be output to the user terminal 2 becomes faster as the acceleration in the traveling direction of the vehicle 4 becomes greater, assuming that the engine speed of the vehicle 4 is higher as the acceleration in the traveling direction of the vehicle 4 becomes greater.

[0056] As described above, the information processing system 100B according to the second embodiment includes the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the output sound control unit 50. Since the engine sound output from the user terminal 2 changes in accordance with the actual behavior (acceleration, engine speed) of the vehicle 4, the user can get a sense of realism as if he or she were actually riding in the vehicle 4 even when outside the vehicle 4.

[0057] Furthermore, the behavior information includes at least one of acceleration information related to the acceleration of the vehicle and rotation speed information related to the rotation speed of the engine of the vehicle, whereby information related to the engine of the vehicle 4 can be easily acquired.

[0058] Furthermore, the output sound control unit changes the engine sound in accordance with at least one of the acceleration information and the rotation speed information, thereby making it possible to effectively reproduce the engine sound of the actual vehicle 4 on the user terminal 2.

[0059] Furthermore, the information processing system 100B according to the second embodiment further includes a vehicle type information acquisition unit 60. This makes it possible to select and play an engine sound file corresponding to the vehicle type information, and effectively reproduce the engine sound of the actual vehicle 4 on the user terminal 2. Therefore, the user can more effectively obtain a sense of realism as if he or she were actually riding in the vehicle 4 even when outside the vehicle 4.

[0060] <Third embodiment> 7 is a block diagram for explaining an information processing system 100C according to a third embodiment. The information processing device 1C according to the third embodiment is different from the first embodiment in that it does not include a vibration control unit 40, and some of the functions of the display control unit 30 are different from those of the first embodiment.

[0061] The display control unit 30 according to the third embodiment uses behavior information to control the display mode of an image 11 (see FIG. 2) to be displayed on the user terminal 2. The behavior information according to the third embodiment includes at least one of speed information related to the speed of the vehicle 4, acceleration information related to the acceleration of the vehicle 4, rotation speed information related to the rotation speed of the engine of the vehicle 4, and gear shift information related to the state of the transmission of the vehicle 4. The acceleration of the vehicle 4 includes at least one of the acceleration in the traveling direction of the vehicle 4, the acceleration in the width direction (lateral direction) of the vehicle 4, and the acceleration of the vehicle 4 in the vertical direction. A specific example of controlling the display mode will be described below.

[0062] <Display mode control example 1: Adding behavior status information> (Behavior information example 1) Fig. 8 is a diagram showing a first behavior state information example added to an image 11 based on video information. Reference numeral 14 in Fig. 8 denotes a radial object 14 suggesting acceleration of the vehicle 4. In Fig. 8, a plurality of radial objects 14 are added to the image 11 based on video information. The display control unit 30 according to the third embodiment may add a plurality of radial objects 14 to the image 11 using behavior information that the vehicle 4 is accelerating in the traveling direction, and control the display mode of the image 11 displayed on the user terminal 2.

[0063] In other words, the display control unit 30 may control the display mode of the image 11 displayed on the user terminal 2 by adding behavior state information to the image 11 based on the video information and displaying the image. The behavior state information is information suggesting the behavior state of the vehicle 4. The behavior state information includes the radial object 14. The radial object 14 suggests the state in which the vehicle 4 accelerates in the traveling direction. For example, when the imaging device 3 is installed on the vehicle 4 so that the imaging direction of the imaging device 3 faces the front of the vehicle 4, the radial object 14 may be a line segment drawn with the vicinity of the center of the image 11 as the center point of radiation. By adding the radial object 14, the user can experience the state in which the vehicle 4 accelerates even when outside the vehicle 4.

[0064] Furthermore, the vehicle 4 or the imaging device 3 of the information processing system 100C according to the third embodiment may transmit gear shift information related to the state of the transmission of the vehicle 4 to the information processing device 1C in addition to or instead of the speed information related to the speed of the vehicle 4, the acceleration information related to the acceleration of the vehicle 4, and the rotation speed information related to the rotation speed of the engine of the vehicle 4. The imaging device 3 may transmit the above-mentioned gear shift information acquired via an OBD (On board diagnostics) interface of the vehicle 4 as behavior information to the information processing device 1C, and the behavior information acquisition unit 20 may acquire the behavior information transmitted from the imaging device 3. The display control unit 30 may change the behavior state information using at least one of the speed information related to the speed of the vehicle 4, the acceleration information related to the acceleration of the vehicle 4, the rotation speed information related to the rotation speed of the engine of the vehicle 4, and the gear shift information related to the state of the transmission of the vehicle 4. An example of changing the behavior state information will be described below.

[0065] (Example of behavior information 2) FIG. 9 is a diagram showing a behavior state information example 2 added to an image 11 based on video information. In FIG. 9, the number of radial objects 14 is greater than that in FIG. 8, and the length of the radial objects 14 is longer. The state in FIG. 9 shows a state in which the acceleration (or speed) of the vehicle 4 is greater than that in FIG. 8. The display control unit 30 may change the radial objects 14 as behavior state information, for example, by increasing the number of radial objects 14 or lengthening the length of the radial objects 14 to show the state in which the vehicle 4 gradually accelerates. In addition, for example, the color of the radial objects 14 may be changed according to the acceleration of the vehicle 4 (e.g., by turning the color dark red as the acceleration increases). In addition, instead of changing the display mode of the radial objects 14 according to the acceleration information of the vehicle 4, the display control unit 30 may consider that the vehicle 4 is accelerating when the change amount of the engine speed of the vehicle 4 within a predetermined time is positive, and change the display mode of the radial objects 14 according to the change amount. In other words, if the engine speed of the vehicle 4 increases over time, it may be assumed that the acceleration of the vehicle 4 in the direction of travel has increased, and the length of the radial object 14 may be increased or the color may be changed.

[0066] (Example 3 of Behavior Information) FIG. 10 is a diagram showing a behavior state information example 3 added to an image 11 based on video information. The radial object 14 in FIG. 8 (or FIG. 9) extends toward the center of the image 11, but the radial object 14 in FIG. 10 extends toward a portion eccentric to the right of the center of the image 11. This makes it possible to express the acceleration in the width direction (lateral direction) of the vehicle 4. In the example of FIG. 10, since the vehicle 4 is approaching a right curve, the vehicle 4 accelerates in the lateral direction of the vehicle 4 toward the center of the arc related to the right curve. In the example of FIG. 10, the radial object 14 is eccentric to the right with respect to the center of the image 11 to express the acceleration of the vehicle 4 in the central direction of the arc. In addition, the display control unit 30 may increase or decrease the amount of eccentricity of the radial object 14 according to the magnitude of the acceleration in the lateral direction of the vehicle 4. In other words, the display control unit 30 moves the center point of radiation of the radial object 14 in the image 11 to the left or right depending on the direction of the lateral acceleration of the vehicle 4. For example, the greater the acceleration of the vehicle 4 toward the left, the greater the center point of radiation of the radial object 14 in the image 11 is moved to the right, thereby drawing the radial object 14.

[0067] (Behavior Information Example 4) FIG. 11 is a diagram showing a fourth example of behavior state information added to an image 11 based on video information. Reference numeral 15 in FIG. 11 denotes a polygonal line object 15 suggesting vibration of the vehicle 4. In FIG. 11, a plurality of polygonal line objects 15 are added to the image 11 based on video information. The display control unit 30 may add a plurality of polygonal line objects 15 to the image 11 using behavior information that the vehicle 4 is accelerating in the vertical direction (the vehicle 4 is vibrating) to control the display mode of the image 11. The display control unit 30 may also increase or decrease the number of polygonal lines of the polygonal line object 15 depending on the magnitude of the acceleration of the vehicle 4 in the vertical direction.

[0068] (Behavior Information Example 5) FIG. 12 is a diagram showing a behavior state information example 5 added to an image 11 based on video information. Reference numeral 16 in FIG. 12 denotes gear shift text information 16 that suggests the state of the transmission of the vehicle 4. In FIG. 12, the gear shift text information 16 is added to an image 11 based on video information. The display control unit 30 may add the gear shift text information 16 to the image 11 using behavior information that the driver has shifted up a gear (gear shift information related to the state of the transmission of the vehicle 4) to control the display mode of the image 11. Furthermore, in addition to adding the gear shift text information 16 to the image 11, the display control unit 30 may add a radial object 14.

[0069] <Display mode control example 2: Blur processing> (Blur processing example 1) Fig. 13 is a diagram showing a first example of blurring applied to an image 11 based on video information. The display control unit 30 according to the third embodiment may use the speed information to blur an area 17 along an edge E of the image 11, thereby controlling the display mode of the image 11 to be displayed on the user terminal 2. It is generally known that the faster the speed of the vehicle 4, the narrower the field of view of the passenger (driver). In Fig. 13, the blurring is applied to the area 17 along the edge E of the image 11, and the area C near the center is displayed normally, thereby expressing a state in which the field of view of the passenger (driver) is narrowed.

[0070] (Blur processing example 2) Fig. 14 is a diagram showing a second example of blurring processing applied to an image 11 based on video information. In Fig. 14, the blurred range is narrower than in Fig. 13 (the size of the area C near the center is larger than in Fig. 13). The state in Fig. 14 shows a state in which the speed of the vehicle 4 is slower than that in Fig. 13. That is, the display control unit 30 may adjust the blurred range so that the faster the speed of the vehicle 4 is, the larger the blurred range becomes.

[0071] <Display mode control example 3: Rotating image 11> Fig. 15 is a diagram showing a state in which the image 11 based on the video information is rotated. In Fig. 15, the image 11 is rotated counterclockwise and tilted, as compared with Fig. 14 and the like. The display control unit 30 may control the display mode of the image 11 by rotating the image 11 using acceleration information. The display control unit 30 may rotate the image 11, for example, using acceleration information related to the acceleration of the vehicle 4 in the lateral direction.

[0072] Generally, when a vehicle is traveling around a curve, the vehicle occupant tilts his / her head in the opposite direction to the centrifugal force to resist the centrifugal force. More specifically, when traveling around a right curve, a centrifugal force is generated in the vehicle 4 in the left direction, and the occupant of the vehicle 4 tilts his / her head in the right direction opposite to the direction of the centrifugal force. FIG. 15 shows a scene viewed when the occupant tilts his / her head. In this manner, the display control unit 30 may rotate the image 11 using acceleration information related to the acceleration of the vehicle 4 in the lateral direction. In FIG. 15, the end of the image 11 is displayed on the display of the user terminal 2, but the image 11 may be enlarged in advance and displayed on the display of the user terminal 2 so that the end of the image 11 is not displayed on the display of the user terminal 2 even when the image 11 is rotated.

[0073] (Operation example of the third embodiment) 16 is a flow diagram showing the processing of the information processing system 100C according to the third embodiment. Steps S100 to S300 are similar to those in the first embodiment. In step S520 after step S300, the display control unit 30 controls the display mode of the image 11 by using the behavior information.

[0074] For example, in step S300, the behavior information acquisition unit 20 acquires, as behavior information, acceleration information related to the acceleration of the vehicle 4 in the lateral direction of the vehicle 4 (the behavior information acquisition unit 20 acquires acceleration information in the right direction (the direction of the center of the above-mentioned arc) with respect to the traveling direction of the vehicle 4). In this case, in step S520, the display control unit 30 may control the display mode of the image 11 by rotating the image 11 counterclockwise using the behavior information as the acceleration information of the vehicle 4 in the right direction.

[0075] In addition, the process of controlling the display mode of the image 11 using the behavior information is as described above.

[0076] As described above, the information processing system 100C according to the third embodiment includes the video information acquisition unit 10, the behavior information acquisition unit 20, and the display control unit 30, and the display control unit 30 uses the behavior information to control the display mode of the image 11 to be displayed on the user terminal 2. Since the display mode of the image 11 displayed on the user terminal 2 changes according to the actual behavior of the vehicle 4 (speed, acceleration, engine RPM, transmission state, etc.), the user can get a sense of realism as if he or she is actually riding in the vehicle 4 even when outside the vehicle 4.

[0077] Furthermore, the behavior information includes at least one of speed information related to the speed of the vehicle 4, acceleration information related to the acceleration of the vehicle 4, rotation speed information related to the rotation speed of the engine of the vehicle 4, and gear shift information related to the state of the transmission of the vehicle 4. This makes it possible to easily obtain information related to the behavior of the vehicle 4.

[0078] Furthermore, the display control unit 30 controls the display mode of the image 11 displayed on the user terminal 2 by adding the behavior state information to the image 11 and displaying it. The display control unit 30 also changes the behavior state information using at least one of the speed information, the acceleration information, the number of rotations information, and the gear shift information. This allows the user to more effectively obtain a sense of realism as if he or she were actually riding in the vehicle 4 even when the user is outside the vehicle 4.

[0079] Furthermore, the display control unit 30 controls the display mode of the image 11 by applying blurring to an area along the edge of the image 11 using at least one of the speed information, the acceleration information, and the rotation speed information. This makes it possible to express the size of the occupant's visual field on the user terminal 2.

[0080] Furthermore, the display control unit 30 uses the acceleration information to rotate the image 11, thereby controlling the display mode of the image 11. This makes it possible to present the scenery visually recognized by the passenger on the user terminal 2.

[0081] Although the embodiments have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0082] In the first embodiment, the information processing device 1A has all of the functions related to the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the vibration control unit 40, but some or all of these functions may be mounted on the user terminal 2. For example, in the first embodiment, the user terminal 2 may have the video information acquisition unit 10. The same applies to the second and third embodiments.

[0083] In the first embodiment, the information processing device 1A may further include an output sound control unit 50. The functions and effects of the output sound control unit 50 according to the first embodiment are as described in the second embodiment. In the second embodiment, the information processing device 1B may further include a vibration control unit 40. The functions and effects of the vibration control unit 40 according to the second embodiment are as described in the first embodiment. In the third embodiment, the information processing device 1C may further include at least one of the vibration control unit 40 and the output sound control unit 50. The functions and effects of the vibration control unit 40 and the output sound control unit 50 according to the third embodiment are as described in the first and second embodiments.

[0084] Below, examples of reference forms are given. 1. A video information acquisition unit that acquires video information related to a video captured by an imaging device that moves together with a vehicle; a behavior information acquisition unit that acquires behavior information regarding a behavior of the vehicle when the imaging device captures the video; a display control unit that displays an image based on the video information on a user terminal located outside the vehicle; and an output sound control unit that uses the behavior information to control a sound to be output by the user terminal. 2. In the information processing system according to 1., The behavior information includes at least one of acceleration information related to the acceleration of the vehicle and rotation speed information related to the rotation speed of an engine of the vehicle. 3. In the information processing system according to 2., The output sound control unit outputs an engine sound as the sound to the user terminal. 4. In the information processing system according to 3., The output sound control unit changes the engine sound in accordance with at least one of the acceleration information and the rotation speed information. 5. In the information processing system according to any one of 1. to 4., A vehicle model information acquisition unit that acquires vehicle model information relating to the vehicle model, The output sound control unit controls the sound using the vehicle model information. 6. In the information processing system according to any one of 1. to 5., The information processing system further includes a vibration control unit that controls vibration of the user terminal using the behavior information acquired by the behavior information acquisition unit. 7. The computer: acquiring video information relating to a video captured by an imaging device moving together with the vehicle; acquiring behavior information regarding a behavior of the vehicle when the imaging device captures the video; Displaying an image based on the video information on a user terminal located outside the vehicle; A control method for an information processing system, comprising: controlling a sound to be output from the user terminal using the behavior information. 8. On your computer: acquiring video information relating to a video captured by an imaging device moving with the vehicle; acquiring behavior information regarding a behavior of the vehicle when the imaging device captures the video; a step of displaying an image based on the video information on a user terminal located outside the vehicle; A program for executing a procedure for controlling a sound to be output by the user terminal using the behavior information. [Explanation of symbols]

[0085] 1A, 1B, 1C Information processing device 2. User terminal 3. Imaging device 4. Vehicles 10 Video information acquisition section 20 Behavior information acquisition unit 30 Display control unit 40 Vibration control section 50 Output sound control section 60 Vehicle information acquisition section 100A, 100B, 100C Information Processing Systems 101 Communication Network 1010 Bus 1020 Processor 1030 Memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface

Claims

1. a video information acquisition unit that acquires video information related to a video captured by an imaging device that moves together with the vehicle; a behavior information acquisition unit that acquires behavior information regarding a behavior of the vehicle when the imaging device captures the video; a display control unit that displays an image based on the video information on a user terminal located outside the vehicle; and an output sound control unit that uses the behavior information to control a sound to be output by the user terminal.

2. 2. The information processing system according to claim 1, The behavior information includes at least one of acceleration information related to the acceleration of the vehicle and revolution speed information related to the revolution speed of an engine of the vehicle.

3. 3. The information processing system according to claim 2, The output sound control unit outputs an engine sound as the sound to the user terminal.

4. 4. The information processing system according to claim 3, The output sound control unit changes the engine sound in accordance with at least one of the acceleration information and the rotation speed information.

5. 5. The information processing system according to claim 1, A vehicle model information acquisition unit that acquires vehicle model information relating to the vehicle model, The output sound control unit controls the sound using the vehicle model information.

6. 5. The information processing system according to claim 1, The information processing system further includes a vibration control unit that controls vibration of the user terminal using the behavior information acquired by the behavior information acquisition unit.

7. The computer acquiring video information relating to a video captured by an imaging device moving together with the vehicle; acquiring behavior information regarding a behavior of the vehicle when the imaging device captures the video; Displaying an image based on the video information on a user terminal located outside the vehicle; A control method for an information processing system, comprising: controlling a sound to be output from the user terminal using the behavior information.

8. On the computer, acquiring video information relating to a video captured by an imaging device moving with the vehicle; acquiring behavior information regarding a behavior of the vehicle when the imaging device captures the video; a step of displaying an image based on the video information on a user terminal located outside the vehicle; A program for executing a procedure for controlling a sound to be output by the user terminal using the behavior information.

Images