Information processing system, information processing system control method, and program

Abstract

A video information acquisition unit (10) acquires video information pertaining to a video captured by an imaging device (3) that moves together with a vehicle (4). A behavior information acquisition unit (20) acquires behavior information pertaining to the behavior of the vehicle (4) when the imaging device (3) captured the video. A display control unit (30) causes a user terminal (2) located outside the vehicle (4) to display an image based on the video information. The display control unit (30) uses the behavior information to control the display appearance of the image that the user terminal (2) is made to display.

Description

Information processing system, control method for information processing system, and program

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program.

[0002] Patent Document 1 discloses a technology for displaying an image of the area in front of a vehicle on an external terminal.

[0003] JP 2016-213791 A

[0004] In a case such as that of Patent Document 1, the vehicle information that a user outside the vehicle can obtain is limited to images of the vehicle, audio from inside the vehicle, and vehicle location information, making it difficult for the user outside the vehicle to feel as if they are actually riding in the vehicle.

[0005] One example of a problem to be solved by the present invention is to provide an information processing system that allows a user to feel as if they are inside a vehicle even when they are outside the vehicle.

[0006] The invention described in claim 1 is an information processing system comprising: a video information acquisition unit that acquires video information related to a video captured by an imaging device that moves with a vehicle; a behavior information acquisition unit that acquires behavior information related to the behavior of the vehicle when the imaging device captured the video; and a display control unit that displays an image based on the video information on a user terminal located outside the vehicle, wherein the display control unit uses the behavior information to control the display mode of the image to be displayed on the user terminal.

[0007] The invention described in claim 9 is a control method for an information processing system, in which a computer acquires video information regarding a video captured by an imaging device moving with a vehicle, acquires behavior information regarding the behavior of the vehicle when the imaging device captured the video, displays an image based on the video information on a user terminal located outside the vehicle, and uses the behavior information to control the display mode of the image displayed on the user terminal.

[0008] The invention described in claim 10 is a program for causing a computer to execute the following steps: acquiring video information regarding a video captured by an imaging device moving with a vehicle; acquiring behavior information regarding the behavior of the vehicle when the imaging device captured the video; displaying an image based on the video information on a user terminal located outside the vehicle; and using the behavior information to control the display mode of the image displayed on the user terminal.

[0009] 1 is a block diagram for explaining an information processing system according to a first embodiment; FIG. 2 is a diagram showing an image based on video information displayed on a user terminal; FIG. 3 is a diagram showing an example of the hardware configuration of an information processing device; FIG. 4 is a flow diagram showing the processing of the information processing system according to the first embodiment; FIG. 5 is a block diagram for explaining an information processing system according to a second embodiment; FIG. 6 is a flow diagram showing the processing of the information processing system according to the second embodiment; FIG. 7 is a block diagram for explaining an information processing system according to a third embodiment; FIG. 8 is a diagram showing example 1 of behavior state information added to an image based on video information; FIG. 9 is a diagram showing example 2 of behavior state information added to an image based on video information; FIG. 10 is a diagram showing example 3 of behavior state information added to an image based on video information; FIG. 11 is a diagram showing example 4 of behavior state information added to an image based on video information; FIG. 12 is a diagram showing example 5 of behavior state information added to an image based on video information; FIG. 13 is a diagram showing example 1 of blurring applied to an image based on video information; FIG. 14 is a diagram showing example 2 of blurring applied to an image based on video information; FIG. 15 is a diagram showing a rotated state of an image based on video information; and FIG. 16 is a flow diagram showing the processing of an information processing system according to a third embodiment.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0011] In the following description, each component of each device represents a functional block, not a hardware configuration. Each component of each device is realized by any combination of hardware and software, centered around the CPU of any computer, memory, a program loaded into the memory, a storage medium such as a hard disk that stores the program, and a network connection interface. There are many variations in the realization method and device.

[0012] <First embodiment> (Information processing system 100A) Fig. 1 is a block diagram for explaining an information processing system 100A according to the first embodiment. The information processing system 100A according to the first embodiment includes an information processing device 1A, a user terminal 2, and an imaging device 3. In the first embodiment, the information processing device 1A is provided outside a vehicle 4. The information processing device 1A may be a so-called cloud server.

[0013] The information processing device 1A is configured to be able to communicate with a user terminal 2 (described later) and an imaging device 3 (described later) via a communication network 101. The communication network 101 includes, for example, communication over a 4G or 5G line. The information processing device 1A is configured to include an external vehicle communication unit (not shown) for configuring the communication network 101.

[0014] (Imaging device 3) The vehicle 4 is equipped with the imaging device 3. In the first embodiment, the imaging device 3 is located inside the vehicle 4. However, the imaging device 3 may be located outside the vehicle 4. The imaging device 3 may be brought in from outside the vehicle 4 or may be attached to the vehicle 4 in advance. The imaging device 3 moves together with the vehicle 4.

[0015] In the first embodiment, the imaging device 3 captures an image in front of (in the direction of travel of) the vehicle 4. The imaging device 3 may also be capable of capturing an image in the width direction (lateral direction) of the vehicle 4 or behind the vehicle 4. The imaging device 3 generates video information related to the captured video. The imaging device 3 may also include a sound collection unit (not shown), generate audio information related to audio inside the vehicle 4 acquired by the sound collection unit, and transmit the audio information to the information processing device 1A. The imaging device 3 may also transmit behavior information (described later) related to the behavior of the vehicle 4 to the information processing device 1A.

[0016] The imaging device 3 is configured to be able to communicate with the information processing device 1A via a communication network 101. The communication network 101 includes, for example, communication over a 4G or 5G line. The imaging device 3 is configured to include an external vehicle communication unit (not shown) for configuring the communication network 101. Note that the imaging device 3 and the information processing device 1A may communicate using WebRTC (Web Real-Time Communication).

[0017] (User Terminal 2) The user terminal 2 is located outside the vehicle 4. The user terminal 2 is a terminal of a user outside the vehicle 4, and is a terminal that is owned, managed, carried, or used by a user outside the vehicle 4. The user terminal 2 includes any one of a smartphone, a tablet, and a PC (personal computer).

[0018] An application program for the user, which is used in the information processing system 100A, is installed on the user terminal 2. The user uses the application by starting the application on the user terminal 2. The application will be described in detail later.

[0019] The user terminal 2 is configured to be able to communicate with the information processing device 1A via a communication network 101. The communication network 101 includes, for example, communication over a 4G or 5G line. The user terminal 2 is configured to include an external vehicle communication unit (not shown) for configuring the communication network 101. Note that the user terminal 2 and the information processing device 1A may communicate using WebRTC (Web Real-Time Communication).

[0020] (Video information acquisition unit 10) The information processing device 1A according to the first embodiment includes a video information acquisition unit 10, a behavior information acquisition unit 20, a display control unit 30, and a vibration control unit 40. The video information acquisition unit 10 acquires video information related to videos captured by the imaging device 3. In the first embodiment, the video information includes, for example, a video captured in the forward direction (travel direction) of the vehicle 4.

[0021] (Display control unit 30) The display control unit 30 causes the user terminal 2 to display an image based on the video information. The "image based on video information" may be a video. The "image based on video information" may include a video captured by the imaging device 3 with a reduced frame rate.

[0022] FIG. 2 is a diagram showing an image 11 based on video information displayed on the user terminal 2. In FIG. 2, the imaging device 3 captures an image of the area ahead of the vehicle 4, and the display control unit 30 displays the video captured by the imaging device 3 on the display of the user terminal 2 in real time. Note that real time does not necessarily mean complete simultaneousness, and there may be a time lag of several seconds to several tens of seconds. In FIG. 2, the display control unit 30 displays a video of the vehicle 4 traveling through a tunnel approaching a right curve on the user terminal 2. Reference numeral 12 in FIG. 2 indicates a road 12 with one lane in each direction, and reference numeral 13 in FIG. 2 indicates a lane marking 13. Note that, in addition to the video captured by the imaging device 3, the display of the user terminal 2 may simultaneously display map information indicating the current position of the vehicle 4, speed information indicating the current traveling speed of the vehicle 4, and the like.

[0023] When the user starts a user application from the user terminal 2, the user terminal 2 can perform the function of displaying an image 11 based on video information on the user terminal 2.

[0024] (Behavior information acquisition unit 20) Returning to Fig. 1 , the behavior information acquisition unit 20 acquires behavior information transmitted from the vehicle 4. The behavior information is information related to the behavior of the vehicle 4 when the imaging device 3 captures a video. The behavior information is information related to the behavior of the vehicle 4 when the imaging device 3 is capturing a video. The behavior information is information related to the behavior of the vehicle 4 when an image 11 (see Fig. 2 ) based on the video information is displayed in real time on the user terminal 2.

[0025] The behavior information includes at least one of acceleration information related to the acceleration of the vehicle 4 and rotation speed information related to the rotation speed of the engine of the vehicle 4. The acceleration of the vehicle 4 includes at least one of acceleration in the traveling direction of the vehicle 4, acceleration in the width direction (lateral direction) of the vehicle 4, and acceleration of the vehicle 4 in the vertical direction. The acceleration information may be generated by a three-axis acceleration sensor (not shown) mounted on the vehicle 4 or the imaging device 3. The engine rotation speed may also be generated by an engine rotation speed sensor (not shown) mounted on the vehicle 4. The imaging device 3 may transmit the acceleration information and the rotation speed information acquired via an OBD (On Board Diagnostics) interface of the vehicle 4 to the information processing device 1A, and the behavior information acquisition unit 20 may acquire the behavior information transmitted from the imaging device 3.

[0026] (Vibration control unit 40) The vibration control unit 40 uses the behavior information to control the vibration of the user terminal 2. The vibration control unit 40 controls the vibration of the user terminal 2 by adjusting at least one of the interval at which the user terminal 2 vibrates and the duration of vibration of the user terminal 2.

[0027] In the first embodiment, the vibration control unit 40 may control the vibration of the user terminal 2 so that the vibration interval of the user terminal 2 becomes shorter as the acceleration of the vehicle 4 increases or the engine rotation speed of the vehicle 4 increases. Furthermore, the vibration control unit 40 may control the vibration of the user terminal 2 so that the vibration duration of the user terminal 2 becomes longer as the acceleration of the vehicle 4 increases or the engine rotation speed of the vehicle 4 increases. Furthermore, for example, when the engine rotation speed is low, the vibration interval may be set to a longer pulse interval, and the pulse interval may be shortened as the engine rotation speed increases.

[0028] 3 is a diagram showing an example of the hardware configuration of the information processing device 1 A. The information processing device 1 A has a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0029] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0030] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0031] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0032] The storage device 1040 is an auxiliary storage device realized by removable media such as a hard disk drive (HDD), a solid state drive (SSD), or a memory card, or a read-only memory (ROM), and has a recording medium. The recording medium of the storage device 1040 stores program modules that realize each function of the information processing device 1A (e.g., the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the vibration control unit 40). The processor 1020 loads and executes each of these program modules into the memory 1030, thereby realizing each function corresponding to the program module. The storage device 1040 may also function as a memory unit (not shown) included in the information processing device 1A.

[0033] The input / output interface 1050 is an interface for connecting the information processing device 1A to various input / output devices.

[0034] The network interface 1060 is an interface for connecting the information processing device 1A to a network. This network is, for example, a local area network (LAN) or a wide area network (WAN). The network interface 1060 may connect to the network wirelessly or by wire. The information processing device 1A may communicate with the user terminal 2 and the imaging device 3 via the network interface 1060.

[0035] 4 is a flow diagram showing the processing of the information processing system 100A according to the first embodiment. First, in step S100, the video information acquisition unit 10 acquires video information captured by the imaging device 3. In the first embodiment, the imaging device 3 captures an image ahead of the vehicle 4, and the video information acquisition unit 10 acquires video information relating to the image ahead of the vehicle 4.

[0036] Next, in step S200, the display control unit 30 displays, in real time, an image 11 (see FIG. 2 ) based on video information captured by the imaging device 3 on the user terminal 2. This allows the user to share the view of the scenery ahead from the vehicle 4 with the driver even when the user is outside the vehicle 4.

[0037] Next, in step S300, the behavior information acquisition unit 20 acquires behavior information. For example, the behavior information acquisition unit 20 acquires the vertical acceleration of the vehicle 4 or the engine speed as behavior information transmitted from the vehicle 4.

[0038] In step S400, the behavior information acquisition unit 20 determines whether the behavior information acquired in step S300 exceeds a predetermined standard. For example, if the acquired behavior information is the acceleration of the vehicle 4 in the vertical direction, the threshold value of the acceleration is 3 [m / s ２ ], and the acceleration of the vehicle 4 in the vertical direction as the behavior information acquired in step S300 is 7 [m / s ２ ], the behavior information acquisition unit 20 may determine that the behavior information acquired in step S300 has exceeded the predetermined standard. Furthermore, if the acquired behavior information is the engine speed of the vehicle 4, the threshold value of the engine speed is 4000 [rpm], and if the engine speed of the vehicle 4 as the behavior information acquired in step S300 is 4500 [rpm], the behavior information acquisition unit 20 may determine that the behavior information acquired in step S300 has exceeded the predetermined standard.

[0039] If the behavior information acquired in step S300 exceeds the predetermined standard (YES in step S400), the process proceeds to step S500. If the behavior information acquired in step S300 does not exceed the predetermined standard (NO in step S400), the process may end. Note that step S400 may be omitted.

[0040] In step S500, the vibration control unit 40 controls the vibration of the user terminal 2. The vibration control unit 40 may control the vibration function of the user terminal 2. The vibration control unit 40 may control at least one of the vibration interval when the user terminal 2 is intermittently vibrated and the vibration duration when the user terminal 2 is continuously vibrated, in accordance with the magnitude of the acceleration of the vehicle 4 in the vertical direction. The vibration control unit 40 may shorten the vibration interval of the user terminal 2 or lengthen the vibration duration of the user terminal 2, as the acceleration of the vehicle 4 in the vertical direction increases. Alternatively, in step S500, the vibration control unit 40 may control at least one of the vibration interval when the user terminal 2 is intermittently vibrated and the vibration duration when the user terminal 2 is continuously vibrated, in accordance with the engine rotation speed of the vehicle 4. The vibration control unit 40 may shorten the vibration interval of the user terminal 2 or lengthen the vibration duration of the user terminal 2, as the engine rotation speed of the vehicle 4 increases.

[0041] As described above, the information processing system 100A according to the first embodiment includes the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the vibration control unit 40. The user terminal 2 vibrates in accordance with the behavior (acceleration, engine speed) of the actual vehicle 4, so the user can virtually experience the vibrations of the vehicle 4 as if they were actually riding in the vehicle 4. Therefore, even when the user is outside the vehicle 4, the user can get a sense of realism as if they were actually riding in the vehicle 4.

[0042] Furthermore, the behavior information includes at least one of acceleration information relating to the acceleration of the vehicle and rotation speed information relating to the rotation speed of the engine of the vehicle, thereby making it possible to easily obtain information relating to the vibration of the vehicle 4.

[0043] Furthermore, the acceleration of the vehicle 4 includes the acceleration of the vehicle 4 in the vertical direction. As a result, for example, when the vehicle 4 vibrates in the vertical direction at a bump in the road, the user can virtually experience the vibration caused by the bump.

[0044] Furthermore, the vibration control unit 40 controls the vibration of the user terminal 2 by adjusting at least one of the interval at which the user terminal 2 is vibrated and the duration of the vibration of the user terminal 2. This allows the vibration of an actual vehicle 4 to be effectively reproduced on the user terminal 2.

[0045] 5 is a block diagram illustrating an information processing system 100B according to a second embodiment. The information processing system 100B according to the second embodiment does not include a vibration control unit 40, but includes an output sound control unit 50 and a vehicle model information acquisition unit 60. Functions other than the output sound control unit 50 and the vehicle model information acquisition unit 60 are the same as those in the first embodiment.

[0046] (Output sound control unit 50) The output sound control unit 50 uses behavior information to control the sound to be output by the user terminal 2. In the second embodiment, the sound is an engine sound. The engine sound may be a pseudo engine sound that imitates the actual engine sound of the vehicle 4. The engine sound may be stored in advance in a storage unit (not shown) as an engine audio file. The output sound control unit 50 according to the second embodiment causes the user terminal 2 to output the engine sound as the sound.

[0047] The output sound control unit 50 changes the engine sound in accordance with at least one of acceleration information relating to the acceleration in the traveling direction of the vehicle 4 and rotation speed information relating to the rotation speed of the engine of the vehicle 4 .

[0048] In the second embodiment, for example, the output sound control unit 50 may control the engine sound to be output by the user terminal 2 so that the playback speed of the engine sound to be output by the user terminal 2 increases as the rotation speed of the engine of the vehicle 4 increases. Alternatively, the output sound control unit 50 may control the engine sound to be output by the user terminal 2 so that the playback speed of the engine sound to be output by the user terminal 2 increases as the acceleration of the vehicle 4 in the traveling direction increases, assuming that the rotation speed of the engine of the vehicle 4 is increasing in a scene where the acceleration of the vehicle 4 in the traveling direction is large. By increasing the playback speed of the engine sound in accordance with the rotation speed of the engine of the vehicle 4 or the acceleration of the vehicle 4 in the traveling direction, it is possible to reproduce the high-speed rotation sound of an actual engine of the vehicle 4. Therefore, for example, the user can experience the sensation of the vehicle 4 accelerating.

[0049] (Vehicle model information acquisition unit 60) The vehicle model information acquisition unit 60 acquires vehicle model information related to the vehicle model of the vehicle 4. The vehicle model information includes information about the vehicle model (model) of the vehicle 4. The vehicle model information may include, for example, information about the vehicle classification, such as a passenger car or a light vehicle. The vehicle model information may include, for example, information about the vehicle category, such as a sports car, a sedan, or a van, of the vehicle 4. The vehicle model information may also include, for example, a specific manufacturer name (e.g., Ferrari (registered trademark), Lamborghini (registered trademark), etc.) or a product name (e.g., Ferrari 458).

[0050] The output sound control unit 50 controls the sound using the vehicle model information. Specifically, for example, when the vehicle model information acquisition unit 60 acquires information related to a product name (e.g., Ferrari 458) as vehicle model information, the output sound control unit 50 selects an engine sound file corresponding to the product name and outputs the engine sound to the user terminal 2.

[0051] The hardware configuration of the information processing apparatus 1A according to the second embodiment is the same as that of the first embodiment.

[0052] 6 is a flowchart showing the processing of the information processing system 100B according to the second embodiment. Steps S100 to S300 are the same as those in the first embodiment.

[0053] In step S350 after step S300, the vehicle model information acquisition unit 60 acquires vehicle model information. In the second embodiment, for example, it is assumed that the vehicle model information acquisition unit 60 acquires information on the product name of the vehicle 4 (e.g., Ferrari 458) as the vehicle model information.

[0054] In step S510 after step S350, the output sound control unit 50 uses the behavior information and vehicle model information to control the sound to be output by the user terminal 2. In step S510, the output sound control unit 50 uses the product name of the vehicle 4 (e.g., Ferrari 458) as the vehicle model information acquired in step S350 to acquire an engine audio file corresponding to Ferrari 458 from a storage unit (not shown), and causes the user terminal 2 to output the engine sound stored in the audio file. Then, using rotation speed information related to the engine rotation speed of the vehicle 4 as the behavior information acquired in step S300, the output sound control unit 50 plays the audio file at a playback speed corresponding to the engine rotation speed of the vehicle 4. In the second embodiment, for example, the output sound control unit 50 may control the engine sound to be output by the user terminal 2 so that the playback speed of the engine sound to be output by the user terminal 2 increases as the engine rotation speed of the vehicle 4 increases.

[0055] Alternatively, in step S510, instead of controlling the playback speed of the engine audio file using rotation speed information related to the rotation speed of the engine of the vehicle 4 as the behavior information acquired in step S300, the output sound control unit 50 may control the playback speed of the engine audio file using acceleration information related to the acceleration in the traveling direction of the vehicle 4 as the behavior information acquired in step S300. In other words, the output sound control unit 50 may consider that the greater the acceleration in the traveling direction of the vehicle 4, the higher the engine rotation speed of the vehicle 4, and control the engine sound to be output by the user terminal 2 so that the playback speed of the engine sound to be output by the user terminal 2 becomes faster as the acceleration in the traveling direction of the vehicle 4 becomes greater.

[0056] As described above, the information processing system 100B according to the second embodiment includes the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the output sound control unit 50. Since the engine sound output from the user terminal 2 changes in accordance with the behavior (acceleration, engine speed) of the actual vehicle 4, the user can get a sense of realism as if they were actually riding in the vehicle 4 even when they are outside the vehicle 4.

[0057] Furthermore, the behavior information includes at least one of acceleration information relating to the acceleration of the vehicle and rotation speed information relating to the rotation speed of the engine of the vehicle, thereby making it possible to easily obtain information relating to the engine of the vehicle 4.

[0058] Furthermore, the output sound control unit changes the engine sound in accordance with at least one of the acceleration information and the rotation speed information, thereby enabling the engine sound of the actual vehicle 4 to be effectively reproduced on the user terminal 2.

[0059] Furthermore, the information processing system 100B according to the second embodiment further includes a vehicle model information acquisition unit 60. This makes it possible to select and play an engine sound file corresponding to the vehicle model information, and effectively reproduce the engine sound of the actual vehicle 4 on the user terminal 2. This allows the user to more effectively experience a sense of realism as if they were actually in the vehicle 4, even when they are outside the vehicle 4.

[0060] 7 is a block diagram illustrating an information processing system 100C according to a third embodiment. The information processing device 1C according to the third embodiment does not include a vibration control unit 40, unlike the first embodiment, and some of the functions of the display control unit 30 are different from those of the first embodiment.

[0061] The display control unit 30 according to the third embodiment uses behavior information to control the display mode of the image 11 (see FIG. 2) displayed on the user terminal 2. The behavior information according to the third embodiment includes at least one of speed information related to the speed of the vehicle 4, acceleration information related to the acceleration of the vehicle 4, rotation speed information related to the rotation speed of the engine of the vehicle 4, and gear shift information related to the state of the transmission of the vehicle 4. The acceleration of the vehicle 4 includes at least one of acceleration in the traveling direction of the vehicle 4, acceleration in the width direction (lateral direction) of the vehicle 4, and acceleration of the vehicle 4 in the vertical direction. Specific examples of control of the display mode will be described below.

[0062] <Display Mode Control Example 1: Addition of Behavior State Information> (Behavior State Information Example 1) Fig. 8 is a diagram showing behavior state information example 1 added to image 11 based on video information. Reference numeral 14 in Fig. 8 denotes a radial object 14 suggesting acceleration of vehicle 4. In Fig. 8, a plurality of radial objects 14 are added to image 11 based on video information. The display control unit 30 according to the third embodiment may add a plurality of radial objects 14 to image 11 using behavior information that the vehicle 4 is accelerating in the traveling direction, and control the display mode of image 11 displayed on the user terminal 2.

[0063] In other words, the display control unit 30 may control the display mode of the image 11 displayed on the user terminal 2 by adding behavior state information to the image 11 based on the video information and displaying the image 11. The behavior state information is information that suggests the behavior state of the vehicle 4. The behavior state information includes radial objects 14. The radial objects 14 suggest the state in which the vehicle 4 is accelerating in the traveling direction. For example, when the imaging device 3 is installed on the vehicle 4 so that the imaging direction faces directly ahead of the vehicle 4, the radial objects 14 may be line segments drawn with the vicinity of the center of the image 11 as the center point of the radiation. By adding the radial objects 14, the user can experience the state in which the vehicle 4 is accelerating even when outside the vehicle 4.

[0064] Furthermore, the vehicle 4 or the imaging device 3 of the information processing system 100C according to the third embodiment may transmit gear shift information related to the state of the transmission of the vehicle 4 to the information processing device 1C in addition to or instead of speed information related to the speed of the vehicle 4, acceleration information related to the acceleration of the vehicle 4, and rotation speed information related to the rotation speed of the engine of the vehicle 4. The imaging device 3 may transmit the gear shift information acquired via an OBD (On Board Diagnostics) interface of the vehicle 4 to the information processing device 1C as behavior information, and the behavior information acquisition unit 20 may acquire the behavior information transmitted from the imaging device 3. The display control unit 30 may change the behavior state information using at least one of the speed information related to the speed of the vehicle 4, acceleration information related to the acceleration of the vehicle 4, rotation speed information related to the rotation speed of the engine of the vehicle 4, and gear shift information related to the state of the transmission of the vehicle 4. An example of changing the behavior state information will be described below.

[0065] (Behavior State Information Example 2) FIG. 9 is a diagram showing behavior state information example 2 added to an image 11 based on video information. In FIG. 9, the number of radial objects 14 is greater and the length of the radial objects 14 is longer than in FIG. 8. The state in FIG. 9 shows a state in which the acceleration (or speed) of the vehicle 4 is greater than that in FIG. 8. The display control unit 30 may change the radial objects 14 as behavior state information, for example, by increasing the number of radial objects 14 or lengthening the length of the radial objects 14 to show the gradual acceleration of the vehicle 4. Furthermore, for example, the color of the radial objects 14 may be changed according to the acceleration of the vehicle 4 (e.g., by turning the color darker red as the acceleration increases). Furthermore, instead of changing the display mode of the radial objects 14 according to the acceleration information of the vehicle 4, the display control unit 30 may determine that the vehicle 4 is accelerating when the change in the engine speed of the vehicle 4 within a predetermined time is positive, and change the display mode of the radial objects 14 according to the change. In other words, if the engine speed of the vehicle 4 increases over time, it may be assumed that the acceleration of the vehicle 4 in the direction of travel has increased, and the length of the radial object 14 may be increased or the color may be changed.

[0066] (Behavior State Information Example 3) FIG. 10 is a diagram showing behavior state information example 3 added to an image 11 based on video information. While the radial object 14 in FIG. 8 (or FIG. 9) extends toward the center of the image 11, the radial object 14 in FIG. 10 extends toward a portion eccentric to the right of the center of the image 11. This makes it possible to express the acceleration of the vehicle 4 in the width direction (lateral direction). In the example of FIG. 10, the vehicle 4 is approaching a right curve, so the vehicle 4 accelerates in the lateral direction of the vehicle 4, toward the center of the arc of the right curve. In the example of FIG. 10, the radial object 14 is eccentric to the right with respect to the center of the image 11, thereby expressing the acceleration of the vehicle 4 in the center direction of the arc. Furthermore, the display control unit 30 may increase or decrease the amount of eccentricity of the radial object 14 depending on the magnitude of the acceleration of the vehicle 4 in the lateral direction. That is, the display control unit 30 moves the center point of radiation of the radial object 14 in the image 11 left and right depending on the direction of the lateral acceleration of the vehicle 4; for example, the greater the acceleration of the vehicle 4 toward the left, the more the display control unit 30 moves the center point of radiation of the radial object 14 in the image 11 to the right, thereby drawing the radial object 14.

[0067] (Behavior Status Information Example 4) FIG. 11 is a diagram showing behavior status information example 4 added to image 11 based on video information. Reference numeral 15 in FIG. 11 denotes a polygonal line object 15 suggesting vibration of vehicle 4. In FIG. 11 , a plurality of polygonal line objects 15 are added to image 11 based on video information. The display control unit 30 may add a plurality of polygonal line objects 15 to image 11 using behavior information that the vehicle 4 is accelerating in the vertical direction (the vehicle 4 is vibrating), thereby controlling the display mode of image 11. Furthermore, the display control unit 30 may increase or decrease the number of polygonal lines in the polygonal line object 15 depending on the magnitude of the acceleration of the vehicle 4 in the vertical direction.

[0068] (Behavior Status Information Example 5) FIG. 12 is a diagram showing behavior status information example 5 added to image 11 based on video information. Reference numeral 16 in FIG. 12 denotes gearshift text information 16 that indicates the state of the transmission of vehicle 4. In FIG. 12, gearshift text information 16 is added to image 11 based on video information. The display control unit 30 may add gearshift text information 16 to image 11 using behavior information that the driver has shifted gears (gearshift information related to the state of the transmission of vehicle 4), and control the display mode of image 11. Furthermore, in addition to adding gearshift text information 16 to image 11, the display control unit 30 may also add radial objects 14.

[0069] <Display Mode Control Example 2: Blur Processing> (Blur Processing Example 1) Fig. 13 is a diagram showing blur processing example 1 applied to an image 11 based on video information. The display control unit 30 according to the third embodiment may use the speed information to blur an area 17 along an edge E of the image 11, thereby controlling the display mode of the image 11 to be displayed on the user terminal 2. It is generally known that the faster the speed of the vehicle 4, the narrower the field of view of the passenger (driver). In Fig. 13, blur processing is applied to an area 17 along the edge E of the image 11, and an area C near the center is displayed as normal, thereby representing a state in which the field of view of the passenger (driver) is narrowed.

[0070] (Blurring Example 2) Fig. 14 is a diagram showing blurring example 2 applied to image 11 based on video information. In Fig. 14, the blurring range is narrower than in Fig. 13 (the size of the central region C is larger than in Fig. 13). The state in Fig. 14 shows a state in which the speed of vehicle 4 is slower than in the state in Fig. 13. In other words, the display control unit 30 may adjust the blurring range so that the faster the speed of vehicle 4 is, the larger the blurring range becomes.

[0071] <Display Mode Control Example 3: Rotating Image 11> Fig. 15 is a diagram showing a state in which the image 11 based on video information is rotated. In Fig. 15, the image 11 is rotated counterclockwise and tilted compared to Fig. 14 and the like. The display control unit 30 may control the display mode of the image 11 by rotating the image 11 using acceleration information. The display control unit 30 may rotate the image 11 using, for example, acceleration information related to the acceleration of the vehicle 4 in the lateral direction.

[0072] Generally, when a vehicle is traveling around a curve, the vehicle occupant tilts their head in the opposite direction to the centrifugal force to resist the centrifugal force. More specifically, when traveling around a right curve, a centrifugal force is generated in the left direction on the vehicle 4, and the occupant of the vehicle 4 tilts their head to the right, opposite the direction of the centrifugal force. FIG. 15 illustrates a scene viewed by the occupant when they tilt their head. In this manner, the display control unit 30 may rotate the image 11 using acceleration information related to the acceleration of the vehicle 4 in the lateral direction. Note that in FIG. 15 , the edge of the image 11 is displayed on the display of the user terminal 2. However, the image 11 may be enlarged in advance and displayed on the display of the user terminal 2 so that the edge of the image 11 is not displayed on the display of the user terminal 2 even when the image 11 is rotated.

[0073] 16 is a flow diagram showing the processing of the information processing system 100C according to the third embodiment. Steps S100 to S300 are the same as those in the first embodiment. In step S520 after step S300, the display control unit 30 uses the behavior information to control the display mode of the image 11.

[0074] For example, in step S300, the behavior information acquisition unit 20 may acquire, as behavior information, acceleration information related to the acceleration of the vehicle 4 in the lateral direction of the vehicle 4 (the behavior information acquisition unit 20 may acquire acceleration information in the right direction (toward the center of the arc) with respect to the traveling direction of the vehicle 4). In this case, in step S520, the display control unit 30 may control the display mode of the image 11 by rotating the image 11 counterclockwise using the behavior information as the acceleration information of the vehicle 4 in the right direction.

[0075] In addition, the process of controlling the display mode of the image 11 using the behavior information is as described above.

[0076] As described above, the information processing system 100C according to the third embodiment includes the video information acquisition unit 10, the behavior information acquisition unit 20, and the display control unit 30. The display control unit 30 uses the behavior information to control the display mode of the image 11 to be displayed on the user terminal 2. Since the display mode of the image 11 displayed on the user terminal 2 changes in accordance with the actual behavior of the vehicle 4 (speed, acceleration, engine RPM, transmission state, etc.), the user can get a sense of realism as if they were actually riding in the vehicle 4 even when they are outside the vehicle 4.

[0077] Furthermore, the behavior information includes at least one of speed information related to the speed of the vehicle 4, acceleration information related to the acceleration of the vehicle 4, rotation speed information related to the rotation speed of the engine of the vehicle 4, and gear shift information related to the state of the transmission of the vehicle 4. This makes it possible to easily obtain information related to the behavior of the vehicle 4.

[0078] Furthermore, the display control unit 30 controls the display mode of the image 11 displayed on the user terminal 2 by adding the behavior state information to the image 11 and displaying it. The display control unit 30 also changes the behavior state information using at least one of speed information, acceleration information, rotation speed information, and gear shift information. This allows the user to more effectively obtain a sense of realism as if they were actually riding in the vehicle 4 even when they are outside the vehicle 4.

[0079] Furthermore, the display control unit 30 controls the display mode of the image 11 by blurring an area along the edge of the image 11 using at least one of the speed information, the acceleration information, and the rotation speed information. This makes it possible to represent the size of the occupant's field of view on the user terminal 2.

[0080] Furthermore, the display control unit 30 uses the acceleration information to rotate the image 11, thereby controlling the display mode of the image 11. In this way, the scenery visually recognized by the occupant can be displayed on the user terminal 2.

[0081] Although the embodiments have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0082] In the first embodiment, the information processing device 1A has all of the functions related to the video information acquisition unit 10, the behavior information acquisition unit 20, the display control unit 30, and the vibration control unit 40, but some or all of these functions may be installed in the user terminal 2. For example, in the first embodiment, the user terminal 2 may have the video information acquisition unit 10. The same applies to the second and third embodiments.

[0083] In the first embodiment, the information processing device 1A may further include an output sound control unit 50. The functions and effects of the output sound control unit 50 according to the first embodiment are as described in the second embodiment. In addition, in the second embodiment, the information processing device 1B may further include a vibration control unit 40. The functions and effects of the vibration control unit 40 according to the second embodiment are as described in the first embodiment. In addition, in the third embodiment, the information processing device 1C may further include at least one of a vibration control unit 40 and an output sound control unit 50. The functions and effects of the vibration control unit 40 and the output sound control unit 50 according to the third embodiment are as described in the first and second embodiments.

[0084] Examples of reference forms are given below. 1. An information processing system comprising: a video information acquisition unit that acquires video information related to a video captured by an imaging device that moves with a vehicle; a behavior information acquisition unit that acquires behavior information related to the behavior of the vehicle when the imaging device captured the video; and a display control unit that displays an image based on the video information on a user terminal located outside the vehicle, wherein the display control unit uses the behavior information to control the display mode of the image to be displayed on the user terminal. 2. The information processing system described in 1., wherein the behavior information includes at least one of speed information related to the speed of the vehicle, acceleration information related to the acceleration of the vehicle, rotation speed information related to the rotation speed of the vehicle's engine, and gear shift information related to the state of a transmission of the vehicle. 3. The information processing system described in 1. or 2., wherein the display control unit controls the display mode of the image to be displayed on the user terminal by adding behavior state information suggesting the state of the behavior to the image and displaying it. 4. 3. In the information processing system described in 5.2, the behavior information includes at least one of speed information related to the speed of the vehicle, acceleration information related to the acceleration of the vehicle, rotation speed information related to the rotation speed of the vehicle's engine, and gear shift information related to the state of the vehicle's transmission, and the display control unit changes the behavior state information using at least one of the speed information, the acceleration information, the rotation speed information, and the gear shift information.In the information processing system described in 5.2, the display control unit controls the display mode of the image to be displayed on the user terminal by blurring an area along an edge of the image using at least one of the speed information, the acceleration information, and the rotation speed information.In the information processing system described in 6.2, the display control unit controls the display mode of the image to be displayed on the user terminal by rotating the image using the acceleration information.In the information processing system described in 7.1 to 6. 10. The information processing system according to claim 1, further comprising: a vibration control unit that controls vibration of the user terminal using the behavior information acquired by the behavior information acquisition unit.8. The information processing system described in any one of 1. to 7., further comprising an output sound control unit that controls sound to be output by the user terminal using the behavior information acquired by the behavior information acquisition unit. 9. A control method for an information processing system, in which a computer acquires video information related to a video captured by an imaging device that moves with a vehicle, acquires behavior information related to the behavior of the vehicle when the imaging device captured the video, displays an image based on the video information on a user terminal located outside the vehicle, and controls a display mode of the image to be displayed on the user terminal using the behavior information. 10. A program that causes a computer to execute the steps of: acquiring video information related to a video captured by an imaging device that moves with a vehicle, acquiring behavior information related to the behavior of the vehicle when the imaging device captured the video, displaying an image based on the video information on a user terminal located outside the vehicle, and controlling a display mode of the image to be displayed on the user terminal using the behavior information.

[0085] DESCRIPTION OF SYMBOLS 1A, 1B, 1C Information processing device 2 User terminal 3 Imaging device 4 Vehicle 10 Video information acquisition unit 20 Behavior information acquisition unit 30 Display control unit 40 Vibration control unit 50 Output sound control unit 60 Vehicle type information acquisition unit 100A, 100B, 100C Information processing system 101 Communication network 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface

Claims

1. An information processing system comprising: a video information acquisition unit that acquires video information regarding a video captured by an imaging device moving with a vehicle; a behavior information acquisition unit that acquires behavior information regarding the behavior of the vehicle when the imaging device captured the video; and a display control unit that displays an image based on the video information on a user terminal located outside the vehicle, wherein the display control unit uses the behavior information to control the display mode of the image to be displayed on the user terminal.

2. An information processing system according to claim 1, wherein the behavior information includes at least one of speed information relating to the speed of the vehicle, acceleration information relating to the acceleration of the vehicle, engine speed information relating to the engine speed of the vehicle, and gear shift information relating to the state of the transmission of the vehicle.

3. An information processing system according to claim 1 or 2, wherein the display control unit controls the display mode of the image displayed on the user terminal by adding behavior status information suggesting the behavior status to the image and displaying it.

4. An information processing system according to claim 3, wherein the behavior information includes at least one of speed information relating to the speed of the vehicle, acceleration information relating to the acceleration of the vehicle, RPM information relating to the RPM of the engine of the vehicle, and gear shift information relating to the state of the transmission of the vehicle, and the display control unit changes the behavior state information using at least one of the speed information, the acceleration information, the RPM information, and the gear shift information.

5. An information processing system as described in claim 2, wherein the display control unit controls the display mode of the image to be displayed on the user terminal by blurring areas along the edges of the image using at least one of the speed information, the acceleration information, and the rotation speed information.

6. An information processing system according to claim 2, wherein the display control unit controls the display mode of the image displayed on the user terminal by rotating the image using the acceleration information.

7. An information processing system according to claim 1 or 2, further comprising a vibration control unit that controls vibration of the user terminal using the behavior information acquired by the behavior information acquisition unit.

8. An information processing system according to claim 1 or 2, further comprising an output sound control unit that controls the sound to be output by the user terminal using the behavior information acquired by the behavior information acquisition unit.

9. A control method for an information processing system, in which a computer acquires video information relating to a video captured by an imaging device moving with a vehicle, acquires behavior information relating to the behavior of the vehicle when the imaging device captured the video, displays an image based on the video information on a user terminal located outside the vehicle, and uses the behavior information to control the display mode of the image displayed on the user terminal.

10. A program for causing a computer to execute the following steps: acquiring video information regarding a video captured by an imaging device moving with a vehicle; acquiring behavior information regarding the behavior of the vehicle when the imaging device captured the video; displaying an image based on the video information on a user terminal located outside the vehicle; and using the behavior information to control the display mode of the image to be displayed on the user terminal.

Images