Simulation device and server device

Abstract

To provide a simulation device for enabling a subject to perform a pseudo-experience of driving with a sense of presence by reproducing appropriate sound corresponding to a road state.SOLUTION: A simulation system 100 includes a server device 10 and a simulation device 20. The server device 10 comprises: a reception section 11, a recognition section 12, a sound source processing section 13; a delivery section 14; and a database 15 including an event information storage section 15a and a sound source information storage section 15b. The recognition section 12 recognizes a road state on an image based on the image recorded by an on-vehicle camera 1. The sound source processing section 13 generates travel sound data based on the recognized road state. The delivery section 14 transmits event information and the travel sound data to the simulation device 20. The simulation device 20 controls sound to be reproduced by a seat speaker 43 and a rear speaker 44 based on the road state.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a simulation device and a server device. [Background technology]

[0002] Conventionally, simulation devices that allow subjects to have a simulated driving experience are known (see, for example, Patent Documents 1 to 5). Patent Document 1 discloses that a reaction force acting on the steering wheel when the vehicle is traveling is calculated based on the steering angle, acceleration, road surface conditions, and traveling speed, and that the reaction force is applied to the steering wheel as a load. Patent Document 2 discloses that a simulated brake pedal is vibrated at a period corresponding to a friction coefficient between a simulated road surface on which the simulated vehicle is traveling and a simulated wheel when the ABS is activated, that the traveling sound of the simulated vehicle is output by a speaker, and that the traveling sound corresponding to the vehicle speed and road surface conditions is recorded in a memory.

[0003] Patent Document 3 discloses that drive sound data stored in a storage means is used to generate simulated sounds that approximate various actual drive sounds in various driving conditions, and that actual engine sounds are recorded as drive sound data by a microphone installed in the vehicle. Patent Document 4 discloses that a driving simulator is loaded with actual road surface data, such as a road surface profile and road surface friction coefficient, that are measured in advance on an actual road surface, and that a driver is made to drive freely on the simulated road surface, and that cockpit motion and excessive vibration are generated based on the input information and road profile data.

[0004] Patent document 5 discloses a method for generating vibrations that allow occupants of a moving body to feel the vibrations occurring in the moving body being photographed and the centrifugal force acting on the moving body from a field of view image showing the outside scenery photographed from the moving body, without using a library in which information related to vibrations is registered, by correlating the field of view image with the state of the moving body that captured the image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 4-232829 [Patent Document 2] Japanese Patent Application Publication No. 10-10963 [Patent Document 3] Japanese Patent Application Publication No. 11-249672 [Patent Document 4] JP 2005-316004 A [Patent Document 5] JP 2020-3358 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, Patent Document 2 discloses that the driver can select road conditions such as "dry road surface" or "packed snow road surface" according to the weather, but does not disclose automatic selection of whether the road condition is asphalt pavement, gravel pavement, etc. Patent Document 2 also does not disclose reproducing driving sounds according to the road conditions when the road conditions change during driving. Cited Documents 1, 4 to 5 change the steering reaction force and vibration according to the road conditions, but do not disclose reproducing driving sounds according to the road conditions. Cited Document 3 does not disclose reproducing driving sounds according to the road conditions.

[0007] Therefore, an object of the present disclosure is to provide a simulation device that can provide a subject with a more realistic simulated driving experience by reproducing appropriate sounds corresponding to road conditions. [Means for solving the problem]

[0008] In order to achieve the above object, the simulation device of the present disclosure is a simulation device that plays back images recorded by an information recording device installed in a vehicle, and includes a recognition unit that recognizes the state of the road in the image based on the image recorded by the information recording device, and a control unit that controls the sound played back by the playback unit based on the state of the road in the image. The recognition unit recognizes the state of the road surface in the image by performing image recognition on the image recorded by the information recording device. Effect of the Invention

[0009] The simulation device of the present disclosure configured in this manner can provide the test subject with a more realistic simulated driving experience by reproducing appropriate sounds corresponding to road conditions. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a conceptual diagram showing a schematic configuration of a simulation device in a simulation system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a functional configuration of a simulation system according to a first embodiment. [Diagram 3] 4 is a diagram for explaining a process in which a recognition unit recognizes the surface condition of a road. FIG. [Figure 4] FIG. 2 is an explanatory diagram for explaining a seat speaker, a rear speaker, and a VR speaker. [Diagram 5] FIG. 2 is a sequence diagram for explaining an example of an operation flow in the simulation system according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing the functional configuration of a simulation system according to a second embodiment. [Figure 7] FIG. 13 is a block diagram showing the functional configuration of a simulation system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (First embodiment) A simulation system 100 according to a first embodiment of the present disclosure will be described below with reference to the drawings. Fig. 1 is a diagram showing a schematic configuration of a simulation device 20 in the simulation system 100 according to the first embodiment. Fig. 2 is a block diagram showing a functional configuration of the simulation system 100 according to the first embodiment.

[0012] The simulation system 100 of this embodiment reproduces video data recorded by an in-vehicle camera 1, which is an example of an information recording device, to allow a subject P to have a simulated driving experience. This system reproduces the sound of a vehicle running in synchronization with the reproduction of the video data, thereby enabling the subject P to have a more realistic simulated experience.

[0013] A running sound is a sound generated when a moving body such as an automobile or a train runs on the ground. More specifically, when the moving body is a vehicle, the running sound is a sound generated when the tires pass over unevenness in the road surface. The running sound is a combination of sounds such as sounds caused by friction between the tires and the road surface, sounds caused by vibrations when the tires touch the road surface, sounds generated when the tires pass over uneven parts, sounds generated by the power of an engine or the like, sounds generated by applying the brakes, and the like.

[0014] The running sound changes depending on the road condition, more specifically, whether the road is asphalt paved or gravel paved. Furthermore, the running sound also changes depending on the size of the space inside the vehicle cabin, the running speed of the vehicle, and the like. For this reason, when a simulation is performed, it is preferable that a running sound corresponding to the state of the road in the image reproduced during the simulation is reproduced. During the simulation, a more realistic simulation is possible by reproducing a running sound close to reality according to the road condition, the size of the space inside the vehicle cabin, and the speed. However, the vehicle-mounted camera 1 generally equipped in a vehicle has a camera unit 2 and a microphone 7, and is equipped for the purpose of recording mainly in-vehicle voice data such as the conversation of the passengers together with image data, and further, high performance is not required due to cost and the like. For this reason, the microphone 7 of the vehicle-mounted camera 1 does not record or is difficult to record the running sound of the vehicle, which is a sound with a lower frequency than a human voice. For this reason, even if the microphone 7 of the vehicle-mounted camera 1 reproduces pre-recorded voice data in the simulation, it is difficult to reproduce a running sound close to reality.

[0015] Therefore, the simulation system 100 of this embodiment generates driving sounds according to the road conditions, the size of the space inside the vehicle, and the speed of the vehicle, and plays them during the simulation. The driving sounds include sounds in a range not recorded by the vehicle-mounted camera 1, and may be prepared separately from the audio data recorded in the video data.

[0016] The details of the simulation system 100 according to this embodiment will be described below with reference to the drawings. As shown in Fig. 1 and Fig. 2, the simulation system 100 according to this embodiment mainly includes a server device 10 and a simulation device 20.

[0017] The server device 10 generates driving sound data based on event information acquired from the vehicle-mounted camera 1, and transmits the generated driving sound data and event information to the simulation device 20. The server device 10 is configured from a cloud server or the like. As shown in Fig. 2, the server device 10 is capable of communicating with a plurality of vehicle-mounted cameras 1 via a communication network L1. The server device 10 is capable of communicating with the simulation device 20 via a communication network line L2 such as the Internet.

[0018] The server device 10 acquires various event information from multiple vehicle-mounted cameras 1. The event information is information acquired and recorded by the vehicle-mounted cameras 1 when an accident or an event that does not result in an accident but has a high risk of causing an accident occurs. The vehicle-mounted cameras 1 are installed in vehicles (not shown). Examples of vehicles include so-called commercial vehicles such as trucks, taxis, buses, and company cars, and so-called personal vehicles owned by individuals.

[0019] The vehicle-mounted camera 1 can be a "video recording type drive recorder." As shown in FIG. 2, the vehicle-mounted camera 1 mainly includes a camera unit 2, a G-sensor 3, a GPS 4, a communication unit 5, a memory 6, a microphone 7, and the like. The camera unit 2 is composed of a 360-degree camera and can capture images in all directions of 360 degrees around the vehicle. The G-sensor 3 is also called an acceleration sensor, and is a sensor that detects acceleration data detected by the vehicle-mounted camera 1. The microphone 7 records the sound inside the vehicle.

[0020] The vehicle-mounted camera 1 records in the memory 6 as event information the time of a predetermined period before and after the G-sensor 3 detects an acceleration equal to or greater than a predetermined value, location information acquired by the GPS 4, video data including image data captured by the camera unit 2 and audio data recorded by the microphone 7, impact information detected by the G-sensor 3, and various data such as blinker operation and brake operation. The communication unit 5 of the vehicle-mounted camera 1 transmits the event information recorded in the memory 6 to the server device 10 via the communication network L1.

[0021] 2, the server device 10 mainly includes a receiving unit 11, a recognizing unit 12, a sound source processing unit 13, a distributing unit 14, and a database 15. The database 15 includes an event information storage unit 15a and a sound source information storage unit 15b.

[0022] The event information storage unit 15a stores event information acquired from the vehicle-mounted camera 1. The event information includes video data, impact information data, position information data, etc. recorded by the vehicle-mounted camera 1. The video data includes image data consisting of moving images and audio data.

[0023] The sound source information storage unit 15b stores sound source data in advance. This sound source data is used to generate running sound data to be played simultaneously with the video data of the event information. The sound source data is actually recorded or generated by a business providing the simulation system 100, such as by a voice synthesis software or AI synthesis. The business stores the recorded or generated sound source data in the sound source information storage unit 15b by uploading or the like.

[0024] The sound source data is actual sounds generated when a vehicle travels on a road or sounds synthesized to resemble actual sounds. The sound source data is recorded or synthesized according to the condition of the road on which the vehicle travels and the size of the space inside the vehicle. The recorded or synthesized sound source data are classified by vehicle type and road condition, and stored as sound source data in the sound source information storage unit 15b.

[0025] The sound source data desirably includes sounds in a range not recorded by the vehicle-mounted camera 1, more desirably includes low-frequency sounds, and even more desirably includes running sounds when the vehicle runs on a road. The running sounds desirably include road noise. Road noise is a sound caused by vibrations when the tires touch the road surface. Road noise is a sound in a relatively low range (low-frequency to mid-frequency range) of 100 Hz to 1 kHz, and is a sound that is difficult to record by the microphone 7 of the vehicle-mounted camera 1. For this reason, by playing back running sounds including road noise during the simulation, the subject P can have a more realistic simulated experience.

[0026] The receiving unit 11 receives event information including video data from the vehicle-mounted camera 1 via the communication network L1. The receiving unit 11 stores the event information received from the vehicle-mounted camera 1 in the event information storage unit 15a. At this time, the receiving unit 11 can appropriately classify the event information according to, for example, the car name, the car model, the type of accident, and the like, and then store the event information in the event information storage unit 15a. The receiving unit 11 can also classify the event information by distinguishing between accidents and events that do not result in an accident but have a high risk of causing an accident, and store the event information in the event information storage unit 15a.

[0027] The receiving unit 11 receives information on the vehicle to be simulated and the ID number of the event information from the simulation device 20 via the communication network line L2. The receiving unit 11 outputs the received vehicle information (for example, the vehicle name in this embodiment) to the sound source processing unit 13, and outputs the ID number of the event information to the recognition unit 12.

[0028] The vehicle information is used when the sound source processing unit 13 generates running sound data. Since the running sound changes depending on the size of the space in the vehicle cabin, in this embodiment, the running sound data is generated taking into consideration the change in sound due to the size of the vehicle cabin.

[0029] The vehicle information is preferably the type of vehicle, more specifically, the vehicle name, the model, etc., and among these, the vehicle name is most preferably cited since it allows the size of the space inside the vehicle to be more appropriately grasped. Note that the vehicle information is not limited to the vehicle name and the model, and may be the volume of the vehicle interior as long as it allows the size of the space inside the vehicle interior to be grasped.

[0030] A vehicle name is a unique name given to a vehicle by an automobile manufacturer, etc. A vehicle model is a name classified according to use, structure, size, function, etc., and includes names by use such as passenger car, small freight car, and light four-wheel passenger car, and names by body type such as sedan, sports car, and station wagon.

[0031] The recognition unit 12 acquires event information corresponding to the ID number input from the receiving unit 11 from the event information storage unit 15a. The recognition unit 12 performs image analysis on image data in the video data of the acquired event information to recognize the road condition. This "road condition" indicates the state of the road on which the vehicle is traveling, including unevenness, and includes conditions due to differences in the materials that make up the road, such as asphalt and gravel, conditions due to differences in road conditions, such as sunny weather, rainy weather, and snowy weather, and conditions due to differences in time periods, such as daytime and nighttime.

[0032] In order to appropriately recognize this "road condition," the recognition unit 12 of this embodiment recognizes the "road surface condition" by performing image recognition on the image data. This "road surface condition" is recognized by identifying objects present on the road surface and the condition of the objects. Examples of such objects include materials that constitute the road, water, snow, ice, paint, and foreign objects present on the road.

[0033] The image recognition method is not particularly limited, and any suitable well-known image recognition method can be used, such as a machine learning method using a trained model that uses images of various road surface conditions as training data, a deep learning method using a neural network, a pattern matching method, or a method that extracts image features.

[0034] FIG. 3 is a diagram for explaining the process by which the recognition unit 12 recognizes the surface condition of the road, and G1 in FIG. 3 indicates an image reproduced based on image data recorded by the vehicle-mounted camera 1. Note that image G1 shown in FIG. 3 is a part of an image based on image data recorded by the camera unit 2 consisting of a 360-degree camera. The recognition unit 12 actually performs image recognition of an image in a 360-degree direction. The recognition unit 12 detects, for example, an area R1 where a road appears from the image G1 by image recognition, and performs image recognition on this area R1 to recognize the texture, color, etc., and recognizes the surface condition of the road based on these.

[0035] "Texture" generally refers to the feel and touch (touch) of the surface of an object, but here it refers to the texture of the road surface in image G1. The texture of the road surface differs depending on the materials that make up the road, its condition, etc. The recognition unit 12 recognizes the unevenness of the road based on the texture of the road surface in image G1, and recognizes the condition of the road surface based on the unevenness, color, brightness, etc.

[0036] The recognition unit 12 recognizes the type of road pavement as the "road surface condition." The types of pavement include, but are not limited to, asphalt pavement, concrete pavement, brick pavement, block pavement, cobblestone pavement, gravel pavement, and the like. The types of pavement may include appropriate pavement applied to improve the road or increase its strength, such as pavement using resin, wood, soil, sand, and the like. The road may have a special structure that changes the sound of the road or plays music or the like when a vehicle runs on it. The recognition unit 12 recognizes such a special structure as one type of pavement.

[0037] On the road, center lines, parking frame lines, no parking marks, etc. may be drawn on the paved road surface as described above by painting or using a material different from the paving material. Such lines and marks may also change the unevenness of the road surface, and the running sound may change when the tire (wheel) runs on the lines or marks. At this time, the road surface on which the tire runs may frequently switch between the center line or marks and the paved portion depending on the steering operation, etc., so it is desirable that the running sound also changes in response to the change in the road surface in order to properly recognize the change in the road surface. For this reason, the recognition unit 12 may be configured to detect an area R2 near the tire in the traveling direction of the vehicle by image recognition instead of the area R1 including the entire road, and recognize the surface condition of the road within this area R2. This allows the recognition unit 12 to recognize the change in the surface condition of the road on which the tire runs in more detail and more appropriately.

[0038] The recognition unit 12 can recognize road surface conditions such as snow, dry, wet, and frozen as the "road surface condition" and can recognize the time of day based on the brightness of objects, etc. This is because the running sound changes depending on the road surface condition and the time of day.

[0039] The sound source processing unit 13 generates driving sound data to be reproduced by the simulation device 20 based on the surface condition of the road and the size of the space inside the vehicle cabin. For this purpose, the sound source processing unit 13 acquires corresponding sound source information data from the sound source information storage unit 15b based on the vehicle name input from the receiving unit 11 and the surface condition of the road recognized by the recognition unit 12.

[0040] In this embodiment, the sound source information storage unit 15b stores sound sources according to the road surface condition for each size of the space inside the vehicle corresponding to the vehicle name. Therefore, the sound source processing unit 13 can obtain appropriate sound source data corresponding to the road surface condition and the size of the space inside the vehicle. The sound source processing unit 13 uses the obtained sound source data to generate running sound data in association with the video data.

[0041] More specifically, when the recognition result of the road surface condition is asphalt pavement, the sound source processing unit 13 acquires sound source data when traveling on asphalt pavement corresponding to the vehicle name from the sound source information storage unit 15b and generates traveling sound data. When the recognition result of the road surface condition switches from asphalt pavement to gravel pavement, the sound source processing unit 13 acquires sound source data when traveling on asphalt pavement corresponding to the vehicle name and sound source data when traveling on gravel pavement from the sound source information storage unit 15b, and connects these sound source data in accordance with the change in the road surface condition in the image G1 based on the image data to generate traveling sound data.

[0042] The distribution unit 14 transmits, to the simulation device 20, the event information corresponding to the ID number from the simulation device 20 and the running sound data generated by the sound source processing unit 13 in response to this event information, as one of the recognition results.

[0043] The simulation device 20 is a device that allows the subject P to have a simulated driving experience, and is also called a "driving simulator" or the like. As shown in FIG. 2, the simulation device 20 of the first embodiment mainly includes a controller 30, a seat unit 40, an image display unit 50, and an operation unit 60.

[0044] The controller 30 is an information terminal operated by the operator of the simulation device 20, the subject P, etc. The controller 30 is composed of a personal computer equipped with a CPU, RAM, ROM, a touch panel, a liquid crystal display, etc., but can also be composed of a tablet terminal, a smartphone, a notebook computer, etc.

[0045] As shown in FIG. 2, the controller 30 mainly includes an input unit 31, a communication unit 32, an information processing unit 33, and a storage unit .

[0046] The input unit 31 is a device for an operator or a subject P to input various instructions and information to the simulation device 20. The input unit 31 is composed of a touch panel, a keyboard, a mouse, etc. The operator, etc. inputs, from the input unit 31, the vehicle name as information on the vehicle to be simulated and the ID number of the event information.

[0047] The communication unit 32 is a communication module or a communication interface for communicating with the server device 10 via the communication network line L2. The communication unit 32 transmits the vehicle name and ID number input from the input unit 31 to the server device 10. The communication unit 32 receives event information and running sound data from the server device 10, and outputs them to the control unit 35.

[0048] The information processing unit 33 is mainly composed of a CPU, a RAM, etc. The storage unit 34 is mainly composed of a storage device such as a ROM, a HDD, etc., but may also include an external server or database.

[0049] The storage unit 34 temporarily or non-temporarily stores a control program for operating the simulation device 20, and various data and parameters used during various operations in the control unit 35. The storage unit 34 temporarily or non-temporarily stores event information and voice data.

[0050] The information processing unit 33 loads a predetermined program stored in the ROM into the RAM and executes it to control the operation of the entire simulation device 20. The information processing unit 33 functions as a control unit 35 that controls the operation of the running sound reproducing unit 42 and the image display unit 50 of the seat unit 40, as shown in FIG.

[0051] The control unit 35 stores the event information and running sound data received by the communication unit 32 from the server device 10 in the storage unit 34. The control unit 35 controls the image display unit 50 to play the video data of the event information. In synchronization with the playback of this video data, the control unit 35 controls the running sound playback unit 42 of the seat unit 40 to play the running sound data. The volume of the running sound data has a maximum volume and a minimum volume that are determined in advance, and the control unit 35 determines the volume within the range between the maximum volume and the minimum volume according to the speed of the vehicle.

[0052] The volume is determined as follows. When playing back the video data and the running sound data, the control unit 35 acquires the amount of operation from the accelerator detection unit 62a provided in the accelerator unit 62 of the operation unit 60 at any time, and calculates the speed of the vehicle during the simulated driving based on the acquired amount of operation. The control unit 35 determines the volume based on the calculated speed, and causes the running sound playback unit 42 to play back the running sound data at the determined volume.

[0053] The control unit 35 can also evaluate the driving of the subject P based on the amount of operation obtained from an accelerator detection unit 62a provided in the accelerator unit 62 of the operation unit 60, the steering angle obtained from a steering angle detection unit 61a provided in the steering unit 61 of the operation unit 60, the amount of operation obtained from a brake detection unit 63a provided in the brake unit 73, and further the direction of the subject P's line of sight detected by the line of sight detection unit 52 of the image display unit 50. The control unit 35 can output the evaluation result to the liquid crystal display of the controller 30, a printer, etc.

[0054] The seat unit 40 mainly includes a seat body 41 and a running sound reproducing unit 42 which is a second reproducing unit. The seat body 41 is a member on which the subject P sits. The running sound reproducing unit 42 is a device that reproduces running sounds, and includes a seat speaker 43 and a rear speaker 44.

[0055] 4, the seat speaker 43 is provided on the seat surface of the seat body 41. The rear speaker 44 is provided on the backrest of the seat body 41. The seat speaker 43 and the rear speaker 44 play driving sound data under the control of the control unit 35, and output driving sounds including road noise toward the subject P. Since the driving sounds are low-frequency sounds, the seat body 41 vibrates, and the vibrations are transmitted to the subject P sitting in the seat body 41. Therefore, the subject P can have a more realistic simulated experience due to the realistic driving sounds and vibrations.

[0056] The control unit 35 can output the same running sounds at the same volume from the seat speaker 43 and the rear speaker 44. Alternatively, the control unit 35 can output the running sounds at a louder volume from the seat speaker 43, which is closer to the road, than from the rear speaker 44, or output the running sounds by changing the range of sound reproduced by the seat speaker 43 and the rear speaker 44. This allows the seat speaker 43 and the rear speaker 44 to output more realistic running sounds.

[0057] The seat unit 40 may include a vibration generating unit. The control unit 35 drives the vibration generating unit to generate vibrations based on the detection result of the G sensor 3 when the surroundings of the vehicle are photographed by the camera unit 2 in accordance with the reproduction of image data and audio data, and applies vibrations to the subject P similar to those experienced when driving an actual vehicle, thereby enhancing the sense of realism.

[0058] The image display unit 50 is an image display device that reproduces image data and audio data of the video data recorded by the camera unit 2 under the control of the control unit 35 and presents them to the subject P. The image display unit 50 of this embodiment is configured as a head mounted display and is worn on the head of the subject P, but is not limited to this and can be configured as a known image display device.

[0059] The image display unit 50 mainly includes an image reproduction unit 51, a line of sight detection unit 52, a VR display unit 53, and a VR speaker 54 serving as a first playback unit. The image display unit 50 includes components that are included in a typical image display device, such as an optical system such as a backlight, a projection optical system, and the like.

[0060] Under the control of the control unit 35, the image reproducing unit 51 displays an image G1 based on the image data on the display surface of the VR display unit 53 and reproduces the audio data through the VR speaker 54. As shown in FIG. 1, the image reproducing unit 51 sets a spherical projection surface V in a 360-degree direction around the subject P. The image reproducing unit 51 displays an image G1 of a predetermined range of an area W centered on the direction of the line of sight detected by the line of sight detection unit 52 on the VR display unit 53 while moving the area in response to changes in the line of sight direction. This allows the subject P to visually recognize the image G1 corresponding to his or her own line of sight direction.

[0061] That is, when the subject P faces forward or backward, the image reproducing unit 51 causes the VR display unit 53 to display an image of the front or rear of the vehicle, respectively. When the subject P faces left, right, up, or down, the image reproducing unit 51 causes the VR display unit 53 to display an image of the left, right, up, or down of the vehicle, respectively.

[0062] The gaze detection unit 52 is composed of an acceleration sensor and the like, and detects the gaze direction based on the movement of the head of the subject P. In this embodiment, the gaze direction is expressed as the position where an extension of the gaze of the subject P intersects with the image G1 displayed on the display surface of the VR display unit 53, that is, the three-dimensional coordinates in the image G1. The gaze detection unit 52 outputs position information of the detected gaze direction to the image reproduction unit 51.

[0063] The line-of-sight detection unit 52 is not limited to an acceleration sensor, and may be, as another example, an imaging device that captures an image of the subject P. The line-of-sight detection unit 52 detects the eyes from the captured image of the subject P by image recognition, and detects the direction of the line of sight.

[0064] The VR display unit 53 is placed in front of the eyes of the subject P who wears the image display unit 50 on his / her head, and the image G1 is displayed on the display surface. The VR display unit 53 is composed of a liquid crystal display, an organic EL display, or the like. The VR display unit 53 is sufficient if the subject P can view the image G1, and can be of a transmissive type that allows the subject P to view the external real environment along with the displayed image, but can also be of a non-transmissive type.

[0065] The VR speaker 54 is provided on the temple of the image display unit 50 and outputs sound to the surroundings. The VR speaker 54 may be a headphone-type speaker that is attached to the ear of the subject P, an earphone-type speaker, or may be built into or attached to any position of the image display unit 50 and configured to output sound to the surroundings. This allows the subject P to hear the sound when the image data is recorded while viewing the image G1, enabling a more realistic simulated experience.

[0066] The operation unit 60 is a member that accepts various operations (motions) of the subject P. The operation unit 60 mainly includes a steering unit 61, an accelerator unit 62, and a brake unit 63. The operation unit 60 may also include a clutch unit, a turn signal unit, etc.

[0067] The steering unit 61 is a member simulating a steering wheel of a vehicle, and receives input of a steering operation by the subject P. The steering unit 61 includes a steering angle detection unit 61a. The steering angle detection unit 61a detects the steering angle (steering angle) and the like based on the operation of the steering unit 61 by the subject P, and transmits the detection result to the information processing unit 33.

[0068] The accelerator unit 62 is a component simulating a vehicle's accelerator pedal, and receives the accelerator operation of the subject P. The accelerator unit 62 is equipped with an accelerator detection unit 62a. The accelerator detection unit 62a detects the amount of operation of the accelerator unit 62 by the subject P, and transmits the detection result to the information processing unit 33. The brake unit 63 is a component simulating a vehicle's brake pedal, and receives the brake operation of the subject P. The brake unit 63 is equipped with a brake detection unit 63a. The brake detection unit 63a detects the amount of operation, etc. based on the operation of the brake unit 63 by the subject P, and transmits the detection result to the information processing unit 33.

[0069] An example of the flow of operations of the simulation system 100 according to the first embodiment having the above-mentioned configuration will be described below with reference to the sequence diagram of Fig. 5. Fig. 5 shows an example of the flow of operations of the simulation system 100, but the operations of the simulation system 100 are not limited to those shown in Fig. 5.

[0070] 5 is started after the power supply of the simulation device 20 is turned on by an operator or the like who operates the simulation system 100 and the simulation system 100 is started. The server device 10 collects various event information from the vehicle-mounted camera 1 and stores it in the event information storage unit 15a, and stores sound source data corresponding to various road surface conditions for each of various vehicle names in the sound source information storage unit 15b.

[0071] First, the subject P sits in the seat body 41 of the simulation device 20, and the operator or the subject P inputs the vehicle name of the vehicle to be simulated and the ID number of the event information from the input unit 31 of the simulation device 20. In response to this input, the communication unit 32 transmits the vehicle name and ID number to the server device 10. In step S2, the receiving unit 11 of the server device 10 receives the vehicle name and ID number, and outputs them to the recognition unit 12. In the next step S3, the recognition unit 12 obtains the event information corresponding to the ID number from the event information storage unit 15a, and performs image analysis of the image data in the obtained event information to recognize the condition of the road surface.

[0072] In the next step S4, the sound source processing unit 13 acquires corresponding sound source information data from the sound source information storage unit 15b based on the vehicle name and the road surface state that is the recognition result by the recognition unit 12, and generates running sound data using the acquired sound source data. In the next step S5, the distribution unit 14 transmits the event information acquired by the recognition unit 12 and the running sound data generated by the sound source processing unit 13 to the simulation device 20.

[0073] In the next step S6, the communication unit 32 of the simulation device 20 receives the event information and the running sound data from the server device 10. These data are temporarily or non-temporarily stored in the storage unit 34 by the control unit 35. With the above, the simulation device 20 is ready for the simulation.

[0074] When the operator or subject P presses a start button or the like provided on the input unit 31, an instruction to start the simulation is sent to the control unit 35. When the control unit 35 receives this instruction to start the simulation in step S7, the program proceeds to step S8, where the control unit 35 controls the image display unit 50 to play one frame of video data of the event information, and in synchronization with the playback of this video data, controls the running sound playback unit 42 to play the running sound data.

[0075] As a result, an image G1 according to the direction of the subject P's line of sight is displayed on the display surface of the VR display unit 53, the audio recorded in the video data is output from the VR speaker 54, and driving sounds including road noise are output from the seat speaker 43 and the back speaker 44. Therefore, the subject P can see the realistic image G1 recorded by the vehicle-mounted camera 1 and hear the audio, while also hearing low-frequency sounds that cannot be recorded by the vehicle-mounted camera 1, i.e., more realistic driving sounds including road noise.

[0076] During playback of the video data and the running sound data, in step S9, the control unit 35 calculates the speed based on the operation amount obtained from the accelerator detection unit 62a, and determines whether the speed of the vehicle during the simulated driving has changed. If the control unit 35 determines that the speed has changed, that is, that the vehicle has accelerated or decelerated (YES), the program proceeds to step S10, and if the control unit 35 determines that the speed has not changed (NO), the program proceeds to step S11.

[0077] In step S10, the control unit 35 controls the seat speaker 43 and the rear speaker 44 to adjust the volume of the driving sound data to a volume corresponding to the calculated speed within a range from a predetermined lower limit value to an upper limit value, and plays the data.

[0078] In the next step S11, the control unit 35 determines whether the video data (frame) has ended. When the control unit 35 determines that the video data has not ended, that is, that there is a next frame to be played back (NO), the program returns to step S8 and repeats the process of steps S8 to S10 for the next frame.

[0079] As a result, the subject P can have a simulated driving experience while viewing the image G1 displayed on the VR display unit 53, which changes according to the direction of the subject P's line of sight, and listening to the audio data recorded by the in-vehicle camera 1, which is output from the VR speaker 54. The subject P can hear the driving sounds, including road noise according to the road surface condition and the size of the space inside the vehicle, which are output from the seat speaker 43 and the rear speaker 44, allowing for a more realistic simulated experience. The seat body 41 vibrates due to the playback of low-frequency road noise, and the volume changes according to the amount of operation of the accelerator unit 72 by the subject P, allowing the subject P to simulate the vibration and sense of speed when driving, further improving the sense of realism.

[0080] In step S11, when the control unit 35 determines that the playback of all video data (frames) has ended (YES), the program proceeds to END, and the simulation for the event information ends. When the subject P or another subject P inputs the ID number and car name of a desired event from the input unit 31, the process from step S1 is executed, and a new simulation can be performed.

[0081] Second embodiment In a simulation system 100A according to the second embodiment shown in Fig. 7, a recognition unit 36 ​​and a sound source processing unit 37 are provided in a simulation device 20 instead of a server device 10. Except for this, the simulation system 100A according to the second embodiment has the same basic configuration as the simulation system 100 according to the first embodiment shown in Fig. 1, Fig. 2, etc. Therefore, in the following, a description of the same configuration as the first embodiment will be omitted, and configurations different from the first embodiment will be mainly described.

[0082] The simulation system 100A of the second embodiment mainly includes a server device 10 and a simulation device 20. The server device 10 mainly includes a receiving unit 11, a distribution unit 14, and a database 15 having an event information storage unit 15a and a sound source information storage unit 15b. The simulation device 20 mainly includes a controller 30, a seat unit 40, an image display unit 50, and an operation unit 60.

[0083] The controller 30 mainly includes an input unit 31, a communication unit 32, an information processing unit 33, and a storage unit 34. The information processing unit 33 includes a control unit 35, a recognition unit 36, and a sound source processing unit 37.

[0084] In the simulation system 100A of the second embodiment, the receiving unit 11 of the server device 10 receives event information from multiple vehicle-mounted cameras 1 and stores it in the event information storage unit 15a of the database 15. The receiving unit 11 receives the car name and the ID number of the event information from the simulation device 20. The distribution unit 14 obtains the event information corresponding to the ID number received by the receiving unit 11 from the event information storage unit 15a, obtains the sound source data corresponding to the car name from the sound source information storage unit 15b, and transmits them to the simulation device 20.

[0085] In the simulation device 20, the recognition unit 36 ​​performs image recognition of the received event information and recognizes the surface condition of the road. Next, the sound source processing unit 37 generates driving sound data using the received sound source data based on the recognition result. The control unit 35 controls the image display unit 50 to play the video data of the event information. In synchronization with the playback of this video data, the control unit 35 controls the seat speaker 43 and the back speaker 44 of the seat unit 40 to play the driving sound data at a volume corresponding to the amount of operation of the accelerator unit 72.

[0086] Third embodiment In a simulation system 100B according to the third embodiment shown in Fig. 7, a recognition unit 36, a sound source processing unit 37, and a sound source information storage unit 34a are provided in a simulation device 20 instead of the server device 10. Except for this, the simulation system 100B according to the second embodiment has the same basic configuration as the simulation system 100 according to the first embodiment shown in Figs.

[0087] The simulation system 100B of the third embodiment mainly includes a server device 10 and a simulation device 20. The server device 10 mainly includes a receiving unit 11, a distribution unit 14, and a database 15 having an event information storage unit 15a. The simulation device 20 mainly includes a controller 30, a seat unit 40, an image display unit 50, and an operation unit 60.

[0088] The controller 30 mainly includes an input unit 31, a communication unit 32, an information processing unit 33, and a storage unit 34. The information processing unit 33 includes a control unit 35, a recognition unit 36, and a sound source processing unit 37. In the third embodiment, the storage unit 34 has a sound source information storage unit 34a in which a plurality of sound source data corresponding to various vehicle names and various road surface conditions are stored.

[0089] In the simulation system 100B of the third embodiment, the receiving unit 11 of the server device 10 receives event information from multiple vehicle-mounted cameras 1 and stores it in the event information storage unit 15a of the database 15. The receiving unit 11 receives the ID number of the event information from the simulation device 20. The distribution unit 14 obtains the event information corresponding to the ID number from the event information storage unit 15a and transmits it to the simulation device 20.

[0090] In the simulation device 20, the recognition unit 36 ​​performs image recognition of the received event information and recognizes the surface condition of the road. Based on the surface condition of the road recognized by the recognition unit 36 ​​and the vehicle name input from the input unit 31, the sound source processing unit 37 acquires sound source data from the sound source information storage unit 15b and generates running sound data. The control unit 35 controls the image display unit 50 to play the video data of the event information. In synchronization with the playback of this video data, the control unit 35 controls the seat speaker 43 and the back speaker 44 of the seat unit 40 to play the running sound data at a volume corresponding to the amount of operation of the accelerator unit 72.

[0091] As described above, the simulation systems 100A and 100B of the second and third embodiments also provide the same effects as the simulation system 100 of the first embodiment. In the simulation systems 100A and 100B of the second and third embodiments, the server device 10 only needs to transmit event information and / or sound source data in response to a request from the simulation device 20, and does not need to perform image recognition or generation of running sounds. This reduces the capacity and calculation load of the server device 10. The simulation systems 100A and 100B of the present embodiment can be realized by installing a simulation program that causes a computer to function as the recognition unit 36 ​​and the sound source processing unit 37.

[0092] In contrast, in the simulation system 100 of the first embodiment, the server device 10 performs image recognition and generation of running sounds, so that the simulation device 20 only needs to play back the video data and running sound data, thereby reducing the capacity and load of the simulation device 20 and reducing the manufacturing costs and sales costs of the simulation device 20 and the program.

[0093] In the simulation systems 100 and 100A of the first and second embodiments, the sound source information storage unit 15b is provided in the database 15 of the server device 10. Therefore, the business providing the simulation systems 100 and 100A can provide the latest sound source data to each simulation device 20 simply by updating the sound source information storage unit 15b of the server device 10, thereby reducing the burden on the business. In contrast, in the simulation system 100B of the third embodiment, the sound source information storage unit 34a is provided in the storage unit 34 of the simulation device 20. Therefore, the business does not need to provide the sound source information storage unit 15b in the server device 10, and can use the existing server device 10 that collects and records event information, thereby reducing manufacturing costs, etc.

[0094] As described above, the simulation system 100 according to the first embodiment includes a recognition unit 12 that recognizes the state of the road in an image based on an image recorded by an information recording device (e.g., an on-board camera 1) installed in a vehicle, and a distribution unit 14 that transmits the recognition result by the recognition unit 12 to a simulation device 20 having a control unit 35 that plays back the image recorded by the information recording device and controls the sound played by the playback unit (e.g., the VR speaker 54, the seat speaker 43, and the rear speaker 44) based on the state of the road.

[0095] With this configuration, the control unit 35 can control the sound reproduced by the reproduction unit based on the surface state of the road recognized by the recognition unit 12 of the server device 10. Therefore, the sound corresponding to the state of the road in the image is reproduced by the reproduction unit, and even when the state of the road in the image changes, the sound corresponding to the changed state of the road is reproduced by the reproduction unit. Therefore, the simulation system 100 according to the first embodiment can provide the subject P with a more realistic simulated driving experience by reproducing appropriate sounds corresponding to the state of the road.

[0096] The simulation device 20 included in the simulation systems 100A, 100B according to the second and third embodiments is a simulation device 20 that plays back images recorded by an information recording device (e.g., the vehicle-mounted camera 1) installed in a vehicle. The simulation device 20 includes a recognition unit 36 ​​that recognizes the state of the road in the image based on the image recorded by the vehicle-mounted camera 1, and a control unit 35 that controls the sound reproduced by the reproduction unit (e.g., the VR speaker 54, the seat speaker 43, and the back speaker 44) based on the state of the road in the image. The recognition unit 12 recognizes the surface state of the road in the image by performing image recognition on the image recorded by the information recording device.

[0097] As a result, the control unit 35 controls the playback unit to play sounds corresponding to the road conditions in the image, and even when the road conditions change, the control unit 35 can play sounds corresponding to the changed road conditions. Therefore, the simulation device 20 according to the second and third embodiments and the simulation systems 100A and 100B including the same can play appropriate sounds corresponding to the road conditions, allowing the subject P to have a more realistic simulated driving experience.

[0098] The control unit 35 in each of the above embodiments controls the playback unit to output sounds in a range that is not recorded by the information recording device. The sounds in a range that is not recorded by the information recording device preferably include sounds in a range lower than the human voice, preferably include running sounds generated when a vehicle runs on a road, and preferably include road noise. By having the playback unit play such sounds, more realistic sounds like those when a vehicle is actually running are played, and the subject P can have a more realistic simulated experience.

[0099] It is known that the sound reverberating in a vehicle changes depending on the size of the space in the vehicle cabin. The size of the space in the vehicle cabin differs depending on the type of vehicle. For this reason, the simulation systems 100, 100A, and 100B of the above-mentioned embodiments include an input unit 31 that inputs information about the vehicle to be simulated. The control unit 35 controls the playback unit to output sounds in a range not recorded by the information recording device, based on both the size of the space in the vehicle cabin of the vehicle to be simulated and the state of the road in the image. Therefore, the simulation systems 100, 100A, and 100B of the above-mentioned embodiments can cause the playback unit to play sounds corresponding to not only the state of the road surface but also the size of the space in the vehicle cabin, thereby further enhancing the sense of realism.

[0100] The reproducing unit in each of the above embodiments includes a first reproducing unit (VR speaker 54) that outputs sounds in the range recorded by the information recording device, and a second reproducing unit (running sound reproducing unit 42 having seat speaker 43 and rear speaker 44) that outputs sounds in the range not recorded by the information recording device. Therefore, the subject P can hear both the sounds in the vehicle cabin recorded by the information recording device and the running sounds generated in response to the surface condition of the road. It is preferable that the second reproducing unit is provided in the seat 40 where the subject P sits, so that the subject P can not only hear more realistic running sounds but also feel the vibrations caused by the running sounds, and can have a more realistic simulated experience.

[0101] The recognition unit 12, 36 in each of the above embodiments may detect an area B near the wheels (tires) in the traveling direction of the vehicle from the image recorded by the information recording device, and recognize the surface condition of the road in this area B. With this configuration, even if the surface condition of the road on which the vehicle is traveling changes frequently, this change is recognized, and the control unit 35 can cause the playback unit to play a running sound corresponding to this change. When the vehicle deviates from the traveling lane and travels on the center line, etc., the running sound changes in response to the change in the surface condition of the road. Therefore, the subject P can properly recognize that the vehicle has deviated from the traveling lane, and can have a more realistic simulated experience.

[0102] Although the embodiments of the present disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and design changes that do not deviate from the gist of the present disclosure are included in the present disclosure. [Explanation of symbols]

[0103] 1: Vehicle-mounted camera (information recording device) 10: Server device 12: Recognition section 14: Distribution section 20: Simulation device 31: Input unit 35: Control unit 36: Recognition unit 42: Running sound reproduction unit (second reproduction unit) 43: Seat speaker (second reproduction unit) 44: Rear speaker (second playback unit) 54: VR speaker (first playback unit) G1: Image R2: Area

Claims

1. A simulation device that plays back images recorded by an information recording device installed in a vehicle, A recognition unit that recognizes the road conditions in the image based on the image recorded by the information recording device, The system includes a control unit that controls the sound reproduced by the playback unit based on the road conditions in the aforementioned image, The recognition unit recognizes the condition of the road surface in the image by performing image recognition on the image recorded by the information recording device. A simulation device characterized by the following features.

2. The control unit controls the playback unit to output sounds in a frequency range not recorded by the information recording device. The simulation apparatus according to feature 1.

3. It is equipped with an input section for inputting information about the vehicle to be simulated, The control unit controls the playback unit to output sounds in frequency ranges not recorded by the information recording device, based on both the size of the space inside the vehicle being simulated and the road conditions in the image. The simulation apparatus according to feature 1.

4. The playback unit includes a first playback unit that outputs sounds in the frequency range recorded by the information recording device, and a second playback unit that outputs sounds in the frequency range not recorded by the information recording device. The simulation apparatus according to feature 2 or 3.

5. The recognition unit detects the area near the wheels in the direction of travel of the vehicle from the image recorded by the information recording device, and recognizes the condition of the road surface within the detected area. The simulation apparatus according to feature 1.

6. A recognition unit recognizes the road conditions in an image based on an image recorded by an information recording device installed in the vehicle, The system includes a distribution unit that transmits the recognition result from the recognition unit to a simulation device which has a control unit that plays back an image recorded by the information recording device and controls the sound played back by the playback unit based on the road conditions. A server device characterized by the following features.

7. A playback method for playing back images recorded by an information recording device installed in a vehicle, A receiving step of receiving the image from the information recording device, A recognition step of recognizing the road conditions in the image based on the received image, The system includes a control step that controls the sound to be reproduced by the playback unit based on the road conditions in the aforementioned image, The recognition step involves recognizing the condition of the road surface in the received image by performing image recognition on the image. A regeneration method characterized by the following.

Images