Information processing device, and control method thereof

Abstract

To provide an information processing device capable of reducing the storage capacity of a storage unit.SOLUTION: An information processing device 2 includes: a first acquisition unit 26 that acquires location information indicating the current position of a vehicle 4; a second acquisition unit 28 that acquires sensor information sensed around the vehicle 4 at the current position; a setting unit 30 that sets area information representing a predetermined area 46, which is input by a user 6 using the vehicle 4; a determination unit 32 that determines whether the vehicle 4 exists in the predetermined area 46 based on the location information and the area information; and a data storage unit 34 that stores the location information and the sensor information. The determination unit 32 is configured so as to, when determining that vehicle 4 is in the predetermined area 46, store the location information and the sensor information in the data storage unit 34 while associating the same with each other.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and a control method thereof. [Background technology]

[0002] A system is known that simulates driving a car by operating a model car (see, for example, Patent Document 1). In this system, a real field on which the model car runs is created in real space, and a virtual field similar to the real field is created in virtual space. When the model car is located on the real field, the virtual car is displayed on the virtual field based on location information indicating the current location of the model car.

[0003] In addition, a virtual camera is installed in the virtual car, and a CG (Computer Graphics) image obtained from the virtual camera is displayed on the display. This allows the user to operate the model car using the control device while looking at the CG image displayed on the display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-99790 Summary of the Invention [Problem to be solved by the invention]

[0005] There is a demand for a system that simulates driving an automobile using a real automobile instead of a model automobile. In this system, data acquired when an automobile is actually driven on a road or the like is stored in a memory unit, and a three-dimensional space model is generated based on the data stored in the memory unit.

[0006] However, in such a system, the amount of data stored in the storage unit becomes enormous, and a problem arises in that it takes an enormous amount of time to process the enormous amount of data.

[0007] Therefore, the present disclosure provides an information processing device and a control method thereof that can reduce the storage capacity of a storage unit. [Means for solving the problem]

[0008] An information processing device according to one embodiment of the present disclosure is an information processing device mounted on a vehicle, and includes: a first acquisition unit that acquires location information indicating the current location of the vehicle; a second acquisition unit that acquires sensor information obtained by sensing the area around the vehicle at the current location; a setting unit that sets area information indicating a predetermined area input by a user using the vehicle; a determination unit that determines whether the vehicle is present in the predetermined area based on the location information and the area information; and a memory unit that stores the location information and the sensor information.When the determination unit determines that the vehicle is present in the predetermined area, the location information and the sensor information are linked and stored in the memory unit.

[0009] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM (Compact Disc-Read Only Memory), or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] According to an information processing device or the like according to an aspect of the present disclosure, it is possible to reduce the storage capacity of the storage unit. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a conceptual diagram illustrating an application example of an information processing device according to an embodiment. [Figure 2]1 is a block diagram showing a functional configuration of an information processing device according to an embodiment; [Figure 3] FIG. 10 is a diagram showing an example of a predetermined area input by a user. [Figure 4] FIG. 1 is a diagram illustrating an example of a parking lot where vehicles are parked. [Figure 5] FIG. 1 is a diagram illustrating an example of a virtual parking lot in which a virtual vehicle is parked. [Figure 6] FIG. 1 is a diagram illustrating an example of a VR image displayed on a head-mounted display. [Figure 7] 10 is a flowchart showing a flow of generating a three-dimensional space model in the information processing device according to the embodiment. [Figure 8] 10 is a flowchart showing a flow of a simulation mode in the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Technology 1) An information processing device mounted on a vehicle, comprising: a first acquisition unit that acquires location information indicating the current location of the vehicle; a second acquisition unit that acquires sensor information obtained by sensing the area around the vehicle at the current location; a setting unit that sets area information indicating a predetermined area input by a user of the vehicle; a determination unit that determines whether the vehicle is present in the predetermined area based on the location information and the area information; and a memory unit that stores the location information and the sensor information, wherein when the determination unit determines that the vehicle is present in the predetermined area, the information processing device links the location information and the sensor information and stores them in the memory unit.

[0013] According to Technology 1, when the determination unit determines that the vehicle is present in a predetermined area, the determination unit associates the location information with the sensor information and stores the information in the storage unit. This makes it possible to prevent the location information and sensor information acquired when the vehicle is not present in the predetermined area from being stored in the storage unit unnecessarily, thereby reducing the storage capacity of the storage unit.

[0014] (Technology 2) The information processing device further includes a generation unit that generates a three-dimensional space model that virtually shows the three-dimensional space in which the vehicle actually traveled, based on the position information and the sensor information stored in the memory unit.

[0015] According to technique 2, a three-dimensional space model can be generated efficiently in a short time.

[0016] (Technology 3) The predetermined area is a parking lot where the vehicle is parked, and the second acquisition unit acquires an image showing the surroundings of the vehicle taken by a camera mounted on the vehicle as the sensor information, and when the determination unit determines that the vehicle is present in the parking lot, the determination unit associates the location information showing the location of the parking lot with the image showing the parking lot taken by the camera and stores them in the memory unit, and the generation unit generates the three-dimensional space model showing the virtual parking lot that is a real-life image of the parking lot, which is the three-dimensional space, based on the location information and the image stored in the memory unit, and the information processing device further includes a display control unit that displays, on a display device arranged in the vehicle, a VR (Virtual Reality) image showing the virtual parking lot as seen from the viewpoint of the user virtually riding in a virtual vehicle present in the virtual parking lot, based on the three-dimensional space model generated by the generation unit.

[0017] According to Technology 3, a user can drive a virtual vehicle in a virtual parking lot while viewing a VR image displayed on a display device, allowing the user to practice parking in a realistic manner, even if the user is a paper driver with little driving experience, without worrying about the risk of damaging property by hitting other parked vehicles.

[0018] (Technology 4) The information processing device according to technology 3, wherein the display device is a head-mounted display worn on the head of the user seated in the driver's seat of the vehicle.

[0019] According to Technique 4, a user can drive a virtual vehicle in a virtual parking lot and practice parking while viewing a VR image displayed on a head-mounted display.

[0020] (Technology 5) The information processing device further includes a calculation unit that calculates behavior information indicating the current position of the virtual vehicle in the virtual parking lot shown by the VR image and the orientation of the virtual vehicle at the current position based on the user's operation of at least one of the steering wheel, accelerator, brake, and shift lever of the vehicle while the vehicle is stopped and the VR image is displayed on the head-mounted display, and the display control unit changes the display area of ​​the VR image based on the behavior information calculated by the calculation unit.

[0021] According to Technology 5, a user can drive a virtual vehicle in a virtual parking lot by using at least one of the steering wheel, accelerator, brake, and shift lever of an actual vehicle as a controller, allowing for more realistic parking practice.

[0022] (Technology 6) The vehicle includes a drive mechanism for driving the vehicle, and the information processing device further includes a switching unit that switches between a driving mode in which the user drives the vehicle and a simulation mode in which the user simulates parking the virtual vehicle in the virtual parking lot using the vehicle and the head-mounted display while the vehicle is stopped, and the switching unit (i) connects the steering wheel, accelerator, brake, and shift lever of the vehicle to the drive mechanism in the driving mode, and (ii) disconnects the steering wheel, accelerator, brake, and shift lever of the vehicle from the drive mechanism in the simulation mode.

[0023] According to Technique 6, it is possible to prevent the actual vehicle from suddenly starting to move when the user operates the accelerator in the simulation mode, for example, and it is possible to execute the simulation mode while ensuring safety.

[0024] (Technology 7) The information processing device according to any one of techniques 3 to 6, wherein the display control unit further displays an obstacle image showing a virtual obstacle superimposed on the VR image.

[0025] Technique 7 allows users to practice parking simulations at a higher level of difficulty depending on their own driving skills.

[0026] (Technology 8) The information processing device according to any one of techniques 3 to 7, wherein the first acquisition unit acquires, via a network, location information indicating the current location of a vehicle other than the vehicle, and the second acquisition unit acquires, via the network, an image showing the surroundings of the other vehicle captured by a camera mounted on the other vehicle at the current location, as sensor information.

[0027] According to Technique 8, when the predetermined area is, for example, a public parking lot, the location information and sensor information can be efficiently stored in the storage unit in a short period of time.

[0028] (Technology 9) 9. The information processing device according to any one of techniques 3 to 8, wherein the display control unit corrects the image according to an image capturing condition when the surroundings of the vehicle are captured by the camera.

[0029] According to Technique 9, natural VR images obtained by pseudo-matching multiple shooting conditions taken by cameras at multiple points in time can be displayed on a display device such as a head-mounted display.

[0030] (Technology 10) A control method in an information processing device mounted on a vehicle, the control method including the steps of: acquiring location information indicating the current location of the vehicle; acquiring sensor information obtained by sensing the area around the vehicle at the current location; setting area information indicating a predetermined area input by a user using the vehicle; and determining whether the vehicle is present in the predetermined area based on the location information and the area information, and if it is determined that the vehicle is present in the predetermined area, linking the location information and the sensor information and storing them in a memory unit.

[0031] According to Technology 10, when it is determined that a vehicle is present in a predetermined area, the location information and the sensor information are linked and stored in a storage unit. This makes it possible to prevent the location information and the sensor information acquired when the vehicle is not present in the predetermined area from being accumulated in the storage unit unnecessarily, thereby reducing the storage capacity of the storage unit.

[0032] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, or a recording medium.

[0033] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0034] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components.

[0035] (Embodiment) [1. Configuration of information processing device] First, the configuration of an information processing device 2 according to an embodiment will be described with reference to FIGS. 1 to 6. FIG. 1 is a conceptual diagram showing an application example of the information processing device 2 according to an embodiment. FIG. 2 is a block diagram showing the functional configuration of the information processing device 2 according to an embodiment. FIG. 3 is a diagram showing an example of a predetermined area 46 input by a user 6. FIG. 4 is a diagram showing an example of a parking lot 50 in which a vehicle 4 is parked. FIG. 5 is a diagram showing an example of a virtual parking lot 56 in which a virtual vehicle 54 is parked. FIG. 6 is a diagram showing an example of a VR image 60 displayed on a head-mounted display 10.

[0036] As shown in FIG. 1, an information processing device 2 is mounted on a vehicle 4 such as an automobile, and is a device for assisting a user 6 who uses the vehicle 4 in practicing parking.

[0037] Rather than actually driving the vehicle 4 in a parking lot 50 (see Figure 4 described below) to practice parking, the user 6 parks the vehicle 4 in a safe location of their choice, sits in the driver's seat 8 of the vehicle 4, and wears a head-mounted display 10 (an example of a display device) located inside the vehicle 4 on their head to practice parking the vehicle 4 in the parking lot 50.

[0038] The head-mounted display 10 is connected by wire or wirelessly to the information processing device 2. A VR (Virtual Reality) image 60 (see FIG. 6, which will be described later) showing a virtual parking lot 56 (see FIG. 5, which will be described later) that is a realistic image of an actual parking lot 50 is displayed on the head-mounted display 10. In this specification, the term "image" is a concept that includes not only still images but also moving images.

[0039] This allows the user 6 to use the steering wheel 18 or the like (described later) of the vehicle 4 parked at any location as a controller to drive the virtual vehicle 54 in which the user 6 is virtually riding in a virtual parking lot 56 while looking at the VR image 60 displayed on the head-mounted display 10, thereby practicing parking.

[0040] As shown in FIG. 2, the vehicle 4 is equipped with a camera 12, a center display 14, a drive mechanism 16, a steering wheel 18, an accelerator 20, a brake 22, and a shift lever 24.

[0041] The camera 12 is a camera sensor for capturing images showing the surroundings of the vehicle 4. Specifically, the camera 12 includes, for example, a front camera, a rear camera, a left side camera, and a right side camera. The front camera is disposed at the front of the vehicle 4 and captures a front image showing the front of the vehicle 4. The rear camera is disposed at the rear of the vehicle 4 and captures a rear image showing the rear of the vehicle 4. The left side camera is disposed on the left side of the vehicle 4 and captures a left side image showing the left side of the vehicle 4. The right side camera is disposed on the right side of the vehicle 4 and captures a right side image showing the right side of the vehicle 4. In this specification, "front," "rear," "left," and "right" respectively mean the front, rear, left, and right as seen from the user 6 seated in the driver's seat 8 of the vehicle 4 facing forward.

[0042] The center display 14 is disposed, for example, on the center console of the vehicle 4, and displays a car navigation map, an audio menu screen, and the like.

[0043] The drive mechanism 16 is a mechanism for driving the vehicle 4 in response to the operation of the steering wheel 18, the accelerator 20, the brake 22, and the shift lever 24. The drive mechanism 16 is mechanically or electrically connected to each of the steering wheel 18, the accelerator 20, the brake 22, and the shift lever 24.

[0044] The steering wheel 18 is rotatably supported on a steering column arranged in the driver's seat 8 of the vehicle 4, and is gripped by a user 6 seated in the driver's seat 8. When the user 6 rotates the steering wheel 18, the driving mechanism 16 changes the traveling direction of the vehicle 4.

[0045] The accelerator 20 is an accelerator pedal located at the feet of the driver's seat 8 of the vehicle 4, and is operated by the foot of a user 6 seated in the driver's seat 8. When the user 6 operates the accelerator 20, the drive mechanism 16 accelerates the vehicle 4.

[0046] The brake 22 is a brake pedal located at the feet of the driver's seat 8 of the vehicle 4, and is operated by the foot of a user 6 seated in the driver's seat 8. When the user 6 operates the brake 22, the drive mechanism 16 decelerates the vehicle 4.

[0047] The shift lever 24 is disposed on a center console or the like of the vehicle 4, and is manually operated by a user 6 seated in the driver's seat 8. For example, when the user 6 operates the accelerator 20 with the shift position of the shift lever 24 in the D range, the vehicle 4 moves forward by the drive mechanism 16. When the user 6 operates the accelerator 20 with the shift position of the shift lever 24 in the R range, the vehicle 4 moves backward by the drive mechanism 16.

[0048] As shown in FIG. 2, the information processing device 2 includes a first acquisition unit 26, a second acquisition unit 28, a setting unit 30, a determination unit 32, a data storage unit 34 (an example of a storage unit), a generation unit 36, a three-dimensional space model storage unit 38, a switching unit 40, a calculation unit 42, and a display control unit 44.

[0049] The first acquisition unit 26 acquires position information indicating the current position of the vehicle 4 and the orientation of the vehicle 4 at the current position, for example, by using a GPS (Global Positioning System). The first acquisition unit 26 constantly acquires position information while the vehicle 4 is being driven. The current position of the vehicle 4 is expressed, for example, by latitude and longitude. The current orientation of the vehicle 4 is expressed, for example, by azimuth.

[0050] The second acquisition unit 28 acquires sensor information obtained by sensing the surroundings of the vehicle 4 at its current location. Specifically, the second acquisition unit 28 acquires, as sensor information, an image showing the surroundings of the vehicle 4 captured by the camera 12 mounted on the vehicle 4. The second acquisition unit 28 constantly acquires sensor information while the vehicle 4 is operating.

[0051] The setting unit 30 sets area information indicating a predetermined area 46 input by the user 6. The predetermined area 46 is a parking lot 50 where the user 6 wants to practice parking, such as a home parking lot, a monthly parking lot, an hourly parking lot, or a parking lot at a commercial facility that the user 6 frequently uses. The area information is information indicating the location of the predetermined area 46, such as information indicating the latitude and longitude range of the predetermined area 46.

[0052] As shown in FIG. 3 , the user 6 inputs the predetermined area 46 by using, for example, the center display 14 as a user interface. Specifically, the user 6 inputs the predetermined area 46 into the center display 14 by tracing the predetermined area 46 where the user 6 wants to perform the parking practice simulation with his / her finger on a map 48 displayed on the center display 14. At this time, the map 48 displayed on the center display 14 may be a map for car navigation or a map dedicated to inputting the predetermined area 46. Alternatively, the user 6 may input the predetermined area 46 by inputting the address or postal code of the predetermined area 46 where the user 6 wants to perform the parking practice simulation into the center display 14. Note that in the present embodiment, the case where the user 6 inputs one predetermined area 46 has been described. However, the present invention is not limited to this. If there are multiple parking lots where the user 6 wants to perform the parking practice simulation, the user 6 may input multiple predetermined areas 46 (for example, a total of two parking lots, including a home parking lot and a parking lot at a commercial facility).

[0053] The determination unit 32 determines whether the vehicle 4 is currently located in a predetermined area 46, based on the location information acquired by the first acquisition unit 26 and the area information set by the setting unit 30. If the determination unit 32 determines that the vehicle 4 is currently located in the predetermined area 46, the determination unit 32 associates the location information currently acquired by the first acquisition unit 26 with the sensor information currently acquired by the second acquisition unit 28, and stores the associated information in the data storage unit 34. That is, the determination unit 32 associates the location information and sensor information acquired at the same time, and stores the associated information in the data storage unit 34.

[0054] 4(a) and 4(b), when the user 6 parks the vehicle 4 in a parking lot 50, the determination unit 32 determines, based on the location information and the area information, that the vehicle 4 is currently located in the parking lot 50 that is set as the predetermined area 46. The determination unit 32 then associates the location information indicating the location of the parking lot 50 with an image of the parking lot 50 taken by the camera 12, and stores the image in the data storage unit 34.

[0055] Note that (a) in Figure 4 shows the state in which vehicle 4 is moving forward toward parking space 52 in parking lot 50, and (b) in Figure 4 shows the state in which vehicle 4 is turning right to park forward into parking space 52 in parking lot 50.

[0056] In addition, if the judgment unit 32 determines that the vehicle 4 is not currently present in the specified area 46, it discards the location information currently acquired by the first acquisition unit 26 and the sensor information currently acquired by the second acquisition unit 28 without storing them in the data storage unit 34.

[0057] The position information and sensor information acquired at the same time as described above are linked and stored in the data storage unit 34. When the vehicle 4 is parked multiple times in a parking lot 50 set as a predetermined area 46, the position information and sensor information are linked and stored in the data storage unit 34 for each parking. As a result, the data storage unit 34 accumulates multiple data sets each consisting of linked position information and sensor information.

[0058] The generation unit 36 ​​generates a three-dimensional space model that virtually shows, as a three-dimensional space, the parking lot 50 (predetermined area 46) through which the vehicle 4 has actually traveled, based on a plurality of data sets (i.e., a plurality of sets of position information and sensor information) stored in the data storage unit 34. Specifically, the generation unit 36 ​​generates a three-dimensional space model that shows a virtual parking lot 56 that is a real image of the parking lot 50, which is a three-dimensional space, based on the plurality of data sets stored in the data storage unit 34. The generation unit 36 ​​uses, for example, SfM (Structure from Motion) or the like to generate the three-dimensional space model.

[0059] The three-dimensional space model storage unit 38 stores the three-dimensional space model generated by the generation unit 36.

[0060] The switching unit 40 switches the operation mode of the vehicle 4 between a driving mode and a simulation mode based on an instruction from the user 6. The driving mode is a mode in which the user 6 actually drives the vehicle 4. The simulation mode is a mode in which, with the vehicle 4 stopped at a desired location, the user 6 uses the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4 as controllers to simulate parking a virtual vehicle 54 in a virtual parking lot 56, as shown in (a) and (b) of FIG. 5, while viewing a VR image 60 displayed on a head-mounted display 10 worn on the user's head.

[0061] 5(a) shows a state in which the virtual vehicle 54 is moving forward toward a parking space 58 in a virtual parking lot 56, and FIG. 5(b) shows a state in which the virtual vehicle 54 is turning right to park forward into the parking space 58 in the virtual parking lot 56. In addition, in FIGS. 5(a) and 5(b), a triangle with a dotted pattern represents the field of view of the user 6 virtually riding in the virtual vehicle 54. That is, a VR image 60 of this field of view is displayed on the head-mounted display 10.

[0062] In the driving mode, the switching unit 40 mechanically or electrically connects the steering wheel 18, the accelerator 20, the brake 22, and the shift lever 24 of the vehicle 4 to the drive mechanism 16, respectively.

[0063] Meanwhile, in the simulation mode, the switching unit 40 cuts off the mechanical or electrical connection between the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4 and the drive mechanism 16. Also, in the simulation mode, the switching unit 40 outputs operation signals from the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4 to the calculation unit 42 via, for example, a CAN (Controller Area Network). Note that the operation signals are signals indicating, for example, (a) the operation direction and operation amount of the steering wheel 18, (b) the operation amounts of the accelerator 20 and the brake 22, and (c) the operation position of the shift lever 24.

[0064] In the simulation mode, when the user 6 operates at least one of the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4, the calculation unit 42 calculates behavior information indicating the current position of the virtual vehicle 54 in the virtual parking lot 56 shown by the VR image 60 and the orientation of the virtual vehicle 54 at the current position based on an operation signal from at least one of the steering wheel 18, accelerator 20, brake 22, and shift lever 24.

[0065] 6(a) and 6(b), based on the three-dimensional space model stored in the three-dimensional space model storage unit 38, the display control unit 44 displays, on the head-mounted display 10 worn on the head of the user 6, a VR image 60 showing the virtual parking lot 56 as seen from the viewpoint of the user 6 virtually riding in a virtual vehicle 54 located in the virtual parking lot 56. The VR image 60 is a three-dimensional image showing the virtual parking lot 56 that is a realistic image of the actual parking lot 50, and is, for example, a celestial sphere image or a hemispherical image.

[0066] 6(a) and 6(b), the display control unit 44 displays a vehicle interior image 62 showing the interior of the vehicle 4, which is captured in real time by a camera (not shown) mounted on the head-mounted display 10, superimposed on the VR image 60. As a result, the vehicle interior image 62 showing the instrument panel and the like in the line of sight of the user 6 seated in the driver's seat 8 of the vehicle 4 in the interior of the actual vehicle 4 is displayed superimposed on the VR image 60, allowing the user 6 to feel a sense of reality as if he or she were actually driving the virtual vehicle 54.

[0067] Note that (a) of Figure 6 shows a VR image 60 in a state where the virtual vehicle 54 is moving forward toward a parking space 58 in a virtual parking lot 56, and corresponds to the state shown in (a) of Figure 5. Also, (b) of Figure 6 shows a VR image 60 in a state where the virtual vehicle 54 is turning right to park forward in a parking space 58 in a virtual parking lot 56, and corresponds to the state shown in (b) of Figure 5.

[0068] In addition, based on the behavior information calculated by the calculation unit 42, the display control unit 44 changes the display area of ​​the VR image 60 displayed on the head-mounted display 10 in accordance with the current position and orientation of the virtual vehicle 54 indicated by the behavior information.

[0069] For example, when the user 6 operates the accelerator 20 with the shift lever 24 in the D range to move the virtual vehicle 54 forward in the virtual parking lot 56, the display control unit 44 changes the display area of ​​the VR image 60 to the forward direction of the virtual vehicle 54, as shown in (a) of Figure 6.

[0070] Furthermore, when the user 6 turns the steering wheel 18 to the right while operating the accelerator 20 to turn the virtual vehicle 54 right in the virtual parking lot 56, the display control unit 44 changes the display area of ​​the VR image 60 to the right turning direction of the virtual vehicle 54, as shown in (b) of Figure 6.

[0071] [2. Operation of information processing device] Next, a flow of generating a three-dimensional space model will be described with reference to Fig. 4 and Fig. 7. Fig. 7 is a flowchart showing a flow of generating a three-dimensional space model in the information processing device 2 according to the embodiment.

[0072] As shown in FIG. 7, first, the user 6 sits in the driver's seat 8 of the vehicle 4 and turns on the power supply (for example, the ignition power supply) of the vehicle 4 (S101).

[0073] Next, the switching unit 40 switches the operation mode of the vehicle 4 to the driving mode based on an instruction from the user 6 (S102). At this time, the switching unit 40 mechanically or electrically connects the steering wheel 18, the accelerator 20, the brake 22, and the shift lever 24 of the vehicle 4 to the drive mechanism 16.

[0074] Next, the user 6 inputs a predetermined area 46 (e.g., a parking lot 50) where the user 6 would like to perform the parking simulation practice into the center display 14. As a result, the setting unit 30 sets area information indicating the predetermined area 46 input by the user 6 (S103).

[0075] Next, the user 6 actually drives the vehicle 4 (S104). While the user 6 is driving the vehicle 4, the first acquisition unit 26 acquires position information, and the second acquisition unit 28 acquires sensor information (S105).

[0076] Next, the determination unit 32 determines whether the vehicle 4 is currently located in a predetermined area 46 based on the location information acquired by the first acquisition unit 26 and the area information set by the setting unit 30 (S106).

[0077] For example, if the vehicle 4 is traveling somewhere other than the parking lot 50 and is not currently in the specified area 46 (NO in S106), the judgment unit 32 discards the location information currently acquired by the first acquisition unit 26 and the sensor information currently acquired by the second acquisition unit 28 without storing them in the data storage unit 34 (S107).

[0078] On the other hand, for example, as shown in (a) and (b) of Figures 4, if the vehicle 4 is attempting to park in a parking lot 50 and is currently located in a predetermined area 46 (YES in S106), the judgment unit 32 links the location information currently acquired by the first acquisition unit 26 with the sensor information currently acquired by the second acquisition unit 28 and stores them in the data storage unit 34 (S108).

[0079] After step S107 or S108, the generation unit 36 ​​determines whether a predetermined number of pieces of position information and sensor information required to generate a three-dimensional space model have been stored in the data storage unit 34 (S109).

[0080] If the predetermined number of pieces of position information and sensor information required for generating a three-dimensional space model have not been stored in the data storage unit 34 (NO in S109), the process proceeds to step S110. Thereafter, if driving of the vehicle 4 is to be continued (NO in S110), the process returns to step S105, and steps S105 to S109 are executed again as described above. If driving of the vehicle 4 is to be ended (YES in S110), the process of the flowchart in FIG. 7 ends.

[0081] Returning to step S109, if the predetermined number of pieces of position information and sensor information required to generate the three-dimensional space model have been stored in the data storage unit 34 (YES in S109), the generation unit 36 ​​generates a three-dimensional space model based on the predetermined number of pieces of position information and sensor information stored in the data storage unit 34 (S111). Next, the generation unit 36 ​​stores the generated three-dimensional space model in the three-dimensional space model storage unit 38 (S112), and then proceeds to step S110.

[0082] In this way, by driving the vehicle 4 in a predetermined area 46 multiple times while the user 6 is driving the vehicle 4 in his or her daily life, a three-dimensional space model can be efficiently generated without the user 6 being aware of it (without knowing).

[0083] Next, the flow of the simulation mode will be described with reference to Fig. 5, Fig. 6 and Fig. 8. Fig. 8 is a flowchart showing the flow of the simulation mode in the information processing device 2 according to the embodiment.

[0084] As shown in FIG. 8, first, the user 6 stops the vehicle 4 in a safe place (e.g., a home parking lot, etc.), sits in the driver's seat 8 of the vehicle 4, and turns on the power supply (e.g., accessory power supply) of the vehicle 4 (S201).

[0085] Next, the switching unit 40 switches the operation mode of the vehicle 4 to the simulation mode based on an instruction from the user 6 (S202). At this time, the switching unit 40 cuts off the mechanical or electrical connection between the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4 and the drive mechanism 16. As a result, in the simulation mode, the vehicle 4 is held in a stopped state.

[0086] Next, the user 6 wears the head-mounted display 10 on his / her head (S203). Next, as shown in (a) of Fig. 6, the display control unit 44 causes the head-mounted display 10 to display a VR image 60 showing a virtual parking lot 56 (S204).

[0087] Next, the user 6 operates the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4 (S205), thereby driving the virtual vehicle 54 that the user is virtually riding in in the virtual parking lot 56, as shown in (a) of Figure 5, and practicing parking. As described above, the switching unit 40 cuts off the mechanical or electrical connection between the steering wheel 18, accelerator 20, brake 22, and shift lever 24 of the vehicle 4 and the drive mechanism 16, so that even if the user 6 operates the accelerator 20, for example, the vehicle 4 will not start moving unexpectedly.

[0088] In addition, in order to further enhance the realism of the driving sensation experienced by the user 6, for example, when the user 6 rotates each of the steering wheel 18, accelerator 20, and brake 22, a repulsive force may be applied to each of the steering wheel 18, accelerator 20, and brake 22 by an actuator or the like.

[0089] Next, the calculation unit 42 calculates behavior information based on operation signals from the steering wheel 18, the accelerator 20, the brake 22, and the shift lever 24 (S206).

[0090] Next, the display control unit 44 changes the display area of ​​the VR image 60 displayed on the head-mounted display 10 in accordance with the current position and orientation of the virtual vehicle 54 indicated by the behavior information calculated by the calculation unit 42 (S207).

[0091] For example, when the user 6 operates the accelerator 20 with the shift lever 24 in the D range, the virtual vehicle 54 moves forward toward a parking space 58 in a virtual parking lot 56, as shown in (a) of Fig. 5. Accordingly, the display area of ​​the VR image 60 changes so that the virtual vehicle 54 moves forward in the virtual parking lot 56, as shown in (a) of Fig. 6.

[0092] Furthermore, when the user 6 operates the accelerator 20 while rotating the steering wheel 18 rightward, the virtual vehicle 54 turns right to park forward into a parking space 58 in the virtual parking lot 56, as shown in (b) of Fig. 5. Accordingly, the display area of ​​the VR image 60 changes so that the virtual vehicle 54 turns right in the virtual parking lot 56, as shown in (b) of Fig. 6.

[0093] Thereafter, if the simulation mode is to be continued (NO in S208), the process returns to step S205, and steps S205 to S208 are executed again in the same manner as described above.

[0094] On the other hand, if the simulation mode is to be ended (YES in S208), the processing of the flowchart in FIG. 8 is ended.

[0095] [3.Effects] In this embodiment, when the determination unit 32 determines that the vehicle 4 is currently located in the predetermined area 46, the determination unit 32 associates the position information with the sensor information and stores them in the data storage unit 34. This makes it possible to prevent the position information and sensor information acquired when the vehicle 4 is not located in the predetermined area 46 from being stored in the data storage unit 34 unnecessarily, thereby reducing the storage capacity of the data storage unit 34.

[0096] In the simulation mode, the user 6 uses the steering wheel 18 or the like of the actual vehicle 4 parked at a desired location as a controller to drive the virtual vehicle 54 in the virtual parking lot 56 while viewing the VR image 60 displayed on the head-mounted display 10 to practice parking. This allows even the user 6, who is a paper driver with little driving experience, to practice parking with a high level of reality without worrying about the risk of damaging property by hitting other parked vehicles, etc. As a result, the user 6's parking skills can be efficiently improved.

[0097] (Other variations) While the information processing device and the control method thereof according to one or more aspects have been described based on the above-mentioned embodiments, the present disclosure is not limited to the above-mentioned embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the above-mentioned embodiments and forms constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0098] For example, in the above embodiment, the display device for displaying the VR image 60 is configured as the head-mounted display 10, but this is not limited thereto, and the display device for displaying the VR image 60 may be configured as at least one of the front window, side window, rear window, room mirror, and side mirror mounted (disposed) on the vehicle 4. This allows the user 6 to practice parking while viewing the VR image displayed on at least one of the front window, side window, rear window, room mirror, and side mirror. In this case, the user 6 does not need to wear the head-mounted display 10 on his / her head.

[0099] Further, for example, in the above embodiment, the display control unit 44 displays the VR image 60 on the head-mounted display 10, but an obstacle image showing a virtual obstacle (e.g., another parked vehicle) may be superimposed on the VR image 60 and displayed. The obstacle image is, for example, a CG image, and the type and display location of the obstacle image are set arbitrarily by the user 6. Furthermore, the display control unit 44 may display the obstacle image so that it moves on the VR image 60 (e.g., so that other parked vehicles start moving). This allows the user 6 to practice parking simulations with a higher level of difficulty.

[0100] For example, in the above embodiment, the first acquisition unit 26 acquires location information indicating the current location of the vehicle 4. However, this is not limited to this. Location information indicating the current location of vehicles other than the vehicle 4 may be acquired via a network such as the Internet. In the above embodiment, the second acquisition unit 28 acquires, as sensor information, images showing the surroundings of the vehicle 4 captured by the camera 12 mounted on the vehicle 4. However, this is not limited to this. Images showing the surroundings of other vehicles captured by cameras mounted on vehicles other than the vehicle 4 may be acquired as sensor information via a network such as the Internet. In this case, information indicating the optical parameters and mounting positions of the cameras mounted on the other vehicles is managed as auxiliary information, and differences in the imaging conditions between the cameras of each vehicle are calibrated. This allows the location information and sensor information to be efficiently accumulated in the data storage unit 34 in a short period of time, for example, if the predetermined area 46 is a public parking lot. The three-dimensional space model generated based on the location information and sensor information accumulated in this manner can be sold, for example, to a service provider.

[0101] Furthermore, for example, the display control unit 44 may correct the image captured by the camera 12 according to the shooting conditions (e.g., weather, time of day, etc.) when the camera 12 captured the surroundings of the vehicle 4. When correcting the image, for example, (a) a correction formula may be derived from the feature amount of the image, (b) a correction formula may be derived based on the shooting conditions of the same subject, or (c) the image may be corrected to match the shooting conditions using a generating AI (artificial intelligence). In this way, a natural VR image 60 in which multiple shooting conditions captured by the camera 12 at multiple points in time are pseudo-matched can be displayed on the head-mounted display 10.

[0102] For example, in the above embodiment, the second acquisition unit 28 acquired an image showing the surroundings of the vehicle 4 taken by the camera 12 mounted on the vehicle 4 as sensor information, but this is not limited to this, and the sensor information may also include the distance between the vehicle 4 and objects around the vehicle 4 sensed by a ranging sensor such as sonar or radar mounted on the vehicle 4.

[0103] In the above-described embodiments, each component may be configured with dedicated hardware, or may be realized by executing a computer program suitable for each component. Each component may be realized by a program execution unit such as a CPU (Central Processing Unit) or a processor reading and executing a computer program recorded on a recording medium such as a hard disk or semiconductor memory.

[0104] Furthermore, some or all of the functions of the information processing device according to the above-described embodiment may be realized by a processor such as a CPU executing a computer program.

[0105] Some or all of the components constituting each of the above devices may be configured as an IC card or a standalone module that can be attached to each device. The IC card or module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or module may include the above-mentioned ultra-multifunctional LSI. The IC card or module achieves its functions when the microprocessor operates according to a computer program. The IC card or module may be tamper-resistant.

[0106] The present disclosure may be the method described in Technology 9 above. It may also be a computer program for implementing these methods on a computer, or a digital signal including the computer program. The present disclosure may also be a computer program or a digital signal recorded on a computer-readable non-transitory recording medium, such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray (registered trademark) Disc), semiconductor memory, etc. It may also be the digital signal recorded on such a recording medium. The present disclosure may also be a computer program or a digital signal transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc. The present disclosure may also be a computer system including a microprocessor and a memory, in which the memory stores the computer program, and the microprocessor operates according to the computer program. The present disclosure may also be implemented by another independent computer system by recording the computer program or the digital signal on the recording medium and transferring it, or by transferring the computer program or the digital signal via the network, etc. [Industrial Applicability]

[0107] The present disclosure is applicable to, for example, an information processing device for generating a three-dimensional space model used in a parking simulation practice. [Explanation of symbols]

[0108] 2. Information processing equipment 4 vehicles 6 users 8 Driver's seat 10 Head-mounted display 12 Camera 14 Center display 16 Drive mechanism 18 Steering Wheel 20 Axel 22 Brake 24 Shift lever 26 First Acquisition Section 28 Second Acquisition Section 30 Setting section 32 Judgment section 34 Data storage unit 36 Generation part 38 Three-dimensional space model memory section 40 Switching section 42 Calculation section 44 Display control unit 46 designated area 48 Map 50 Parking 52,58 parking spaces 54 Virtual Vehicles 56 Virtual Parking Lot 60 VR images 62 Car interior images

Claims

1. An information processing device mounted on a vehicle, a first acquisition unit that acquires location information indicating a current location of the vehicle; a second acquisition unit that acquires sensor information obtained by sensing the surroundings of the vehicle at the current location; a setting unit that sets area information indicating a predetermined area input by a user who uses the vehicle; a determination unit that determines whether the vehicle is present in the predetermined area based on the location information and the area information; a storage unit that stores the position information and the sensor information, When the determination unit determines that the vehicle is present in the predetermined area, the determination unit associates the location information with the sensor information and stores the information in the storage unit. Information processing device.

2. The information processing device further includes a generation unit that generates a three-dimensional space model that virtually represents a three-dimensional space in which the vehicle actually travels, based on the position information and the sensor information stored in the storage unit. The information processing device according to claim 1 .

3. the predetermined area is a parking lot where the vehicle is parked, The second acquisition unit acquires, as the sensor information, an image showing the surroundings of the vehicle captured by a camera mounted on the vehicle, When the determination unit determines that the vehicle is present in the parking lot, the determination unit associates the location information indicating the location of the parking lot with the image of the parking lot taken by the camera and stores the associated image in the storage unit; the generation unit generates the three-dimensional space model showing the virtual parking lot, which is a real image of the parking lot that is the three-dimensional space, based on the position information and the image stored in the storage unit; The information processing device further includes a display control unit that displays, on a display device disposed in the vehicle, a VR (Virtual Reality) image showing the virtual parking lot seen from a viewpoint of the user virtually riding in a virtual vehicle present in the virtual parking lot, based on the three-dimensional space model generated by the generation unit. The information processing device according to claim 2 .

4. The display device is a head-mounted display that is worn on the head of the user seated in the driver's seat of the vehicle. The information processing device according to claim 3 .

5. The information processing device further includes a calculation unit that calculates behavior information indicating a current position of the virtual vehicle in the virtual parking lot shown by the VR image and an orientation of the virtual vehicle at the current position based on an operation of at least one of a steering wheel, an accelerator, a brake, and a shift lever of the vehicle by the user while the vehicle is stopped and the VR image is displayed on the head-mounted display, The display control unit changes a display area of ​​the VR image based on the behavior information calculated by the calculation unit. The information processing device according to claim 4 .

6. the vehicle includes a drive mechanism for driving the vehicle; The information processing device further includes a switching unit that switches between a driving mode in which the user drives the vehicle and a simulation mode in which the user simulates parking the virtual vehicle in the virtual parking lot using the vehicle and the head-mounted display while the vehicle is stopped, The switching unit (i) connects the steering wheel, the accelerator, the brake, and the shift lever of the vehicle to the drive mechanism in the driving mode, and (ii) disconnects the steering wheel, the accelerator, the brake, and the shift lever of the vehicle from the drive mechanism in the simulation mode. The information processing device according to claim 5 .

7. The display control unit further displays an obstacle image showing a virtual obstacle superimposed on the VR image. The information processing device according to any one of claims 3 to 6.

8. the first acquisition unit acquires, via a network, location information indicating a current location of a vehicle other than the vehicle; The second acquisition unit acquires, via the network, an image showing the surroundings of the other vehicle captured by a camera mounted on the other vehicle at the current location, as sensor information. The information processing device according to any one of claims 3 to 6.

9. The display control unit corrects the image according to a photographing condition when the surroundings of the vehicle are photographed by the camera. The information processing device according to any one of claims 3 to 6.

10. A control method for an information processing device mounted on a vehicle, comprising: acquiring location information indicating a current location of the vehicle; acquiring sensor information obtained by sensing the surroundings of the vehicle at the current position; setting area information indicating a predetermined area input by a user of the vehicle; determining whether the vehicle is present in the predetermined area based on the location information and the area information, and if it is determined that the vehicle is present in the predetermined area, associating the location information with the sensor information and storing the information in a storage unit. Control method.

Images