Control device, control system, control method, and control program

Abstract

To provide a control device capable of suppressing the deterioration of the reality of XR space in a moving body.SOLUTION: A disclosed control device is a control device that outputs control signals for reproducing the state in the virtual space experienced by a user on board a moving body in a moving body, which includes an acquisition unit and an output unit. The acquisition unit acquires ambient information about the surrounding environment in which the user actually exists. The output unit outputs control signals in which the deterioration of the reproducibility in the virtual space due to the surrounding environment indicated by the surrounding information acquired by the acquisition unit is suppressed.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a control device and a control system. 、 Control Method and control program Regarding. [Background technology]

[0002] Conventionally, there is known technology that provides users with digital content that includes virtual space experiences such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), known as XR (Cross Reality) content, using devices such as HMDs (Head Mounted Displays). XR is a collective term that encompasses all virtual space technologies, including VR, AR, and MR, as well as SR (Substitutional Reality) and AV (Audio / Visual).

[0003] In addition, in this technology, an XR system has also been proposed that is mounted on a moving body such as a vehicle and that can use the moving body as a motion platform (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-102401

[0005] However, conventional technology does not take into consideration, for example, the use of such a platform as a motion platform when a moving object is actually running, and if it were to be applied while running, the reality of the XR space inside the moving object would be reduced.

[0006] The present invention has been made in view of the above, and aims to provide a control device, a control system, and a control method that can prevent a decrease in the reality of an XR space inside a moving body. Summary of the Invention [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, a control device according to the present invention is a control device that outputs a control signal for reproducing, within a moving body, a state in a virtual space experienced by a user riding on the moving body, and includes an acquisition unit and an output unit. The acquisition unit acquires ambient information regarding the surrounding environment in which the user actually exists. The output unit outputs the control signal in such a way that a reduction in the degree of reproduction of the virtual space due to the ambient environment indicated by the ambient information acquired by the acquisition unit is suppressed. [Effects of the Invention]

[0008] According to the present invention, it is possible to prevent a decrease in the reality of the XR space inside a moving body. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an outline of the control process. [Figure 2] FIG. 2 is a diagram illustrating an outline of the control process. [Figure 3] FIG. 3 is a block diagram of the in-vehicle device. [Figure 4] FIG. 4 is a schematic diagram of the correction process performed by the correction unit. [Figure 5] FIG. 5 is a flowchart showing a processing procedure executed by the in-vehicle device. [Figure 6] FIG. 6 is a flowchart showing the processing procedure of step S104 shown in FIG. [Figure 7] FIG. 7 is a block diagram of a receiving device according to the second embodiment. [Figure 8] FIG. 8 is a flowchart showing a processing procedure executed by the device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a control device, a control system, and a control method disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0011] First, an overview of a control method according to an embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 and Fig. 2 are diagrams showing an overview of the control method. Note that the following describes a case where the control device is an in-vehicle device 10 mounted on a vehicle. Also, the following describes a case where the XR space (virtual space) is a VR space.

[0012] As shown in FIG. 1, the control system 1 includes an HMD 3, an in-vehicle device 10, a seat control device 50, and a server device 100.

[0013] The HMD 3 is an information processing terminal that presents XR content provided by the in-vehicle device 10 to the user U, allowing the user to enjoy a VR experience. The HMD 3 is a wearable computer that is worn on the head of the user U, and is goggle-shaped in the example of FIG. 1. The HMD 3 may be in the form of glasses or a hat.

[0014] The in-vehicle device 10 is, for example, a computer mounted on the vehicle V, and is connected to the seat control device 50 by wire or wirelessly, and outputs a control signal to the seat control device 50.

[0015] The seat control device 50 is, for example, a six-axis motion base, and controls the posture of the seat S installed in the vehicle in accordance with the situation in the virtual space by being driven in response to a control signal input from the in-vehicle device 10. For example, the seat control device 50 reproduces tilt, acceleration, vibration, etc. in accordance with the state of the user U in the virtual space.

[0016] The server device 100 is, for example, a cloud system, and distributes various XR contents to the HMD 3 and the in-vehicle device 10. The example in FIG. 1 shows a case where a user U operates a controller (not shown) to operate a robot R located in a remote location. In this case, the server device 100 controls the robot R in response to the operation of the user U, and acquires actual images seen from the robot R and changes in the posture of the robot R.

[0017] The server device 100 transmits the image and posture information received from the robot R to the in-vehicle device 10, and the in-vehicle device 10 controls the HMD 3 and the seat control device 50 to provide a VR space in which the user U appears as if he or she were the robot R. Note that the robot R may be, for example, an avatar existing in the virtual space.

[0018] However, if the in-vehicle device 10 simply controls the seat control device 50 in accordance with the state of the robot R while the vehicle V is traveling, the movement of the vehicle V will also affect the user U, causing a discrepancy between the image displayed on the HMD 3 and the body movement of the user U, reducing the degree of realism of the VR space. Furthermore, the discrepancy between the image and the body movement of the user U may cause XR sickness, which is similar to motion sickness.

[0019] Therefore, in the control method according to the embodiment, the surrounding environment of the user U is detected, and a control signal that suppresses a decrease in the degree of reproduction in the VR space due to the influence of the surrounding environment is output to the seat control device 50. Specifically, as shown in Fig. 2, the in-vehicle device 10 acquires surrounding information of the user U who is riding in the vehicle V (step S1).

[0020] The surrounding information is information about the surrounding environment of the user U, for example, driving information (for example, CAN (Controller Area Network) data) indicating the driving state of the vehicle V. Next, the in-vehicle device 10 outputs a control signal that suppresses a decrease in the degree of reproduction in the VR space according to the surrounding environment indicated by the surrounding information (step S2).

[0021] For example, the in-vehicle device 10 estimates the attitude of the vehicle V (i.e., the attitude of the user U) from ambient information. Here, the attitude is a concept that includes, for example, the tilt of the vehicle V, acceleration and vibration occurring in the vehicle V, etc. The in-vehicle device 10 also acquires attitude information regarding the attitude of the robot R, for example, from the server device 100. The in-vehicle device 10 compares the estimated attitude of the user U with the attitude of the robot R, corrects the control signal so that the attitude of the user U matches the attitude of the robot R, and outputs the corrected control signal to the seat control device 50.

[0022] As a result, the seat control device 50, by being driven in response to the control signal, can reduce the discrepancy between the image displayed on the HMD 3 and the posture of the user U. To give a more specific example, when the vehicle V moves backward while the robot R moves forward, the in-vehicle device 10 slides the seat control device 50 forward.

[0023] In this way, the control method according to the embodiment can suppress the deviation between the real world and the virtual space, thereby suppressing the decrease in the degree of realism of the VR space inside the moving body.

[0024] Next, a configuration example of the in-vehicle device 10 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram of the in-vehicle device 10. As shown in Fig. 3, the in-vehicle device 10 according to the embodiment includes a storage unit 11 and a control unit 12. In addition, various sensors 5, a seat control device 50, and an electronic device 60 are connected to the in-vehicle device 10.

[0025] The various sensors 5 are a group of sensors that sense the conditions inside and outside the vehicle, and include, for example, a camera 5a, an accelerator sensor 5b, an acceleration sensor 5c, a steering angle sensor 5d, and the like.

[0026] The camera 5a is a front camera, a rear camera, a side camera, an interior camera, or the like mounted on the vehicle V, and captures images of the inside and outside of the vehicle V. The interior camera captures images of the user U, for example.

[0027] The accelerator sensor 5b measures the accelerator opening of the vehicle V. The acceleration sensor 5c measures the acceleration and vehicle speed applied to the vehicle V. The steering angle sensor 5d measures the steering angle of the vehicle V. The various sensors 5 may include sensors other than the sensors 5a to 5d.

[0028] As described above, the seat control device 50 controls the posture of the seat S in response to the control signal input from the in-vehicle device 10. Note that, for example, the seat control device 50 may be installed when the vehicle V is manufactured, or may be retrofitted after the vehicle V is purchased.

[0029] The electronic devices 60 are various electronic devices mounted on the vehicle V, such as an air conditioner, power windows, lighting, audio equipment, and a navigation system. As will be described later, the in-vehicle device 10 can improve the degree of reproduction of the state of the VR space by controlling the electronic devices 60, for example.

[0030] For example, the in-vehicle device 10 may cooperate with a navigation device to provide VR content that is tailored to the destination of the vehicle V and the route to the destination.

[0031] The storage unit 11 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, and in the example of Fig. 3, stores a VR content DB (database) 11a and coefficient information 11b. The VR content DB 11a is a database that stores a group of VR contents to be provided to the HMD 3. The coefficient information 11b is information related to various coefficients used in arithmetic processing for converting ambient information into control signals.

[0032] The control unit 12 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (not shown) stored in the storage unit 11 using RAM as a work area. The control unit 12 can also be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0033] The control unit 12 includes a providing unit 12a, an acquiring unit 12b, a receiving unit 12c, a correcting unit 12d, and an output unit 12e. The providing unit 12a provides the VR content stored in the VR content DB 11a to the HMD 3. The providing unit 12a also reflects changes due to the tilt of the HMD 3 in the VR content. As described above, the providing unit 12a may provide the VR content received from the server device 100 to the HMD 3.

[0034] The acquisition unit 12b constantly acquires sensing data from the various sensors 5. The acquisition unit 12b also passes the acquired sensing data to the correction unit 12d. Note that the sensing data here corresponds to an example of surrounding information and traveling information that indicate the surrounding environment.

[0035] In addition, the acquisition unit 12b may acquire, for example, information regarding the state of sound, air conditioning, and lighting inside the vehicle V, i.e., information regarding the operating status of the electronic device 60, as ambient information, or may acquire information regarding the route on which the vehicle V is traveling as ambient information.

[0036] The receiving unit 12c receives a state signal relating to the state of the user U in the virtual space from the server device 100. The receiving unit 12c also passes the received state signal (posture signal and environmental signal) to the correcting unit 12d.

[0037] For example, the state signal includes a posture signal related to the posture of the user U in the virtual space and an environmental signal related to the environment in the virtual space. Here, the environment in the virtual space includes, for example, lighting, temperature, wind (wind direction, wind volume), etc., and the environmental signal is a signal for reproducing these.

[0038] The correction unit 12d corrects parameters that reduce the reproducibility of the control signal depending on the surrounding environment. For example, the correction unit 12d converts the sensing data received from the acquisition unit 12b into a correction signal using various parameters in the coefficient information 11b. The correction signal is a signal for correcting the attitude signal.

[0039] For example, the corrector 12d corrects the attitude signal by canceling out the attitude signal received by the receiver 12c from the server device 100 with a correction signal based on sensing data.

[0040] An example of the correction process performed by the corrector 12d will now be described with reference to Fig. 4. Fig. 4 is a schematic diagram of the correction process performed by the corrector 12d. As shown in Fig. 4, the corrector 12d corrects the attitude signal by subtracting a movement component V1 indicated by the correction signal from a movement component V2 indicated by the attitude signal.

[0041] In the example of Figure 4, the movement component V1 indicates a leftward direction, and the movement component V2 indicates a rightward direction. Therefore, for example, by subtracting the movement component V1 from the movement component V2 ("V2-V1"), the movement component V2 is corrected to the movement component V3.

[0042] Then, the corrector 12d outputs an attitude signal corresponding to the movement component V3 to the output unit 12e. That is, the corrector 12d offsets the attitude signal and the correction signal, thereby correcting the attitude signal to one that suppresses a decrease in the degree of reproduction in the virtual space that occurs as the vehicle V travels.

[0043] In this case, the corrector 12d may compare each parameter of the status signal with each parameter of the correction signal, and correct only those parameters whose corresponding positive and negative signs are opposite to each other.

[0044] That is, in this case, the movement component V1 and the movement component V2 may be compared, and only the components whose signs are opposite in the X-axis coordinate, the Y-axis coordinate, and the Z-axis coordinate may be corrected. Also, the corrector 12d may multiply the movement component V1 by a predetermined coefficient before correcting it.

[0045] That is, the correction unit 12d may appropriately adjust the offset level of the posture signal (movement component V2) by the correction signal (movement component V1). In this case, for example, from the viewpoint of preventing XR sickness, it is sufficient to align the image displayed on the HMD 3 with the direction in which the seat control device 50 moves, and the correction unit 12d may minimize the correction of the posture signal by the correction signal.

[0046] The offset level may be set automatically in accordance with the VR content, or may be set by selection by the user U.

[0047] Furthermore, the correction unit 12d corrects, for example, an environmental signal from among the status signals received from the server device 100. For example, the correction unit 12d converts various parameters related to the temperature, wind direction, brightness, etc. of the real world into a correction signal related to the environment, and corrects the environmental signal by offsetting the correction signal with the environmental signal received from the server device 100. The correction unit 12d outputs the corrected environmental signal to the output unit 12e.

[0048] Returning to the explanation of Fig. 3, the output unit 12e will be explained. The output unit 12e outputs the control signal corrected by the correction unit 12d to the seat control device 50. That is, the output unit 12e outputs the attitude signal corrected by the correction unit 12d to the seat control device 50. This allows the seat control device 50 to perform seat control that suppresses a decrease in the degree of reproduction in the virtual space that accompanies the state of the vehicle V, for example.

[0049] Furthermore, the correction unit 12d outputs the environmental signal corrected by the correction unit 12d to the electronic device 60. This allows the electronic device 60 to make the environment inside the vehicle V closer to the state in the virtual space, thereby improving reality.

[0050] Furthermore, because the seat control device 50 is designed so that it cannot move beyond its movement limit due to installation space limitations, it is more convenient to return it to its initial position (a neutral position where the amount of movement in any direction is appropriate), so it is preferable to control it so that it returns to its initial position appropriately (or always) at a slow speed that is not noticeable to the user.

[0051] Next, the processing procedure executed by the in-vehicle device 10 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a flowchart executed by the in-vehicle device 10. Fig. 6 is a flowchart showing the processing procedure of step S104 shown in Fig. 5. Note that the in-vehicle device 10 has a navigation function for providing route guidance to a destination, as an example.

[0052] 5, when the vehicle V is powered on, the in-vehicle device 10 sets a destination for the vehicle V in response to an operation by the user U, searches for a route to the destination, and starts guiding the user to the destination (step S101). After that, the in-vehicle device 10 sets VR content to be provided to the user U in the vehicle in response to a selection operation or the like by the user U (step S102). Next, the in-vehicle device 10 sets an offset level in response to, for example, the operation or set VR content by the user U and the route guidance information (step S103).

[0053] In addition, in this example, the setting of the offset level according to the route guidance information is added as a parameter because, for example, if the searched route is a route with poor road surface conditions overall, the user may become accustomed to the impact of the poor road surface on the user, so it may be better not to offset too much. However, this control of setting the offset level according to the route guidance information may be omitted. Next, the in-vehicle device 10 proceeds to VR playback processing (step S104). Then, the processing ends.

[0054] 6, in the VR playback process, the in-vehicle device 10 acquires sensing data (step S111) and receives a status signal from the server device 100 (step S112). Subsequently, the in-vehicle device 10 corrects the status signal based on the sensing data (step S113).

[0055] Then, the in-vehicle device 10 outputs the corrected state signal to the seat control device 50 or the electronic device 60 (step S114), and then ends the process.

[0056] Next, an in-vehicle device 10A according to a second embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram of the in-vehicle device 10A according to the second embodiment. The in-vehicle device 10A has a different configuration from the in-vehicle device 10 in that it includes a generation unit 12f instead of the reception unit 12c and the correction unit 12d.

[0057] The generation unit 12f generates a control signal based on the sensing data acquired by the acquisition unit 12b and the status of the VR content provided by the provision unit 12a. For example, the generation unit 12f inputs each parameter indicated by the sensing data for each parameter related to the state of the user U in the virtual space, and generates a control signal.

[0058] As a result, the generation unit 12f can generate a control signal with reduced deterioration in reproducibility, and the generated control signal can be used as is. That is, in this case, the in-vehicle device 10A can generate a control signal without receiving a status signal from the server device 100.

[0059] That is, in this case, it is possible to reduce the processing load of the server device 100. Note that, for example, the functions of the in-vehicle device 10 and the in-vehicle device 10A may be integrated and switched appropriately depending on the type of AR content or the load of the server device 100, for example.

[0060] Next, a process procedure executed by the in-vehicle device 10A according to the second embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the process procedure executed by the in-vehicle device 10A according to the second embodiment.

[0061] As shown in FIG. 8, upon acquiring sensing data (step S201), the in-vehicle device 10A generates a control signal in accordance with the acquired sensing data and the state of the user U in the virtual space (step S202).

[0062] Next, the in-vehicle device 10A outputs the generated control signal to the seat control device 50 or the electronic device 60 (step S203), and then ends the process.

[0063] As described above, the in-vehicle device 10 (10A) according to the embodiment is a control device that outputs a control signal for reproducing, within the vehicle V (an example of a moving body), the state in a virtual space experienced by a user U aboard the vehicle V, and includes an acquisition unit 12b and an output unit 12e. The acquisition unit 12b acquires ambient information related to the surrounding environment in which the user U actually exists. The output unit 12e outputs a control signal that suppresses a decrease in the degree of reproduction in the virtual space due to the surrounding environment indicated by the ambient information acquired by the acquisition unit 12b. Therefore, the in-vehicle device 10 (10A) according to the embodiment can suppress a decrease in the reality of the XR space within the moving body.

[0064] In the above-described embodiment, a case has been described in which a control signal is output according to the current state of the vehicle V, but the present invention is not limited to this. That is, the in-vehicle device 10 may, for example, predict the future state of the vehicle V.

[0065] For example, the in-vehicle device 10 may detect traffic lights, pedestrians on a crosswalk, deceleration of a preceding vehicle, etc. from the image captured by the camera 5a, and predict the timing of deceleration or stopping of the vehicle V. In this case, the in-vehicle device 10 may output a control signal that is corrected or generated in consideration of the deceleration, etc., in accordance with the timing of deceleration.

[0066] Furthermore, the in-vehicle device 10 may predict the steering angle, etc. of the vehicle V according to the planned travel route of the vehicle V, and output a control signal that is corrected or generated in consideration of the predicted steering angle, etc. For example, if there is a curve in the traveling direction of the vehicle V, the in-vehicle device 10 may predict the lateral acceleration predicted from the vehicle speed and the R value of the curve, and output a control signal that is corrected or generated in accordance with entering the curve.

[0067] In the above-described embodiment, the in-vehicle device 10 is described as a control device, but the control device may be the server device 100. In this case, the server device 100 acquires surrounding information from the in-vehicle device 10, generates a control signal, and then controls the seat control device 50 and the electronic device 60 via the in-vehicle device 10.

[0068] In the above-described embodiment, the moving body is a vehicle V, but the present invention is not limited to this. The moving body may be an airplane, a ship, a train, or the like.

[0069] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0070] 1. Control System 10, 10A In-vehicle device (example of control device) 11 Storage section 11b Coefficient Information 12 Control Unit 12a Providing Department 12b Acquisition part 12c Receiver 12d Correction section 12e Output section 12f generator 50 Seat control device 60 Electronic equipment 100 Server device

Claims

1. A control device that outputs a control signal to control a reproduction device that reproduces, in a moving body, a state in a virtual space experienced by a user riding on the moving body, using a control signal based on content, a controller; The controller acquiring ambient information relating to the user's actual surrounding environment; correcting the control signal in accordance with the surrounding information so as to suppress a decrease in the degree of reproduction in the virtual space due to the surrounding environment; The control signal corrected in accordance with the ambient information is output to the reproduction device. A control device characterized by:

2. A control device that controls a reproduction device that reproduces a state in a virtual space experienced by a user riding on a moving body using a content-based control signal, comprising: a controller; The controller acquiring running state information relating to a running state of the mobile object as surrounding information relating to a surrounding environment in which the user actually exists; correcting the control signal in accordance with the running state information so as to suppress a decrease in the degree of realism in the virtual space due to the running state of the moving object; The control signal corrected in accordance with the driving state information is output to the reproduction device. A control device characterized by:

3. A control device that outputs a control signal to control a reproduction device that reproduces, in a moving body, a state in a virtual space experienced by a user riding on the moving body, using a control signal based on content, a controller; The controller acquiring electronic device operation information relating to an operating state of an electronic device installed within the vehicle as surrounding information relating to the surrounding environment where the user actually exists; correcting the control signal in accordance with the electronic device operation information so as to suppress a decrease in the degree of reproducibility in the virtual space due to an operating state of the electronic device; The control signal corrected in accordance with the electronic device operation information is output to the reproduction device. A control device characterized by:

4. A control method for controlling a reproduction device that reproduces a state in a virtual space experienced by a user riding on a moving body using a content-based control signal, comprising: acquiring ambient information relating to the user's actual surrounding environment; correcting the control signal in accordance with the surrounding information so as to suppress a decrease in the reproducibility of the virtual space due to the surrounding environment; outputting the control signal corrected according to the ambient information to the reproduction device; A control method comprising:

5. A control program for controlling a reproduction device that reproduces, within a moving body, a state in a virtual space experienced by a user riding on the moving body, using a control signal based on content, comprising: acquiring surrounding information relating to the surrounding environment in which the user actually exists; correcting the control signal in accordance with the surrounding information so as to suppress a decrease in the reproducibility of the virtual space due to the surrounding environment; outputting the control signal corrected according to the ambient information to the reproduction device; A control program that causes a computer to execute the above.

6. A control device according to any one of claims 1 to 3; a seat control device that controls the posture of a seat installed in the moving body as the reproduction device; A control system comprising:

7. A control device according to any one of claims 1 to 3; a device that transmits, to the control device, a status signal relating to the status of the user in the virtual space as the surrounding information; A control system comprising:

Images