Virtual Space Image Generation Device and Method

Abstract

To provide a virtual space image generation device and method, which allow for generating a virtual space image that provides visibility closed to how things look in the real space when a viewpoint of a user moves.SOLUTION: A virtual space image generation device 2 generates a virtual space image including a visibility changing region where visibility changes according to movement of a viewpoint of a user U. The virtual space image generation device 2 comprises an image generation unit 14 configured to generate a virtual space image with a visibility changing region where transition completion time required for visibility to transition between a first state and a second state different from the first state changes according to given conditions when the user U moves the viewpoint.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a virtual space image generation apparatus and method.

Background Art

[0002] As a conventional technique for generating a virtual space image and displaying it on a display device, for example, Patent Document 1 discloses a virtual space image providing method for providing a virtual space image that a user views to a head-mounted display (HMD). In this virtual space image providing method, the rotation direction and rotation speed of the HMD are acquired, and blurring processing is performed on both end regions of the virtual space image in the direction on the screen corresponding to the rotation direction within a range and with an intensity corresponding to the rotation speed, thereby reducing VR (Virtual Reality) sickness.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Incidentally, in vehicle development and the like, there are cases where an object to be evaluated is displayed on a virtual space image to evaluate the visibility of the object in the real space. In this case, it is necessary to make the appearance of the object displayed on the virtual space image closer to the appearance of the real space. For example, in the real space, immediately after the user's viewpoint changes or immediately after the arrangement or distance of the objects in the visual field changes, the peripheral area of the viewpoint or the object appears blurred. The blurring of the peripheral area accompanying the movement of the viewpoint and the like changes with the passage of time according to the focus adjustment characteristics of the eyes, the state of the user, the state of the vehicle surroundings, and the like. Therefore, when evaluating visibility by displaying an object on a virtual space image, it is desirable to reproduce the temporal change in the blurring in the real space as described above.

[0005] However, in the conventional technology as described above, only the range and strength of the blurring process applied to the virtual space image are set according to the rotation speed of the HMD, and the temporal change in the state of the blurring process applied to the set range is not considered. Usually, the blurring process of image data is executed at a speed according to the performance of the hardware responsible for image processing, and image data in a blurred state with a desired strength is generated at a speed higher than the focus adjustment speed of the eyes. For this reason, in the conventional technology, when the HMD rotates, that is, when the user's head orientation changes and the viewpoint on the virtual space image moves, the temporal change in the blurring of the virtual space image displayed on the HMD is different from the appearance of the real space, and there is room for improvement in evaluating the visibility of the object on the virtual space image.

[0006] The present invention has been made paying attention to the above points, and an object thereof is to provide a virtual space image generation apparatus and method capable of generating a virtual space image that realizes visibility close to the appearance of the real space when the user's viewpoint moves.

Means for Solving the Problems

[0007] To achieve the above object, one aspect of the present invention provides a virtual space image generation device that generates a virtual space image including a visibility change region whose visibility changes based on the movement of the user's viewpoint. When the user's viewpoint moves, this virtual space image generation device To be displayed for the user after the viewpoint movement includes an image generation unit that generates the virtual space image in which the transition completion time required for the visibility of the visibility change region to transition between a first state and a second state different from the first state changes based on a predetermined condition.

Advantages of the Invention

[0008] According to the virtual space image generation device according to one aspect of the present invention, since the visibility of the visibility change region transitions between the first and second states over a transition completion time that changes based on a predetermined condition, it is possible to generate a virtual space image that realizes visibility close to the way the real space looks when the user's viewpoint moves.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a schematic configuration of a driving simulator system to which a virtual space image generation device according to an embodiment of the present invention is applied. In FIG. 1, the driving simulator system 1 is used, for example, for evaluating the visibility of various objects in vehicle development such as automobiles and for simulating driving experiences. This driving simulator system 1 includes a virtual space image generation device 2, a sensor 3, and an image forming device 4 according to the present embodiment.

[0011] The virtual space image generation device 2 detects the operation of the user U based on the output signal from the sensor 3, and generates a virtual space image including a region where the visibility changes according to the detected operation of the user U. Then, the virtual space image generation device 2 transmits the generated virtual space image to the user U via an image forming device 4 such as a head-mounted display (HMD) worn on the head of the user U. The image forming device 4 of the present embodiment has left and right display units corresponding to the left and right eyes of the user, respectively. The image forming device 4 causes the user to perceive a three-dimensional virtual space by displaying virtual space images with parallax on each of the left and right display units. As a method of displaying virtual space images with parallax on each of the left and right display units, different virtual space images may be displayed on each of the left and right display units, or while outputting a common virtual space image to each of the left and right display units, an optical shutter may be provided on each of the left and right display units to generate a parallax between the virtual space images output from each display unit.

[0012] Note that the image forming device 4 is not limited to a configuration in which a display device for displaying a virtual space image, such as an HMD, is worn on the head of the user U, and may be an image display device such as a liquid crystal display arranged in front of the user U. Further, the image forming device 4 may be an image projection device that projects a virtual space image onto a predetermined projection surface (screen, glass, wall surface), such as a projector or a head-up display. In this case, in order to perceive the virtual space from the projected virtual space image, it is preferable to separately wear an optical shutter device on the left and right eyes of the user.

[0013] In such a driving simulator system 1, by inputting virtual space images with different parallaxes to each of the left and right eyes of the user U via the image forming device 4 (or the optical shutter device), the user U can visually recognize (perceive) the virtual space. Thereby, various objects in the real space can be reproduced in the virtual space, and the visibility of various objects can be evaluated on the virtual space image, or the driving of a vehicle can be simulated and experienced on the virtual space image.

[0014] Specifically, the virtual space image generation device 2 includes, as its functional blocks, for example, a viewpoint detection unit 11, an input unit 12, a storage unit 13, an image generation unit 14, and a display control unit 15. Although the hardware configuration of the virtual space image generation device 2 is not shown here, for example, it includes a computer system including a processor, a memory, a user input interface, and a communication interface. That is, in the virtual space image generation device 2, each function of the viewpoint detection unit 11, the image generation unit 14, and the display control unit 15 is realized by the processor of the computer system reading and executing the program stored in the memory.

[0015] The viewpoint detection unit 11 detects the viewpoint of the user U using the output signal of the sensor 3. The viewpoint of the user U is the point on the image forming apparatus 4 where the line of sight of the user U is directed. As the sensor 3, for example, an eye line sensor built in an HMD worn by the user U is used. The sensor 3 detects the movement of the eyes of the user U, measures the direction of the line of sight, and outputs a signal indicating the direction of the line of sight to the viewpoint detection unit 11 via the communication interface of the computer system. The viewpoint detection unit 11 uses the line of sight direction of the user U measured by the sensor 3, the positional relationship between the eyes of the user U and the image forming apparatus 4, and the position information on the virtual space provided based on the image data of the virtual space stored in the storage unit 13 to detect the position (coordinates on a two-dimensional plane) of the viewpoint of the user U on the image forming apparatus 4. Such a function of the viewpoint detection unit 11 for detecting the viewpoint of the user U is sometimes called eye tracking. The viewpoint detection unit 11 transmits the detected position information of the viewpoint to the image generation unit 14.

[0016] In addition to the eye-tracking function of the viewpoint detection unit 11, the virtual space image generation device 2 may also be provided with a head-tracking function for detecting the movement of the head of the user U and a position-tracking function for detecting the movement of the body of the user U. The detection results by these head-tracking and position-tracking are also transmitted to the image generation unit 14 together with the detection result of the viewpoint detection unit 11. The detection results include, for example, information related to the orientation of the head of the user U, and based on this information, the orientation of the line of sight of the user U may be estimated.

[0017] The input unit 12 is realized by a user input interface of a computer system, and has, for example, a keyboard, a mouse, an operation controller, etc. Further, the input unit 12 has a receiving unit for receiving information wired or wirelessly from the outside, and also functions as an external information input interface for receiving information from an external computer. In the input unit 12, predetermined conditions such as conditions related to the environment of the virtual space (hereinafter referred to as "environment conditions"), conditions related to the user U (hereinafter referred to as "user conditions"), and driving conditions (travel route, speed) of the vehicle in the virtual space are input. The environment conditions include the weather (sunny, cloudy, rainy, foggy, etc.), humidity, driving environment (outdoor, indoor, tunnel, etc.), windshield conditions, or a combination thereof in the virtual space. Also, the user conditions include the age, gender, eyesight, eye health, eye opening degree, dominant eye of the user U, or a combination thereof. Information related to the predetermined conditions input by the input unit 12 is transmitted to the image generation unit 14 and stored in the storage unit 13.

[0018] Note that the above user conditions may be input to the virtual space image generation device 2 using the input unit 12 with the values obtained by previously conducting experiments on the subjects assumed to be the user U of the driving simulator system 1, or the user U may be photographed with a camera or the like separately provided in the virtual space image generation device 2, and the user conditions may be determined or detected based on the photographed user image.

[0019] The storage unit 13 is realized by a storage device (e.g., a magnetic disk, an optical disk, a flash memory, etc.) connected to the computer system, and stores various setting information such as the positional relationship between the eyes of the aforementioned user U and the image forming apparatus 4, environmental conditions, user conditions, and driving conditions of the vehicle. Further, the storage unit 13 stores image data of a virtual space including various objects. The various objects are various objects included in the scene that the user U can see from the driver's seat of the vehicle in the virtual space.

[0020] The image generation unit 14 generates a virtual space image to be displayed on the image forming apparatus 4 by using the image data stored in the storage unit 13, the image data received from the input unit 12, and various setting information. At this time, based on the viewpoint detected by the viewpoint detection unit 11, the image generation unit 14 generates a virtual space image in which the visibility within a predetermined visibility change region is in a first state and the visibility outside the visibility change region is in a second state different from the first state. That is, the virtual space image generated by the image generation unit 14 includes an image portion having a relatively high visibility and an image portion having a relatively low visibility. Then, the image generation unit 14 updates the virtual space image so as to transition between a first state in which the image portion located within the visibility change region has a relatively higher visibility than the image portion located outside the visibility change region and a second state in which the image portion located within the visibility change region has a relatively lower visibility than the image portion located outside the peripheral region of the viewpoint. Note that the visibility change region includes not only the peripheral region of the source viewpoint and the peripheral region of the destination viewpoint, which will be described later, but also regions other than the peripheral region of the viewpoint.

[0021] The visibility state is controlled, for example, by applying a blurring process to an image displayed in a target area. The blurring process is a process that changes the amount of information in the image data to make the image appear blurred. In other words, the blurring process is an image process that reduces the amount of information that can be confirmed by the user U through vision. Specific examples of the blurring process include a process of reducing the amount of information, a process of lowering the resolution, or a process of gradually decreasing the display area, a process of gradually increasing the display area, or a combination of these processes, for an image (object) displayed in the area to be processed. As an example of a combination of processes, performing the process of gradually increasing the display area and the process of gradually decreasing the display area in order or alternately makes it easier to reproduce a defocused state. Therefore, the first state with relatively high visibility is, for example, a focused state before the blurring process is applied, representing a state where the amount of information that the user U can confirm through vision for the image is large. Also, the second state with relatively low visibility is, for example, a blurred and defocused state after the blurring process is applied, representing a state where the amount of information that the user U can confirm through vision for the image is small.

[0022] Figure 2 shows an example of a virtual space image before the blurring process is applied. This virtual space image is a virtual space image input to one of the left and right eyes. Another virtual space image (not shown) with a different parallax from the virtual space image in Figure 2 is input to the other of the left and right eyes. The user U can perceive the virtual space by means of virtual space images with different parallaxes input to the left and right eyes respectively. Also, Figure 3 shows an example of a virtual space image after the blurring process is applied outside the area around the viewpoint.

[0023] As shown in FIGS. 2 and 3, the virtual space image generated by the image generation unit 14 represents the scenery in the virtual space that the user U can see from the driver's seat of the vehicle. In the illustrated example, the virtual space image includes, as objects representing the vehicle, the upper part of the steering wheel, the upper part of the dashboard, the right front pillar, the front end of the roof, the rearview mirror, and the right side mirror, etc. The number "8" displayed at the lower center of the virtual space image is an object for evaluating the visibility near the upper end of the steering wheel. Also, the virtual space image includes, as objects representing stationary objects outside the vehicle, roads, sidewalks, buildings, and road signs (stop sign), etc.

[0024] Before the blurring process is applied, the virtual space image (FIG. 2) is in focus for all objects, and the visibility of the entire area of the virtual space image is high. On the other hand, after the blurring process is applied, in the virtual space image (FIG. 3), the viewpoint (□ mark) of the user U is at the position P on the object of the number "8" displayed near the upper end of the steering wheel, and the objects located inside the viewpoint peripheral area A surrounded by the dashed line in the figure are in focus, while the objects located outside the viewpoint peripheral area A are in a blurred state out of focus. That is, the virtual space image generated by the image generation unit 14 and subjected to the blurring process according to the position P of the viewpoint of the user U is in a first state where the visibility inside the viewpoint peripheral area A is relatively high and in a second state where the visibility outside the viewpoint peripheral area A is relatively low. Note that the □ mark indicating the viewpoint of the user U is not displayed in the actual virtual space image.

[0025] In addition, when the viewpoint of the user U moves, the image generation unit 14 updates the virtual space image subjected to the blurring process. This update process of the virtual space image can be executed for any movement of the viewpoint of the user U. For example, in FIG. 3, when the position P of the viewpoint of the user U moves to different positions within the viewpoint surrounding area A, the image of the portion where a shift has occurred in the viewpoint surrounding area before and after the movement is updated. Also, for example, when the position P of the viewpoint of the user U moves to a distant position outside the viewpoint surrounding area A, and the viewpoint surrounding area of the movement destination is located outside the viewpoint surrounding area of the movement origin, the image of the entire viewpoint surrounding area of the movement destination and the image of the entire viewpoint surrounding area of the movement origin are updated. When the amount of movement of the viewpoint is small as in the former case, the data amount of the image to be updated is small, so the load of image processing in the image generation unit 14 is reduced. Hereinafter, a specific example will be given and described in detail for image processing when the amount of movement of the viewpoint is large as in the latter case.

[0026] FIG. 4 shows an example of the viewpoint movement of the user U. In the example of FIG. 4, the viewpoint (□ mark) of the user U moves from the position Pn on the object (first object) of the number "8" displayed near the upper end of the steering wheel to the position Pf on the object (second object) of the road sign installed on the left sidewalk in front of the vehicle. The object of the road sign is located farther in the depth direction within the virtual space than the object near the upper end of the steering wheel. Therefore, the viewpoint of the user U moves in the upper left direction on a two-dimensional plane (on the virtual space image) that extends in the left-right direction and the up-down direction within the virtual space, and also moves farther in the depth direction.

[0027] Note that the depth direction in the virtual space varies depending on the form of the image forming apparatus 4. Specifically, in the case of the image forming apparatus 4 where the head position of the user U does not change, the depth direction is a specific direction (for example, the front-rear direction) preset in the virtual space. On the other hand, in the case of the image forming apparatus 4 where the head position of the user U changes due to head tracking or the like, the depth direction is a predetermined direction that relatively changes according to the position of the head. For example, it may be the direction in which the head faces, or the line-of-sight direction of the user U with respect to the viewpoint before the user U moves, that is, the direction connecting the eye E of the user U and the viewpoint before the actual movement or the viewpoint before the movement in the virtual space.

[0028] FIG. 5 is a conceptual diagram showing the viewpoint movement of the user U in the virtual space as described above, viewed from above. In FIG. 5, the arrow Z direction indicates the depth direction (vehicle front-rear direction) of the virtual space, and the arrow X direction indicates the horizontal direction (vehicle width direction) of the virtual space. As shown in FIG. 5, the viewpoint of the user U moves from the position Pn to the position Pf on the image forming apparatus 4 (on the virtual space image). The first object (the number "8") displayed at the position Pn of the viewpoint of the moving source is at the position Pn' in the virtual space, which is separated from the image forming apparatus 4 by a distance Zn in the depth direction. Also, the second object (road sign) displayed at the position Pf of the viewpoint of the moving destination is at the position Pf' in the virtual space, which is separated from the image forming apparatus 4 by a distance Zf in the depth direction. The distance Zf is longer than the distance Zn by a distance ΔZ.

[0029] The focus (sharpness) of the eye E of the user U in the real space matches the position Pn on the image forming apparatus 4 at the moving source of the viewpoint, and matches the position Pf on the image forming apparatus 4 at the moving destination of the viewpoint. That is, the actual focal length of the eye E of the user U is, as shown by the solid line arrow in FIG. 5, the distance Fn from the eye E to the position Pn at the moving source of the viewpoint, and the distance Ff from the eye E to the position Pf at the moving destination of the viewpoint. d represents the distance between the eye E of the user U and the image forming apparatus 4 in the depth direction. In FIG. 5, since the user U changes the line of sight by moving the line of sight, the distance d between the position Pn and the position Fn is different, but the change amount Δd (not shown) of the distance d before and after the movement is small.

[0030] On the other hand, as indicated by the dotted arrow in FIG. 5, the virtual focal length of the eye E of the user U is the distance Fn' from the eye E to the position Pn' at the origin of the viewpoint movement, and the distance Ff' from the eye E to the position Pf' at the destination of the viewpoint movement. That is, in the virtual space, the position Pn' is arranged behind the position Pn by a distance Zn in the depth direction, and the position Pf' is arranged behind the position Pf by a distance Zf in the depth direction. Also, the position Pf' is located behind the position Pn' by a distance ΔZ in the depth direction. Further, in FIG. 5, the distance ΔZ is much larger than the distance Δd. In the present embodiment, the generation (update) process of the virtual space image to be displayed on the image forming apparatus 4 is performed so as to suppress the difference in the appearance due to the difference between the change in the actual focal lengths Fn and Ff accompanying the viewpoint movement of the user U and the change in the virtual focal lengths Fn' and Ff'.

[0031] When the viewpoint movement of the user U as described above occurs, the image generation unit 14 determines the movement direction and movement amount of the viewpoint on the image forming apparatus 4 (on the virtual space image) from the change in the positions Pn and Pf (coordinates on the two-dimensional plane) of the viewpoint detected by the viewpoint detection unit 11. Also, the image generation unit 14 determines the positions Pn' and Pf' of the viewpoint in the virtual space from the depth information corresponding to the positions Pn and Pf of the viewpoint detected by the viewpoint detection unit 11 among the depth information defined for each pixel (or for each object) on the virtual space image, and determines whether the viewpoint is moving at least in the depth direction in the virtual space, that is, whether the viewpoint is moving in the direction away from or approaching in the virtual space. Then, when the viewpoint moves in the depth direction in the virtual space, the image generation unit 14 executes the update process of the virtual space image.

[0032] In this update process, the image generation unit 14 performs a first process of updating the virtual space image such that the visibility within the surrounding area Af (visibility change area) of the destination viewpoint increases from the second state to the first state, and the transition completion time required for the visibility to transition between the first state and the second state changes based on a predetermined condition, and a second process of updating the virtual space image such that the visibility within the surrounding area An of the source viewpoint decreases from the first state to the second state. The transition completion time is the time from when the viewpoint movement of the user U is completed until the visibility transition is completed. That is, in the update process of the virtual space image by the image generation unit 14, the visibility within the surrounding area Af of the destination viewpoint transitions from the second state to the first state over a transition completion time that changes based on a predetermined condition, and a virtual space image is generated in which the visibility within the surrounding area An of the source viewpoint transitions from the first state to the second state.

[0033] As shown in FIG. 4, in the virtual space image immediately after the viewpoint movement of the user U, the visibility within the surrounding area An of the source viewpoint is in the first state (in-focus state), and the visibility within the surrounding area Af of the destination viewpoint is in the second state (out-of-focus and blurred state). The state of visibility (state of blur processing) in the virtual space image immediately after such a viewpoint movement is the same as the state of visibility in the virtual space image (before the viewpoint movement of the user U) shown in FIG. 3 described above.

[0034] FIG. 6 shows an example of the virtual space image after the transition completion time has elapsed since the viewpoint movement of the user U. As shown in FIG. 6, in the virtual space image in which the update process (first process and second process) is completed after the transition completion time has elapsed, the visibility within the surrounding area An of the source viewpoint has transitioned to the second state (out-of-focus and blurred state), and the visibility within the surrounding area Af of the destination viewpoint has transitioned to the first state (in-focus state).

[0035] Note that when the detection results obtained by the above-described head tracking or position tracking are transmitted, the image generation unit 14 may change the virtual space image following the movement of the head or body of the user U. For example, when the user U turns the head to the left, the image generation unit 14 changes the virtual space image so that a scene on the left side of the user U in the virtual space is displayed in accordance with the movement of the head of the user U detected by head tracking. Also, for example, when the user U moves and the position of the body changes, the image generation unit 14 changes the virtual space image so that the field of view of the user U changes in accordance with the current position of the user U detected by position tracking.

[0036] The display control unit 15 (FIG. 1) generates a control signal for causing the image forming apparatus 4 to display the virtual space image generated by the image generation unit 14, and outputs the control signal to the image forming apparatus 4. The image forming apparatus 4 that has received the control signal from the display control unit 15 displays the virtual space image according to the control signal.

[0037] Next, the operation of the virtual space image generation apparatus 2 according to the present embodiment will be described. FIG. 7 is a flowchart showing an example of a method for generating a virtual space image by the virtual space image generation apparatus 2. In the generation apparatus 2 of the present embodiment, first, in step S10 of FIG. 7, the viewpoint detection unit 11 detects the position (coordinates on a two-dimensional plane) of the viewpoint of the user U on the image forming apparatus 4 using the output signal of the sensor 3. The detection process of the viewpoint position by the viewpoint detection unit 11 is repeatedly executed at a predetermined period. The position information of the viewpoint of the user U detected by the viewpoint detection unit 11 is transmitted to the image generation unit 14.

[0038] In the subsequent step S20, the image generation unit 14 that has received the viewpoint position information from the viewpoint detection unit 11 generates a virtual space image to be displayed on the image forming apparatus 4 by using the image data stored in the storage unit 13 (or the image data received from the input unit 12) and various setting information. At this time, as in the example shown in FIG. 3 described above, the image generation unit 14 performs a blurring process on the image portion located outside the peripheral region A of the viewpoint based on the position P of the viewpoint of the user U. As a result, a virtual space image is generated in which the visibility within the peripheral region A of the viewpoint is in a relatively high first state and the visibility outside the peripheral region A of the viewpoint is in a relatively low second state.

[0039] In the next step S30, the image generation unit 14 executes a process of determining the movement of the viewpoint of the user U based on the viewpoint position information transmitted from the viewpoint detection unit 11 at a predetermined period. In this determination process, it is determined whether the viewpoint of the destination has moved at least in the depth direction with respect to the viewpoint before the movement. If there is a viewpoint movement (YES), the process proceeds to the next step S40. If there is no viewpoint movement (NO), the process moves to step S50.

[0040] In step S40, the image generation unit 14 performs an update process of the virtual space image according to the movement of the user's viewpoint. When the update process is completed, the process proceeds to the next step S50, and the display control unit 15 performs control to display the virtual space image generated (or updated) by the image generation unit 14 on the image forming apparatus 4. When the virtual space image is displayed on the image forming apparatus 4, the process returns to step S30 and the same process is repeatedly executed.

[0041] Here, the update process of the virtual space image in step S40 will be specifically described. As described above, according to the image generation unit 14 of the present embodiment, when the viewpoint of the user U moves, the visibility within the peripheral region Af of the destination viewpoint rises from the second state to the first state, and the transition completion time required for the visibility to transition between the first state and the second state changes based on a predetermined condition. The first process of updating the virtual space image and the second process of updating the virtual space image so that the visibility within the peripheral region An of the source viewpoint decreases from the first state to the second state are performed.

[0042] In such update processing of the virtual space image, the transition from the second state to the first state of visibility within the area around the destination viewpoint is realized by reducing the blurring process applied to the image within the area around the destination viewpoint. That is, by reducing the amount of blurring (blurring ratio) of the blurred image to make it closer to a focused state, the visibility within the area is increased and the transition from the second state to the first state is made. The process of reducing the amount of blurring for the transition of visibility from the second state to the first state is performed over the transition completion time. This transition completion time includes the delay time from when the viewpoint movement of user U is completed until the start of the process of reducing the amount of blurring, and the transition time from the start of the process of reducing the amount of blurring until its completion.

[0043] Also, the transition from the first state to the second state of visibility within the area around the source viewpoint is realized by applying a blurring process to the image within the area around the source viewpoint. That is, by increasing the amount of blurring (blurring ratio) of the image in the blurring process, the visibility within the area is decreased and the transition from the first state to the second state is made.

[0044] In the first process of the virtual space image by the image generation unit 14, the delay time and the transition time included in the transition completion time change according to the moving direction of the viewpoint of user U, which is one of the predetermined conditions. The moving direction of the viewpoint of user U includes the up, down, left, and right directions on the image forming apparatus 4 (on the virtual space image), the depth direction within the virtual space, and combinations of these directions. In the present embodiment, for example, regarding the depth direction within the virtual space, the delay time and the transition time change depending on whether the viewpoint of user U is moving in a direction away from or approaching. The movement of the viewpoint of user U from the position Pn to the position Pf illustrated in FIGS. 4 to 6 corresponds to a viewpoint movement in a direction away, and the reverse movement (from the position Pf to the position Pn) corresponds to a viewpoint movement in a direction approaching. Hereinafter, specific examples will be given for each direction of viewpoint movement to explain in detail how the delay time and the transition time included in the transition completion time change.

[0045] FIG. 8 is a graph showing an example of the temporal change in visibility when the viewpoint moves in the direction away from the viewer. The upper graph in FIG. 8 corresponds to the area Af around the destination viewpoint, and the lower graph in FIG. 8 corresponds to the area An around the source viewpoint. The vertical axis of each graph represents the state of visibility V, and the horizontal axis represents time t. The state of visibility V on the vertical axis increases as it moves away from the intersection point (origin) with the horizontal axis. Note that the state of visibility V corresponds to the amount of blurring (blurring ratio) of the image in the blurring process as described above. Therefore, the vertical axis of each graph in FIG. 8 also represents the amount of blurring of the image, and the amount of blurring decreases as it moves away from the origin.

[0046] In FIG. 8, the viewpoint of the user U is at position Pn (on the first object of the number "8") at time t1 and moves to position Pf (on the second object of the road sign) at time t2. At time t1, the visibility within the area Af around the destination viewpoint is in the relatively low second state V2, and the visibility within the area An around the source viewpoint is in the relatively high first state V1.

[0047] The dashed lines in each graph of FIG. 8 represent the temporal change in visibility corresponding to the blurring process applied to the virtual space image in the prior art as described above. In the prior art, the blurring process applied to the virtual space image is executed at a speed corresponding to the performance of the hardware responsible for image processing. Therefore, the transition between the first state V1 and the second state V2 of visibility (amount of blurring) is completed in a short period (time t1 to t2) almost simultaneously with the movement of the viewpoint of the user U.

[0048] On the other hand, in the update process of the virtual space image by the image generation unit 14 in the present embodiment, as shown by the solid lines in each graph, the visibility within the area Af around the destination viewpoint gradually increases from the second state V2 and transitions to the first state V1, and at the same time, the visibility within the area An around the source viewpoint immediately decreases from the first state V1 and transitions to the second state V2.

[0049] Specifically, as shown in the upper graph of FIG. 8, the temporal change in visibility within the area Af around the destination viewpoint is such that the visibility is maintained in the second state V2 during the period (time t2 to t3) from when the viewpoint movement of the user U is completed until a predetermined delay time L1 elapses, and the visibility starts to increase at time t3. After time t3, the visibility within the area Af around the destination viewpoint gradually changes over time and rises to the first state V1 at time t4.

[0050] That is, after the completion of the viewpoint movement of the user U, a process for reducing the blurring process applied to the image within the area Af around the destination viewpoint is started after the elapse of the delay time L1. Then, as the amount of blurring (blurring ratio) of the image gradually decreases over time, the visibility within the area Af around the destination viewpoint transitions from the second state V2 (a blurred state out of focus) to the first state V1 (a focused state). The transition time α1 from the start to the completion of the transition is from time t3 to t4. The transition completion time T1 (time t2 to t4) required from when the viewpoint movement of the user U is completed until the visibility transition is completed is the sum of the delay time L1 and the transition time α1.

[0051] Here, the state where the image within the area Af around the destination viewpoint is in focus corresponds to the state where the focus of the eye E of the user U is adjusted to the virtual position Pf' of the destination viewpoint, as described with reference to FIG. 5 above. That is, the image processing for increasing the visibility within the area Af around the destination viewpoint to the first state on the virtual space image corresponds to the operation of the user U adjusting the focal length to Ff' by activating the focus adjustment function of the eye E in the real space. Therefore, if the temporal change in the visibility when increasing the visibility to the first state V1 approaches the temporal change in the focal length when setting the focal length to Ff' by the focus adjustment function of the eye E, it becomes possible to generate a virtual space image that realizes visibility similar to the appearance in the real space.

[0052] In the upper graph of FIG. 8, the curve C1 over the transition time α1 represents the temporal change in visibility when increasing the visibility in the area Af around the destination viewpoint to the first state V1. By making the shape of this curve C1 follow, for example, the function shown in the following equation (1), it is possible to approximate the temporal change in the focal length by the focusing function of the eye E. [Number] In the above equation (1), t represents the time [s] since the start of the process of reducing the blurring process. F represents the focal length [m] at time t. Do represents the diopter which is the reciprocal of the focal length at the start of the viewpoint movement. Dt represents the diopter which is the reciprocal of the focal length at the end of the viewpoint movement. e represents the Napier's number (the base of the natural logarithm). τ represents the time constant. When the viewpoint moves in the direction away as shown in FIG. 8, Do means 1 / Fn’ and Dt means 1 / Ff’.

[0053] The focal length F in the above equation (1) corresponds to the state of the visibility V at time t. Specifically, the first state V1 of the visibility corresponds to the state where the focal length F of the equation (1) becomes the virtual focal length Ff’ shown in FIG. 5 above (F = Ff’). Also, the second state V2 of the visibility corresponds to the state where the focal length F of the equation (1) becomes the virtual focal length Fn’ of FIG. 5 (F = Fn’).

[0054] Also, the time constant τ in the above equation (1) is set according to the environmental conditions and user conditions described above, and the length of the transition time α1 (the shape of the curve C1) changes according to the time constant τ. FIG. 9 illustrates the change in the transition time α1 when the value of the time constant τ is changed. In the example of FIG. 9, the transition time α1’ when τ = 0.2 is longer than the transition time α1 when τ = 0.1, and the transition time α1’’ when τ = 0.3 is even longer than the transition time α1’.

[0055] Regarding the temporal change in visibility within the peripheral area Af around the destination viewpoint as described above, the temporal change in visibility within the peripheral area An around the source viewpoint, as shown in the lower graph of Fig. 8, starts to decrease from the first state V1 immediately after the user U's viewpoint movement is completed, and decreases to the second state V2 at time t3'. That is, immediately after the completion of the user U's viewpoint movement, the blurring process of the image within the peripheral area An around the source viewpoint is started. Then, as the amount of image blurring (blurring ratio) increases with the passage of time, the visibility within the peripheral area An around the source viewpoint transitions from the first state V1 (in-focus state) to the second state V2 (out-of-focus and blurred state). By changing the visibility within the peripheral area An around the source viewpoint in this way, it is possible to prevent the user U from feeling uncomfortable with the appearance of the peripheral area An around the source viewpoint.

[0056] Fig. 10 is a graph showing an example of the temporal change in visibility when the viewpoint moves in the approaching direction. The upper graph of Fig. 10 corresponds to the peripheral area An around the destination viewpoint, and the lower graph of Fig. 10 corresponds to the peripheral area Af around the source viewpoint. Similar to Fig. 8 described above, the vertical axis of each graph represents the state (amount of blurring) of visibility V, and the horizontal axis represents time t. The state of visibility V on the vertical axis becomes higher as it moves away from the intersection point (origin) with the horizontal axis (the amount of blurring decreases as it moves away from the origin).

[0057] In Fig. 10, the user U's viewpoint is at position Pf (on the second object of the road sign) at time t1 and moves to position Pn (on the first object of the number "8") at time t2. At time t1, the visibility within the peripheral area An around the destination viewpoint is in the relatively low second state V2, and the visibility within the peripheral area Af around the source viewpoint is in the relatively high first state V1.

[0058] The dashed lines in each graph of Fig. 10 represent the temporal change in visibility corresponding to the blurring process applied to the virtual space image in the prior art, similar to the case of Fig. 8 described above, and the transition between the first state V1 and the second state V2 of visibility is completed in a short period (time t1 to t2).

[0059] On the other hand, in the update process of the virtual space image by the image generation unit 14 in the present embodiment, as shown by the solid lines in each graph, the visibility in the surrounding area An of the destination viewpoint gradually increases from the second state V2 and transitions to the first state V1, while the visibility in the surrounding area Af of the source viewpoint immediately decreases from the first state V1 and transitions to the second state V2.

[0060] Specifically, the temporal change in the visibility in the surrounding area An of the destination viewpoint is, as shown in the upper graph of FIG. 10, during the period from when the viewpoint movement of the user U is completed until a predetermined delay time L2 elapses (time t2 to t3), the visibility is maintained in the second state V2, and at time t3, the visibility starts to increase. The delay time L1 in the viewpoint movement in the receding direction described above is set to be longer than the delay time L2 in the viewpoint movement in the approaching direction (L1 > L2). Preferably, the delay time L1 in the viewpoint movement in the receding direction is set to be 2 times or more the delay time L2 in the viewpoint movement in the approaching direction. Specifically, it is experimentally confirmed that it is preferable to set the delay time L1 in the viewpoint movement in the receding direction in the range from 0.05 seconds to 0.2 seconds (0.05 < L1 < 0.2), and set the delay time L2 in the viewpoint movement in the approaching direction in the range from 0 seconds to 0.05 seconds (0 < L2 < 0.05). After time t3, the visibility in the surrounding area Af of the destination viewpoint gradually changes over time and rises to the first state V1 at time t5.

[0061] That is, after the completion of the viewpoint movement of user U, after the elapse of the delay time L2, a process for reducing the blurring process applied to the image within the peripheral area An of the destination viewpoint is started. Then, as the blurring amount (blurring ratio) of the image gradually decreases over time, the visibility within the peripheral area An of the destination viewpoint transitions from the second state V2 (a blurred state out of focus) to the first state V1 (a focused state). The transition time α2 from the start to the completion of the transition is from time t3 to t5. The transition time α1 in the viewpoint movement in the receding direction described above is set to be longer than the transition time α2 in the viewpoint movement in the approaching direction (α1 > α2). The transition completion time T2 (from time t2 to t5) required from the completion of the viewpoint movement of user U to the completion of the visibility transition is the sum of the delay time L2 and the transition time α2.

[0062] The shape of the curve C2 over the transition time α2 can be made to approximate the temporal change in the focal length by the focus adjustment function of the eye E, for example, by following the function shown in the above-mentioned equation (1). When the viewpoint moves in the approaching direction as shown in FIG. 10, in the above-mentioned equation (1), Do means 1 / Ff’ and Dt means 1 / Fn’. Also, the time constant τ is set according to the above-mentioned environmental conditions and user conditions, and the shape of the curve C2 (the length of the transition time α2) changes according to the time constant τ. FIG. 11 illustrates the change in the transition time α2 when the value of the time constant τ is changed. In the example of FIG. 11, the transition time α2’ when τ = 0.2 is longer than the transition time α2 when τ = 0.1, and the transition time α2’’ when τ = 0.3 is even longer than the transition time α2’. It is preferable to set the time constant τ in the viewpoint movement in the receding direction described above to be at least twice the time constant τ in the viewpoint movement in the approaching direction. Specifically, it has been experimentally confirmed that it is preferable to set the time constant τ in the viewpoint movement in the receding direction in the range from 0.05 to 0.2 (0.05 < τ < 0.2) and the time constant τ in the viewpoint movement in the approaching direction in the range from 0 to 0.1 (0 < τ < 0.1).

[0063] Regarding the temporal change in visibility within the peripheral region An of the destination viewpoint as described above, the temporal change in visibility within the peripheral region Af of the source viewpoint, as shown in the lower graph of FIG. 10, starts to decrease from the first state V1 immediately after the viewpoint movement of the user U is completed, and decreases to the second state V2 at time t3'. That is, immediately after the completion of the viewpoint movement of the user U, the blurring process of the image within the peripheral region Af of the source viewpoint is started. Then, as the amount of image blurring (blurring ratio) increases with the passage of time, the visibility within the peripheral region Af of the source viewpoint transitions from the first state V1 (in-focus state) to the second state V2 (out-of-focus and blurred state) in a short time. By changing the visibility within the peripheral region Af of the source viewpoint in this way, it is possible to prevent the user U from feeling discomfort with the appearance of the peripheral region Af of the source viewpoint.

[0064] As described above, in the virtual space image generation device 2 according to the present embodiment, when the viewpoint of the user U moves, the image generation unit 14 generates a virtual space image in which the transition completion times T1 and T2 required for the visibility of the visibility change region located around the source viewpoint to transition between the first state V1 and the second state V2 change based on a predetermined condition. As a result, since the visibility of the visibility change region transitions between the first and second states V1 and V2 over the transition completion times T1 and T2 that change based on a predetermined condition, it is possible to generate a virtual space image that realizes a visibility close to that of the real space even when the viewpoint of the user U moves. If such a virtual space image generation device 2 is applied to construct the driving simulator system 1, in vehicle development and the like, it becomes possible to reproduce various objects in the real space on the virtual space and accurately evaluate the visibility of various objects on the virtual space image. Further, if a simulated driving experience is performed using the driving simulator system 1, it becomes possible to provide a more realistic driving experience to the user U.

[0065] Also, the transition completion times T1 and T2 in the virtual space image generation device 2 of this embodiment include the delay times L1 and L2 and the transition times α1 and α2. In the eye's focusing function, a delay (time lag) occurs between when the viewpoint is moved and when the focal length changes. Considering such properties of the eye, by appropriately setting the delay times L1 and L2 included in the transition completion times T1 and T2 and starting the transition times α1 and α2 after the elapse of the delay times L1 and L2, the visibility of the generated virtual space image can be made closer to the way things look in the real space.

[0066] Also, in the virtual space image generation device 2 of this embodiment, a virtual space image is generated in which the transition times α1 and α2 change according to the moving direction of the viewpoint of the user U. In the eye's focusing function, the adjustment time of the focal length (the time required for the eye to focus) differs depending on the moving direction of the viewpoint, so the blurring method of the area around the viewpoint changes depending on the moving direction of the viewpoint. To match such properties of the eye, by changing the transition times α1 and α2 according to the moving direction of the viewpoint, the visibility of the generated virtual space image can be made even closer to the way things look in the real space.

[0067] Also, in the virtual space image generation device 2 of this embodiment, a virtual space image is generated in which the transition time α1 in the viewpoint movement in the receding direction is longer than the transition time α2 in the viewpoint movement in the approaching direction. In the eye's focusing function, the adjustment time of the focal length is longer when the viewpoint moves in the receding direction than when the viewpoint moves in the approaching direction. To match such properties of the eye, by adjusting the transition times α1 and α2 in the processing of the virtual space image, the visibility of the generated virtual space image can be made even closer to the way things look in the real space.

[0068] Furthermore, in the virtual space image generation device 2 of the present embodiment, a virtual space image is generated in which the transition times α1 and α2 change according to changes in the focal lengths (Fn’, Ff’) of the user's eyes corresponding to the virtual positions (Pn’, Pf’) of the user's viewpoints in the virtual space. In the eye focus adjustment function, the blurring method of the area around the viewpoint when the viewpoint is moved in the real space changes according to the amount of change in the focal length. In order to match such properties of the eyes, by changing the transition times α1 and α2 in the processing of the virtual space image according to the change in the focal length of the eyes, the visibility of the generated virtual space image can be made closer to the way things look in the real space.

[0069] In addition, in the virtual space image generation device 2 of the present embodiment, the delay time L1 in the viewpoint movement in the receding direction is more than twice as long as the delay time L2 in the viewpoint movement in the approaching direction, and the time constant τ of the function that determines the transition time α1 in the viewpoint movement in the receding direction is more than twice the time constant τ of the function that determines the transition time α2 in the viewpoint movement in the approaching direction. A virtual space image is generated. By making such a setting, the visibility of the generated virtual space image can be made even closer to the way things look in the real space. In particular, in the viewpoint movement in the receding direction, the delay time L1 is set in the range from 0.05 seconds to 0.2 seconds, and the time constant τ is set in the range from 0.05 to 0.2. In the viewpoint movement in the approaching direction, if the delay time L2 is set in the range from 0 seconds to 0.05 seconds and the time constant τ is set in the range from 0 to 0.1, the visibility of the virtual space image can be surely made closer to the way things look in the real space.

[0070] In addition, in the virtual space image generation device 2 of the present embodiment, a virtual space image is generated in which the transition completion time changes based on conditions related to the environment of the virtual space (environmental conditions) and conditions related to the user U (user conditions). In the eye focus adjustment function, the blurring method of the peripheral area of the viewpoint when the viewpoint is moved in the real space varies according to the environment of the virtual space (such as weather, humidity, driving environment, condition of the windshield, etc.) and the state of the user (age, gender, eyesight, eye health, eye opening degree, dominant eye, etc.). By changing the transition completion time in the processing of the virtual space image based on the environmental conditions and user conditions so as to match such properties of the eyes, the visibility of the generated virtual space image can be effectively approximated to the way of seeing in the real space.

[0071] Also, in the virtual space image generation device 2 of the present embodiment, when the viewpoint of the user U moves and the peripheral area of the destination viewpoint is located outside the peripheral area of the source viewpoint, a virtual space image is generated in which the transition completion time changes based on a predetermined condition. Thereby, even when the viewpoint of the user U moves greatly to a distant position outside the peripheral area of the viewpoint, the visibility of the entire peripheral area of the destination viewpoint can be surely approximated to the way of seeing in the real space.

[0072] As described above, the embodiments of the present invention have been described. However, the present invention is not limited to the described embodiments, and various modifications and changes are possible based on the technical idea of the present invention. For example, in the described embodiments, an example in which the viewpoint of the user U moves in the depth direction (receding direction or approaching direction) within the virtual space has been described. However, even in the case of a viewpoint movement without movement in the depth direction, for example, when the viewpoint of the user U moves between the left and right tail lamps of the preceding vehicle displayed in the virtual space image, by applying the virtual space image generation technology according to the present invention, a virtual space image close to the way of seeing in the real space can be realized.

[0073] In the above-described embodiment, an example in which the visibility within the peripheral region of the destination viewpoint is maintained in the second state during the period from the completion of the viewpoint movement of the user U until the elapse of the delay times L1 and L2 has been described. However, a virtual space image in which the visibility slightly increases during the delay times L1 and L2 may be generated. Further, in the above-described embodiment, an example in which the temporal change in visibility at the transition time follows a function such as Equation (1) has been described. However, the visibility may be changed using a map associating the visibility state with the focal length.

[0074] When the temporal change in visibility at the transition time follows the function shown in Equation (1), as the diopter Do (the reciprocal of the focal length at the start of the viewpoint movement) and the diopter Dt (the reciprocal of the focal length at the end of the viewpoint movement) of Equation (1), in the above-described embodiment, an example in which the reciprocals of the virtual focal lengths Fn’ and Ff’ of the eye E of the user U are used has been shown. However, instead of the reciprocals of Fn’ and Ff’, the reciprocals of the distances Zn and Zf in the depth direction may be used.

[0075] In addition, it is also possible to change the degree of change in visibility in the visibility change region according to the deviation between the focal length at the start of the viewpoint movement and the focal length at the end of the viewpoint movement.

Explanation of Reference Numerals

[0076] 1… Driving simulator system 2… Virtual space image generation device 3… Sensor 4… Image forming device 11… Viewpoint detection unit 12… Input unit 13… Storage unit 14… Image generation unit 15… Display control unit A, An, Af… Viewpoint peripheral region F, Fn, Fn’, Ff, Ff’… Focal length L1, L2… Delay time P, Pn, Pf… Position of viewpoint Pn’, Pf’… Position of viewpoint in virtual space T1, T2… Transition completion time U… User V1… First state V2… Second state α1, α2… Transition time τ… Time constant

Claims

1. In a virtual space image generation apparatus that generates a virtual space image including a visibility change region whose visibility changes based on the movement of a user's viewpoint, when the user's viewpoint moves, the transition completion time required for the visibility of the visibility change region to transition between a first state and a second state different from the first state for the user who has moved the viewpoint changes based on a predetermined condition, and an image generation unit that generates the virtual space image is provided. A virtual space image generation apparatus characterized by that.

2. In a virtual space image generation apparatus that generates a virtual space image including a visibility change region whose visibility changes based on the movement of a user's viewpoint, when the user's viewpoint moves, an image generation unit that generates the virtual space image in which the transition completion time required for the visibility of the visibility change region to transition between a first state and a second state different from the first state changes based on a predetermined condition is provided, The virtual space image generation apparatus, wherein the transition completion time includes a delay time from when the user's viewpoint movement is completed until the transition is started and a transition time from when the transition is started until it is completed.

3. In a virtual space image generation apparatus that generates a virtual space image including a visibility change region whose visibility changes based on the movement of a user's viewpoint, when the user's viewpoint moves, an image generation unit that generates the virtual space image in which the transition completion time required for the visibility of the visibility change region to transition between a first state and a second state different from the first state changes based on a predetermined condition is provided, The virtual space image generation apparatus, wherein the image generation unit is configured to generate the virtual space image in which the transition completion time changes according to the moving direction of the viewpoint, using the moving direction of the user's viewpoint as the predetermined condition.

4. In a virtual space image generation apparatus that generates a virtual space image including a visibility change region whose visibility changes based on the movement of a user's viewpoint, when the user's viewpoint moves, an image generation unit that generates the virtual space image in which the transition completion time required for the visibility of the visibility change region to transition between a first state and a second state different from the first state changes based on a predetermined condition is provided, The transition completion time includes a delay time from when the user's viewpoint movement is completed until the transition is started and a transition time from when the transition is started until it is completed, The image generation unit is configured to generate the virtual space image in which the transition time changes according to the moving direction of the viewpoint, using the moving direction of the user's viewpoint as the predetermined condition. A virtual space image generation apparatus characterized by this.

5. The image generation unit is configured to generate the virtual space image in which the transition time when the moving direction of the viewpoint is the direction of moving away is longer than the transition time when the moving direction of the viewpoint is the direction of approaching. The virtual space image generation apparatus according to claim 4, characterized by this.

6. The image generation unit is configured to generate the virtual space image in which the transition time changes according to the change in the focal length of the user's eyes corresponding to the change in the virtual position of the user's viewpoint in the virtual space, using the change in the focal length of the user's eyes as the predetermined condition. The virtual space image generation apparatus according to claim 5, characterized by this.

7. The image generation unit is configured to generate the virtual space image in which the delay time in the viewpoint movement in the direction of moving away is two times or more longer than the delay time in the viewpoint movement in the direction of approaching, and the time constant of the function that determines the transition time in the viewpoint movement in the direction of moving away is two times or more the time constant of the function that determines the transition time in the viewpoint movement in the direction of approaching. The virtual space image generation apparatus according to claim 6, characterized by this.

8. In the viewpoint movement in the direction of moving away, the delay time is set in the range from 0.05 seconds to 0.2 seconds, and the time constant is set in the range from 0.05 to 0.2, In the viewpoint movement in the direction of approaching, the delay time is set in the range from 0 seconds to 0.05 seconds, and the time constant is set in the range from 0 to 0.

1. The virtual space image generation apparatus according to claim 7, characterized by this.

9. The image generation unit is configured to generate the virtual space image in which the transition completion time changes based on the conditions related to the environment of the virtual space, using the conditions related to the environment of the virtual space as the predetermined condition. The virtual space image generation apparatus according to any one of claims 1 to 8, characterized by this.

10. The image generation unit is configured to generate the virtual space image in which the transition completion time changes based on the conditions related to the user, using the conditions related to the user as the predetermined condition. The virtual space image generation apparatus according to any one of claims 1 to 8, characterized by this.

11. The virtual space image generation apparatus according to any one of claims 1 to 10, wherein the image generation unit is configured to generate the virtual space image in which the transition completion time changes based on the predetermined condition when the viewpoint of the user moves and the area around the destination viewpoint is located outside the area around the source viewpoint.

12. In a virtual space image generation method for generating a virtual space image including a visibility change area whose visibility changes based on a user's viewpoint movement, A virtual space image generation method for generating the virtual space image in which a transition completion time required for the visibility of the visibility change area displayed to the user who has moved the viewpoint to transition between a first state and a second state different from the first state changes based on a predetermined condition when the viewpoint of the user moves.

Images