Image processing device, method, and program

Abstract

One embodiment of an image processing device according to the present invention comprises a setting unit and a display control unit. The setting unit sets a movement route for when a viewpoint position inside a virtual space is automatically moved. The display control unit displays, on a display, an image which is inside the field of view and in which the line-of-sight direction from the viewpoint position is oriented rearwards along the movement route while the viewpoint position is automatically moved forward along the movement route.

Description

Image processing device, method and program

[0001] One aspect of the present invention relates to an image processing device, method, and program.

[0002] In recent years, image processing devices that create virtual spaces, such as virtual reality (VR), have been widely used. Generally, such image processing devices use computer graphics (CG) technology or the like to create virtual spaces that represent unreal worlds that many people can log into. Recently, in addition to this, image processing devices that create 3D virtual spaces that simulate the real world based on data measured from the real space have become known.

[0003] Such image processing devices construct a 3D virtual space based on, for example, 3D point cloud data (see, for example, Non-Patent Documents 1 and 2). One method for acquiring 3D point cloud data is the Light Detection and Ranging (LiDAR) method (see, for example, Non-Patent Document 3). The LiDAR method measures the distance to an object, the direction, and the shape of the object by irradiating the object with laser light and measuring the round-trip time it takes for the laser light to reflect off the object and return. The LiDAR method also acquires 3D point cloud data by collecting points representing the results of scanning with the laser light.

[0004] Here, 3D point cloud data refers to data that collects information on points in a virtual space that have attribute information such as three-dimensional coordinate values ​​(X, Y, Z) and color information (R, G, B) (see, for example, Non-Patent Document 4). Because 3D point cloud data is three-dimensional information at the time of measurement, it has the advantage of being able to construct a wide-area 3D virtual space at low cost compared to constructing a 3D space from CG or photographs. Another advantage of 3D point cloud data is that it can construct a 3D virtual space that simulates real space with high accuracy. The image processing device displays a 3D virtual space (point cloud metaverse) based on the 3D point cloud data on VR goggles such as an HMD (Head Mounted Display). This allows the image processing device to move the viewpoint of a user wearing VR goggles within the 3D virtual space that simulates reality.

[0005] Meanwhile, while a user wears VR goggles and moves through a virtual space for a long period of time, visual information about movement and information about stillness from the balance organs are sent to the brain. At this time, the user may experience VR sickness, which makes the user feel unwell due to a mismatch in the information sent to the brain (see, for example, Non-Patent Document 5). Methods (a) to (c) are known as techniques for reducing such VR sickness.

[0006] (a) is a method of using teleportation mode to instantly move to a desired location (see, for example, Non-Patent Document 5).

[0007] (b) is a method of applying a vignette effect (or tunnel effect) that darkens the surrounding area on the screen when the player moves (see, for example, Non-Patent Documents 5 to 7). Non-Patent Document 6 describes an illustration of the vignette effect (tunnel effect). Non-Patent Document 7 describes a view-limiting mask that corresponds to a non-circular vignette effect.

[0008] (c) A method of providing sound and vibration linked to vision to compensate for the lack of visually linked sensory information (see, for example, Non-Patent Document 8).

[0009] "Launch of the 'TENGUN Ogijima Project' to Promote Regional Co-creation," [online], November 15, 2022, Nippon Telegraph and Telephone Corporation, [Retrieved July 5, 2024], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 11 / 15 / 221115b.html> Matsumoto Takahiro, Komazaki Kei, Maki Yuichi, Chiaki Hiroshi, Mochizuki Takayoshi, "TENGUN Ogijima: Research on highly realistic VR through visual, auditory, and tactile sensations based on measurement data,” [online], March 1, 2023, Information Processing Society of Japan Interaction 2023, [Retrieved July 5, 2024], Internet <URL: https: / / www.interaction-ipsj.org / proceedings / 2023 / data / pdf / 2B-17.pdf> “What is LiDAR? "Explaining the principles and uses of optical sensors, which are attracting attention in autonomous driving," [online], September 25, 2023, MEITEC, [Retrieved July 5, 2024], Internet <URL: https: / / engineer.fabcross.jp / archeive / 211109_lidar.html> "What is point cloud data? Acquisition methods and use examples," [online], May 14, 2024, Informatix Co., Ltd., [Retrieved July 5, 2024], Internet <URL: https: / / club.informatix.co.jp / ?p=1125> "The mechanism behind 'VR sickness' in virtual spaces, The Solution," [online], August 8, 2021, WIRED, [Retrieved July 5, 2024], Internet <URL: https: / / wired.jp / 2021 / 08 / 08 / how-to-reduce-motion-sickness-virtual-reality / > Daiki Kayaba, Yoshiaki Miyashita, "Investigation of the Sense of Speed ​​and Sickness Caused by Dynamic Tunneling in a VR Racing Game," [online], August 2021, Entertainment Computing Symposium (EC2021), [Retrieved July 5, 2024], Internet <URL: https: / / ipsj.ixsq.nii.ac.jp / ej / ?action=repository_uri&item_id=212575&file_id=1&file_no=1>Daiki Kayaba, Yoshiaki Miyashita, “VR Sickness Reduction Method Using Noncircular Field-of-View Restriction Based on Optical Flow,” [online], Information Processing Society of Japan Research Report, [Retrieved July 5, 2024], Internet <URL: https: / / www.miyashita.com / researches / 66qeAOtF6mhWEbr8asH6uM> “Reducing VR Sickness Using Engine Sound and Vibration – Demonstrated by Psychological Experiments Using a VR Motorcycle Simulator,” [online], May 12, 2020, Shizuoka University, [Retrieved July 5, 2024], Internet <URL: https: / / www.shizuoka.ac.jp / news / detail.html?CN=6399>.

[0010] Although there are no particular problems with the above methods (a) to (c), the inventors have found that there is room for improvement in the following points of inconvenience.

[0011] The method (a) described above has the disadvantage that, since the user moves to the desired location instantaneously, the user cannot see the scenery on the way to the desired location, which reduces the sense of realism.

[0012] The method (b) above has the disadvantage that the periphery of the screen is darkened, which may cause some users to feel uncomfortable.

[0013] The method (c) above has the disadvantage that it is necessary to add new hardware in addition to the VR goggles to provide sound and vibration linked to the visual sensation, which increases costs.

[0014] Therefore, from the perspective of improving these inconveniences, it is desirable to be able to reduce VR sickness when moving within a virtual space without moving suddenly, without darkening the surroundings on the screen, and without adding new hardware.

[0015] The present invention has been made in consideration of the above circumstances, and provides a technology that can reduce VR sickness when moving within a virtual space without moving suddenly, without darkening the surroundings on the screen, and without adding new hardware.

[0016] In order to solve the above problem, one aspect of the image processing device according to the present invention includes a setting unit and a display control unit. The setting unit sets a movement route for automatically moving a viewpoint position in a virtual space. The display control unit displays, on a display, an image within a field of view in which a line of sight from the viewpoint position is directed backward along the movement route while the viewpoint position is automatically moved forward along the movement route.

[0017] According to one aspect of the present invention, while the viewpoint position in the virtual space is automatically moved forward along the movement route, an image within the field of view is displayed on the display, with the line of sight from the viewpoint position directed backward along the movement route. As a result, while the user is automatically moving forward in the virtual space, the surrounding scenery in the virtual space always experiences a flow from the foreground to the background, thereby reducing VR sickness. Furthermore, according to another aspect of the present invention, when moving within the virtual space, the movement is not instantaneous, the surroundings on the screen are not darkened, and no new hardware is added.

[0018] In other words, according to one aspect of the present invention, when moving within a virtual space, VR sickness can be reduced without moving instantaneously, without darkening the surroundings on the screen, and without adding new hardware.

[0019] FIG. 1 is a block diagram showing an example of the configuration of an image processing device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of the processing unit shown in FIG. 1. FIG. 3 is a flowchart illustrating an example of operation in an embodiment. FIG. 4 is a schematic diagram illustrating an example of operation in an embodiment. FIG. 5 is a schematic diagram illustrating an example of a map image in an embodiment. FIG. 6 is a schematic diagram illustrating the orientation of a user during automatic movement in an embodiment. FIG. 7 is a schematic diagram illustrating the flow of scenery during automatic movement in an embodiment. FIG. 8 is a flowchart illustrating operation in a first modified example of an embodiment. FIG. 9 is a schematic diagram illustrating the orientation of a user during automatic movement in a second modified example of an embodiment. FIG. 10 is a schematic diagram illustrating setting of a movement route in a third modified example of an embodiment.

[0020] An embodiment of the present invention will be described below with reference to the drawings.

[0021] 1 is a block diagram showing an example of the configuration of an image processing device according to an embodiment of the present invention. The image processing device 100 is a computer capable of outputting an image of a virtual space that mimics the real world based on acquired 3D point cloud data. The image processing device 100 may be, for example, a personal computer. Note that the image processing device 100 may also be configured as a server computer installed on the web or in the cloud.

[0022] The image processing device 100 includes a processing unit 110 that uses a hardware processor such as a central processing unit (CPU), and is connected to the processing unit 110 via a bus 150 with a storage unit having a program storage unit 130 and a data storage unit 131, and an input / output I / F (interface) unit 140.

[0023] The input / output I / F unit 140 is connected to an external storage device 102 and a head mounted display (HMD) 104 so as to be able to communicate wirelessly or via a wire. The input / output I / F unit 140 may set various data in the data storage unit 131 in response to a user's operation, or may input operation information representing various commands and the like to the processing unit 110. Note that the operation information can also be input from an input interface of the HMD 104. The term "user" may also be interpreted as "operator."

[0024] The external storage device 102 is a storage device that stores map data and 3D point cloud data, and the stored contents can be read by the image processing device 100. The map data includes image data representing a map image of a bird's-eye view of a virtual space created based on the 3D point cloud data, and additional information related to the map image. The additional information may include, for example, information regarding a route along which the viewpoint position can be automatically moved on the map image, and coordinate information (X, Y, Z) in the virtual space corresponding to coordinate information (X, Y, Z) on the map image. The 3D point cloud data is data used to construct a 3D point cloud model that simulates a real space in a virtual space, and is composed of a collection of multiple point data. The point data that constitutes the 3D point cloud data includes coordinate information (X, Y, Z) in the virtual space and color information (R, G, B). The 3D point cloud data is acquired, for example, using LiDAR (Light Detection and Ranging) and stored in the external storage device 102.

[0025] The HMD 104 is a VR goggle worn by a user and includes, for example, a memory, a processor, a display, an input interface, and a communication interface. The memory stores programs and data for the HMD 104. The processor operates according to the programs in the memory and displays images output from the image processing device 100 on the display. Examples of the displayed images include an image of the field of view along the line of sight from a viewpoint in the virtual space, and a map image showing an overview of the entire virtual space. The input interface inputs operation information to the image processing device 100 in response to user operations. The controller of the HMD 104 can be used as the input interface as appropriate. The communication interface enables data transmission and reception between the HMD 104 and the image processing device 100. A protocol defined by the communication network is used as the communication protocol. For example, a wired LAN is used as the communication interface. For example, an interface adopting a low-power wireless data communication standard such as a wireless LAN or Bluetooth (registered trademark) may also be used as the communication interface. It should be noted that a flat display may be used instead of the HMD 104. The HMD 104, the VR goggles, and the flat display are each an example of a display.

[0026] The input / output I / F unit 140 may include a communication interface. By including the communication interface in the input / output I / F unit 140, 3D point cloud data transmitted from a LiDAR that measures the distance, direction, position, shape, and the like of an object in real space can be acquired without going through the external storage device 102.

[0027] The program storage unit 130 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written and read as needed with a nonvolatile memory such as a read-only memory (ROM) as a storage medium. It stores middleware such as an operating system (OS) as well as application programs necessary for executing various control operations according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The program may be installed on the computer in advance from a network or a non-transitory computer-readable storage medium, or may be pre-recorded on the computer. In either case, the program is executed by the processor to cause the computer to function as the image processing device 100 according to an embodiment and each modification.

[0028] The data storage unit 131 is, for example, a combination of a nonvolatile memory such as an SSD that can be written to and read from at any time and a volatile memory such as a RAM (Random Access Memory) as a storage medium. The data storage unit 131 stores, for example, map data and 3D point cloud data. The map data and 3D point cloud data are acquired from the external storage device 102 or LiDER, but may also be acquired via a network from another terminal device or a server computer on the web or cloud.

[0029] The processing unit 110 executes processing related to manual movement and automatic movement within the virtual space. For example, during manual movement, the processing unit 110 controls the viewpoint position and line of sight direction within the virtual space in response to a user's operation, and causes the HMD 104 to display an image within the field of view along the line of sight from the current viewpoint position. During automatic movement, the processing unit 110 functions as a setting unit that sets a movement route when automatically moving the viewpoint position within the virtual space. For example, during automatic movement, the processing unit 110 also functions as a display control unit that causes the HMD 104 to display an image within the field of view in which the line of sight direction from the viewpoint position is directed backward along the movement route while automatically moving the viewpoint position forward along the movement route.

[0030] Specifically, for example, as shown in FIG. 2, the processing unit 110 includes a travel mode management unit 111, a travel route calculation unit 112, an automatic travel unit 113, an orientation adjustment unit 114, a current location memory unit 115, a manual travel unit 116, a reading unit 117, a map creation unit 118, a calculation unit 119, a rendering unit 120, and an output unit 121.

[0031] Here, among the above-mentioned units 111 to 121, the movement route calculation unit 112 relating to the movement route is an example of the setting unit described above. Furthermore, the automatic movement unit 113, orientation adjustment unit 114, current location storage unit 115, calculation unit 119, rendering unit 120, and output unit 121 relating to automatic movement are examples of the display control unit described above. However, the current location storage unit 115, calculation unit 119, rendering unit 120, and output unit 121 are not limited to automatic movement, but are also used for manual movement. Furthermore, the processing unit 110 is not limited to the processing of the above-mentioned units 111 to 121, and is capable of executing any processing in response to user operation.

[0032] Each of the above units 111 to 121 is realized by causing a hardware processor of the processing unit 110 to execute an application program stored in the program storage unit 130. Note that some or all of the above units 111 to 121 may be realized using hardware including integrated circuits such as LSIs (Large Scale Integration) and ASICs (Application Specific Integrated Circuits).

[0033] The travel mode management unit 111 manages whether the current travel mode is an automatic travel mode or a manual travel mode, which identify travel methods within a virtual space. The travel mode management unit 111 also controls other units depending on the travel mode. For example, the travel mode management unit 111 controls the travel route calculation unit 112 and the automatic travel unit 113 for the automatic travel mode. The travel mode management unit 111 also controls the manual travel unit 116 for the manual travel mode.

[0034] When a destination is specified on a map image of the virtual space, the movement route calculation unit 112 calculates a route connecting the destination and the current location from among the routes on the map image, and sets the calculation result as a movement route. The movement route is a route used when automatically moving the viewpoint position in the virtual space.

[0035] In the automatic movement mode, the automatic movement unit 113 automatically moves the viewpoint position in the virtual space forward along the movement route.

[0036] During automatic movement of the viewpoint position by the automatic movement unit 113, the orientation adjustment unit 114 automatically adjusts the line of sight from the viewpoint position so that it is always "backward (180 degrees opposite)" with respect to the movement direction (direction of travel). That is, in automatic movement mode, during automatic movement of the viewpoint position in the virtual space, the orientation adjustment unit 114 adjusts the line of sight from the viewpoint position so that it faces backward along the movement route. Note that "backward" is the opposite direction to "forward" and is typically a direction approximately 180 degrees different from "forward." However, the "backward" direction may be any direction within a certain range that includes a direction approximately 180 degrees different from "forward." For example, the "backward" direction may be a direction within a range of approximately 90 degrees centered on a direction approximately 180 degrees different from "forward." Note that the approximately 90-degree range is just an example, and the "backward" direction may also be within a range of approximately 120 degrees. That is, any direction within a 180-degree range (±90 degrees) centered on a direction approximately 180 degrees different from "forward" can be used as the "backward" direction.

[0037] The current location storage unit 115 stores coordinate information of the viewpoint position in the virtual space as the current location in the manual movement mode and the automatic movement mode. Furthermore, the current location storage unit 115 stores a predetermined start position in the virtual space as the current location upon login.

[0038] In the manual movement mode, the manual movement unit 116 controls the viewpoint position and line of sight direction in the virtual space in response to a user operation.

[0039] The reading unit 117 reads map data and 3D point cloud data from the external storage device 102. The reading unit 117 may read 3D point cloud data corresponding to the entire virtual space, or may read 3D point cloud data corresponding to a predetermined range of virtual space including current location information. Furthermore, the reading unit 117 may read 3D point cloud data corresponding to a predetermined range of virtual space including a route connecting the current location and the destination based on the current location information and the destination information.

[0040] The map creation unit 118 creates a map image based on the loaded map data and the current location, and sends the map image to the calculation unit 119 .

[0041] The calculation unit 119 creates a 3D model of the virtual space based on the read 3D point cloud data, and creates virtual space data including the viewpoint position in the virtual space, the line of sight direction from the viewpoint position, and the created 3D model. The calculation unit 119 also sends the created virtual space data to a rendering unit. The calculation unit 119 also sends the sent map image to a rendering unit 120.

[0042] The rendering unit 120 creates an image to be displayed on the HMD 104 based on the transmitted virtual space data and map image. For example, the rendering unit 120 performs rendering to create an image of a 3D model viewed from a viewpoint along a viewing direction based on the transmitted virtual space data, and creates an image (still image or video) of the field of view along the line of sight from the viewpoint in the virtual space. The rendering unit 120 may also create an image in which a map image is superimposed on a portion of the image in the field of view, or an image in which a map image is arranged alongside the image in the field of view. However, from the perspective of maintaining the sense of realism of the virtual space, it is preferable not to display the map image or to arrange a reduced map image at the edge of the entire screen. In either case, the rendering unit 120 sends the created image to the output unit 121.

[0043] The output unit 121 outputs the image sent from the rendering unit 120 to the HMD 104 and causes the HMD 104 to display the image.

[0044] Next, an example of the operation of the image processing device configured as described above will be described using the flowchart in FIG. 3 and the schematic diagrams in FIGS. 4 to 7. The following description will cover the steps from data reading to manual movement (steps ST1 to ST6), setting the movement route for automatic movement (steps ST7 to ST8), switching to automatic movement mode and the operation during automatic movement (steps ST9 to ST11), and switching to manual movement mode (step ST12). Roughly speaking, steps ST1 to ST6 represent the process of manual movement (low-speed movement), including short-distance movement and direction changes. Steps ST7 to ST8 represent the process of displaying a map image showing the entire virtual space in the virtual space, specifying the destination location (destination) on the map image, and automatically calculating the movement route by comparing it with the current location when traveling to a long-distance destination. Steps ST9 to ST11 represent the process of instructing movement to start automatic movement mode, automatically moving to the destination without slowing down while always keeping the user's line of sight facing "backward." During automatic movement, the scenery in the virtual space flows backward (from the front to the back), so the user moves to the destination while watching the scenery. Step ST12 represents a process in which, upon arrival at the destination, the automatic movement mode is canceled and the mode is returned to manual movement mode operated by the user. In the following description, each of the units 111 to 121 may be read as the processing unit 110.

[0045] 3 and 4, in step ST1, the reading unit 117 reads map data and 3D point cloud data from the external storage device 102 in response to, for example, a user operation. The reading unit 117 also sends the read map data to the map creation unit 118 and sends the 3D point cloud data to the calculation unit 119.

[0046] (Step ST2) After step ST1, in step ST2, the travel mode management unit 111 receives current location information from the HMD 104. The travel mode management unit 111 sets the received current location information in the current location storage unit 115. The travel mode management unit 111 also manages the travel mode as a manual travel mode. During the manual travel mode, the manual travel unit 116 updates the current location information in the current location storage unit 115 based on operation information input from the HMD 104, and sends the updated current location information to the calculation unit 119. During the manual travel mode, the manual travel unit 116 also sends the line of sight direction from the viewpoint position to the calculation unit 119 based on operation information input from the HMD 104.

[0047] (Step ST3) After step ST2, in step ST3, the map creation unit 118 creates a map image with the current location superimposed thereon from the sent map data and current location information, and sends the map image to the output unit 121 via the calculation unit 119 and the rendering unit 120. The output unit 121 outputs the sent map image to the HMD 104, and causes the HMD 104 to display the map image.

[0048] (Step ST4) After step ST3, in step ST4, the calculation unit 119 acquires the transmitted current location information as a viewpoint position in the virtual space, and creates a 3D model in the virtual space within a predetermined range including the viewpoint position based on the transmitted 3D point cloud data. The calculation unit 119 also creates virtual space data including the viewpoint position in the virtual space, the line of sight from the viewpoint position, and the created 3D model, and sends the virtual space data to the rendering unit 120.

[0049] (Step ST5) After step ST4, in step ST5, the rendering unit 120 performs rendering to create an image of the 3D model viewed from the viewpoint along the viewpoint direction based on the transmitted virtual space data. As a result, the rendering unit 120 creates an image of the field of view along the line of sight from the viewpoint position in the virtual space. The rendering unit 120 then sends the created image of the field of view to the output unit 121. The output unit 121 displays the image of the field of view on the HMD 104. The rendering unit 120 may also send to the output unit 121 an image in which a map image is superimposed on a portion of the image of the field of view, or an image in which the map image is arranged alongside the image of the field of view. In this case, the output unit 121 displays the sent image on the HMD 104.

[0050] (Step ST6) After step ST5, in step ST6, the processing unit 110 manually moves the viewpoint position in response to a user operation. That is, the processing unit 110 controls the viewpoint position and line of sight in the virtual space in response to a user operation, and causes the HMD 104 to display an image of the field of view along the line of sight from the current viewpoint position. Note that, if a map image is being displayed in addition to the image of the field of view, the processing unit 110 causes the HMD 104 to display a map image on which an updated current location (viewpoint position) is superimposed in response to a user operation. The processing unit 110 may also switch between displaying and not displaying the map image in response to a user operation.

[0051] (Step ST7) After step ST6, in step ST7, the travel mode management unit 111 determines whether a destination has been specified, depending on whether an operation to set destination information on the map image has been performed. If not, the process returns to step ST2, and steps ST2 to ST7 are executed in the same manner as described above. On the other hand, if the result of the determination in step ST7 is that a destination has been specified, the travel mode management unit 111 proceeds to step ST8.

[0052] (Step ST8) After step ST7, in step ST8, the travel mode management unit 111 activates the travel route calculation unit 112. When a destination is specified on a map image of the virtual space, the travel route calculation unit 112 calculates a route on the map image connecting the destination and the current location. For example, as shown in FIG. 5, when candidate (A) is specified as destination A among candidate destinations (A) to (D) on a map image g1 representing an island surrounded by sea, the travel route calculation unit 112 calculates a route connecting the current location indicated by the arrow Cs to destination A. Note that destination A can be specified, for example, by clicking candidate (A) or by key input. The travel route calculation unit 112 then sets the calculated route as the travel route rt. This sets the travel route rt when automatically moving the viewpoint position in the virtual space.

[0053] (Step ST9) After step ST8, in step ST9, when the movement mode management unit 111 receives operation information indicating execution of automatic movement (movement command), it changes the manual movement mode to the automatic movement mode. Accordingly, the movement mode management unit 111 stops the manual movement unit 116 and starts the automatic movement unit 113. As a result, the processing unit 110 executes the automatic movement mode.

[0054] (Step ST10) After step ST9, in step ST10, the processing unit 110 automatically moves the viewpoint position in the virtual space forward along the movement route while displaying on the HMD 104 an image in the field of view in which the line of sight from the viewpoint position is directed backward along the movement route. At this time, the automatic movement unit 113 automatically moves the viewpoint position p1 in the virtual space forward to d1 along the movement route rt, as shown in FIG. 6 . Note that the movement speed during automatic movement is faster than the movement speed during manual movement. Furthermore, during the automatic movement of the viewpoint position p1 in the virtual space, the orientation adjustment unit 114 adjusts the line of sight from the viewpoint position p1 to be directed backward d2 along the movement route rt. In the example shown in FIG. 6 , the backward d2 is 180 degrees opposite to the forward d1. As a result, the HMD 104 displays an image g2 in the field of view in which the line of sight from the viewpoint position is directed backward along the movement route, as shown in FIG. 7 . In the image g2 in the field of view, the scenery in the virtual space flows backward (from the front to the back) as indicated by the multiple arrows a2. Note that the multiple arrows a2 are shown for ease of understanding and are not actually displayed on the HMD 104 in order to maintain a sense of realism. The user moves to destination A while watching the scenery flow by in this way.

[0055] (Step ST11) After step ST10, in step ST11, when the automatic movement unit 113 causes the viewpoint position in the virtual space to arrive at the destination along the movement route, it ends the automatic movement of the viewpoint position and sends a notification of the end of automatic movement to the movement mode management unit 111.

[0056] (Step ST12) After step ST11, in step ST12, when the movement mode management unit 111 receives a signal indicating that the automatic movement has ended, it changes the automatic movement mode to the manual movement mode. Accordingly, the movement mode management unit 111 stops the automatic movement unit 113 and activates the manual movement unit 116. This starts the manual movement mode. That is, the processing unit 110 returns to step ST2 and executes steps ST2 to ST7 in the same manner as described above.

[0057] As described above, according to one embodiment, the processing unit 110 sets a movement route rt for automatically moving the viewpoint position p1 in the virtual space. Furthermore, while the processing unit 110 automatically moves the viewpoint position p1 forward d1 along the movement route rt, the processing unit 110 causes the HMD 104 to display an image g2 in the field of view in which the line of sight from the viewpoint position p1 is directed backward d2 along the movement route rt. In this way, by displaying a backward image during automatic forward movement in the virtual space, VR sickness can be reduced when moving in the virtual space without instantaneous movement, without darkening the surroundings on the screen, or without adding new hardware.

[0058] Specifically, for example, when moving within a virtual space using the HMD 104, it is possible to reduce VR sickness by automatically moving while viewing the scenery as the user moves, without darkening the surroundings of the screen while moving, without slowing down, and always keeping the user facing backwards, without adding any devices other than the HMD 104 to the image processing device 100.

[0059] Specifically, when moving within a virtual space using the HMD 104, if the user is moving a short distance, the user manually slows down the movement speed. In contrast, when moving to a long-distance destination, a map image showing the entire virtual space is displayed, and the user specifies the desired location (destination). The route to the destination is automatically set, and the user's line of sight faces "backward" to the destination, automatically moving at a speed equal to or greater than a certain level. The user's sensation at this time is similar to that of traveling with their back to the direction of travel on a train, for example. Furthermore, the system keeps track of the user's current line of sight and the direction of travel of the viewpoint, and automatically adjusts the line of sight to always be 180 degrees opposite (backward) to the direction of travel of the viewpoint during automatic movement. This allows the user to experience the surrounding scenery in the virtual space as constantly flowing from the foreground to the background during automatic movement, thereby reducing VR sickness.

[0060] Therefore, the user can automatically move to a distant destination while always facing backwards, without adding any additional devices other than the HMD 104, without masking the periphery of the screen in black, and without slowing down. As a result, it is possible to reduce VR sickness when moving in a virtual space using an HMD.

[0061] Additionally, it is known that VR sickness is less likely to occur when moving within a virtual space by reducing the movement speed and moving slowly. However, when a wide area such as the point cloud metaverse is converted into a virtual space, a slow movement speed can cause the inconvenience of taking too long to travel long distances. In contrast, according to one embodiment, automatic movement can be performed without slowing down the speed, thereby eliminating this inconvenience.

[0062] Furthermore, VR sickness is more pronounced when the user moves forward while facing forward, and less pronounced when the user moves backward. However, manual operation of moving backward can cause a loss of sense of direction, making it difficult to accurately maintain the direction of movement or to properly turn around a curve. In contrast, according to one embodiment, the user's line of sight is turned backward and the vehicle automatically moves along the travel route rt, eliminating such inconveniences.

[0063] Furthermore, according to one embodiment, when a destination A is specified on the map image g1 of the virtual space, the processing unit 110 sets, among the routes on the map image g1, a route connecting the destination A and the current location as the travel route rt. This not only achieves the above-mentioned effects, but also makes it possible to set a travel route rt from any current location to the destination simply by specifying the destination.

[0064] <First Modification> In the first embodiment, continuous automatic movement from the current location to the destination is performed, but this is not limiting. Specifically, the movement mode during automatic movement may be set or changed from the HMD 104. For example, the processing unit 110 may pause or stop the automatic movement in response to a user operation. Here, pausing the automatic movement means temporarily stopping the automatic movement in automatic movement mode. When the automatic movement is paused, the viewpoint position and line of sight direction of the image in the virtual space are not changed. Furthermore, stopping the automatic movement means stopping the automatic movement in automatic movement mode and changing the movement mode to manual movement mode. When the automatic stop is stopped, the viewpoint position and line of sight direction of the image in the virtual space are changed in response to a user operation. In other words, pausing and stopping differ in whether the viewpoint position and line of sight direction can be changed. Setting or changing the movement mode during automatic movement may be performed by, for example, inputting operation information related to a movement mode setting / changing command from the HMD 104. The movement mode setting / changing command is a command that indicates pausing or stopping the automatic movement. Other configurations are the same as those of the first embodiment. The processing unit 110 is an example of a display control unit.

[0065] Specifically, for example, the processing unit 110 is executing step ST10-1, similar to step ST10 described above, as shown in Fig. 8. At this time, the travel mode management unit 111 determines whether or not operation information has been received (step ST10-2), and if so, proceeds to step ST10-3, and if not, proceeds to step ST10-4.

[0066] In step ST10-3, when the travel mode management unit 111 receives operation information from the HMD 104 indicating, for example, pausing automatic movement, it controls the automatic movement unit 113 in accordance with the operation information. The automatic movement unit 113 pauses automatic movement in accordance with the control of the travel mode management unit 111. As a result, the flow of scenery is paused in the HMD 104, and a still image of the field of view along the line of sight from the current viewpoint is displayed. At this time, the user can carefully enjoy the scenery at the paused location in the still image. Thereafter, when the travel mode management unit 111 receives operation information from the HMD 104 indicating cancellation of the pause of automatic movement, it controls the automatic movement unit 113 in accordance with the operation information to resume automatic movement. As a result, the automatic movement unit 113 proceeds to step ST10-1 and resumes automatic movement.

[0067] Also, for example, in step ST10-3, when the movement mode management unit 111 receives operation information from the HMD 104 indicating the cancellation of automatic movement, it changes the automatic movement mode to the manual movement mode in accordance with the operation information. As a result, the processing unit 110 returns to step ST2 and executes steps ST2 to ST7, as described above. At this time, the user can view the scenery from a desired viewpoint position and viewpoint direction by manually moving from the location where automatic movement was canceled to change the viewpoint position and line of sight direction.

[0068] On the other hand, in step ST10-4, the automatic movement unit 113 determines whether the viewpoint position in the virtual space has arrived at the destination, and if not, returns to step ST10-1 and executes steps ST10-1 to ST10-4 as described above. On the other hand, if the result of the determination in step ST10-4 is that the viewpoint position has arrived at the destination, step ST10 is ended and the process proceeds to step ST11.

[0069] According to the first modification as described above, the processing unit 110 pauses or stops the automatic movement in response to a user operation. Therefore, in addition to the effects of the first embodiment, for example, when the automatic movement is paused, it is possible to enjoy the scenery at your leisure during the automatic movement. Also, for example, when the automatic movement is stopped, it is possible to enjoy the scenery from a desired viewpoint position and line of sight from the middle of the automatic movement.

[0070] <Second Modification> In the first embodiment and the first modification, the line of sight during automatic movement is directed backward d2 along the movement route rt as shown in FIG. 6 , but this is not limited thereto. For example, the processing unit 110 may, in response to a user operation, cause the HMD 104 to display an image of the field of view in which the line of sight during automatic movement is changed from backward d2 to side d3, as shown in FIG. 9 . In the example shown in FIG. 9 , the side d3 is a direction approximately 90 degrees to the side, approximately midway between the forward d1 and the backward d2. That is, the side d3 is typically a direction that intersects the movement route rt approximately perpendicularly on an approximately horizontal plane, but is not limited thereto. For example, in the case of a movement route rt that includes an inclined surface such as a slope, the side d3 may be a direction that intersects the movement route rt approximately perpendicularly on a plane that is approximately parallel to the inclined surface of the movement route rt, instead of on an approximately horizontal plane. For example, the side d3 may be a direction toward the rear d2 within a range of approximately 45 degrees from a direction approximately perpendicular to the travel route rt on a plane approximately parallel to the travel route rt having a road width. Note that approximately 45 degrees in the side d3 is just an example, and other angles may be used as long as they do not overlap with the range of the rear d2. In this second modified example, if the travel route rt is a seaside road, for example, the user can enjoy the scenery even more by directing their line of sight toward the side d3 and looking closely at the sea. User operations include operating the input interface of the HMD 104 (e.g., operating the stick of the controller) and operating the input / output I / F unit 140 of the image processing device 100, as appropriate. Other configurations are the same as those of the first embodiment. The processing unit 110 is an example of a display control unit. The HMD 104 is an example of a display.

[0071] According to the second modified example described above, the processing unit 110, in response to a user's operation, causes the HMD 104 to display an image within the field of view in which the line of sight during automatic movement has been changed from backward d2 to side d3. This not only achieves the effect of the first embodiment, but also allows the line of sight to be changed to side d3 during automatic movement of the viewpoint position p1, making it possible to further enjoy the scenery during automatic movement. Additionally, if the travel route has an open view, such as a seaside road, automatic movement in which the line of sight in the virtual space is directed to side d3 (sideways movement) can also reduce VR sickness.

[0072] <Third Modification> In the embodiment, the first modification, and the second modification, as shown in FIG. 5 , a route connecting a specified destination A and a current location is set as the travel route rt. However, this is not limited to this. For example, as shown in FIG. 10 , multiple route candidates rA, rB, rC, and rD connecting a travel start position ps in a virtual space with multiple destinations A, B, C, and D are pre-set, and the processing unit 110 sets a specified route candidate from among the multiple route candidates as the travel route. For example, when a destination A is specified by a user operation, the processing unit 110 may set the route candidate rA connecting the travel start position ps and the specified destination A as the travel route. Also, for example, when a route candidate rA is directly specified by a user operation, the processing unit 110 may set the specified route candidate rA as the travel route. Here, when the route candidate rA is directly specified, it needs to be specified in a section that does not overlap with another route candidate rB. Furthermore, when destinations A, B, C, and D are specified at the time of initial setup, the movement route calculation unit 112 calculates routes connecting the movement start position ps and each of the destinations A, B, C, and D, and sets each calculation result as route candidates rA, rB, rC, and rD. The movement start position ps may be, for example, the initial placement location at the time of login. The other configurations are the same as those in the first embodiment. Note that the processing unit 110 is an example of a setting unit.

[0073] According to the third modification example described above, multiple route candidates rA, rB, rC, and rD connecting the movement start position ps in the virtual space and multiple destinations A, B, C, and D are set in advance, and the processing unit 110 sets a specified route candidate from among the multiple route candidates as the movement route. This not only achieves the effect of the first embodiment, but also reduces the load on the movement route calculation unit 112 that calculates a route connecting an arbitrary current location and a destination.

[0074] <Other Embodiments> The functional configuration of the image processing device 100 and its peripheral devices, their processing procedures and processing contents, types and uses of data and images, etc. can be modified and implemented in various ways without departing from the spirit of the present invention.

[0075] For example, in one embodiment, VR goggles such as the HMD 104 worn by the user are used, but this is not limiting. For example, instead of the VR goggles, a flat display that is not worn by the user and is positioned facing the user may be used. Additionally, if the flat display has a large screen, the large screen occupies almost the entire field of view, making the user more susceptible to VR sickness. Taking this into consideration, a configuration may be adopted in which a flat display is used instead of VR goggles such as an HMD.

[0076] In the embodiment, the destination is specified from among the candidate destinations on the map image. However, this is not limiting. For example, an arbitrary location on the map image may be specified as the destination. In this case, in addition to the effect of the embodiment, automatic movement to the arbitrary destination is possible.

[0077] Furthermore, for example, in the second modified example, one destination is selected from candidate destinations on a map image, but this is not limiting. For example, a configuration may be adopted in which two destinations can be selected from candidate destinations on a map image. Specifically, for example, in FIG. 5 , a first destination A located along the coast and a second destination B located in the mountains can be selected. In this case, in addition to the effects of the first embodiment, it is possible to automatically travel to the second destination B by turning the line of sight to the side d3 to view the coastal scenery, passing through the first destination A, and then turning the line of sight to the rear d2. In other words, if there are multiple travel routes to the original destination B, it is possible to automatically travel to the original destination B by setting a travel route that passes through a desired location (temporary destination A).

[0078] Furthermore, for example, in one embodiment and modified example, the line of sight is directed backward d2 along the travel route rt, but the backward d2 of the travel route rt does not necessarily have to be a direction along the center line of the travel route rt. For example, if the travel route rt repeatedly meanders over a short distance, from the perspective of reducing VR sickness, the line of sight may be directed backward along a broken line that passes through the shortest path within the travel route rt, rather than along the meandering center line of the travel route rt. In this case, in addition to the effect of one embodiment, a further reduction in VR sickness can be expected.

[0079] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0080] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0081] DESCRIPTION OF SYMBOLS 100...Image processing device 102...External storage device 104...HMD 110...Processing device 111...Movement mode management unit 112...Movement route calculation unit 113...Automatic movement unit 114...Orientation adjustment unit 115...Current location memory unit 116...Manual movement unit 117...Reading unit 118...Map creation unit 119...Calculation unit 120...Rendering unit 121...Output unit 130...Program memory unit 131...Data memory unit 140...Input / output I / F unit 150...Bus A, B, C, D...Destination d1...Forward d2...Backward d3...Side rA, rB, rC, rD...Route candidate rt...Movement route

Claims

1. An image processing device comprising: a setting unit that sets a movement route when automatically moving a viewpoint position in a virtual space; and a display control unit that displays on a display an image within a field of view in which the line of sight from the viewpoint position is directed backward along the movement route while the viewpoint position is automatically moved forward along the movement route.

2. The image processing device according to claim 1, wherein the display control unit temporarily suspends or stops the automatic movement in response to a user operation.

3. The image processing device according to claim 1, wherein the display control unit causes the display to display an image within the field of view obtained by changing the line of sight direction from the rear to the side in response to a user operation.

4. The image processing device according to claim 1, wherein when a destination is specified on the map image of the virtual space, the setting unit sets a route on the map image that connects the destination and the current location as the travel route.

5. The image processing device of claim 1, wherein the setting unit pre-sets multiple route candidates connecting a movement start position in the virtual space with each of multiple destinations, and sets a specified route candidate from among the multiple route candidates as the movement route.

6. A method executed by an image processing device, comprising: setting a movement route when automatically moving a viewpoint position in a virtual space; and displaying on a display an image within the field of view in which the line of sight from the viewpoint position is directed backward along the movement route while the viewpoint position is automatically moved forward along the movement route.

7. A program for causing a computer to function as the image processing device according to any one of claims 1 to 5.

Images