Object display program, object display system, and object display method

By offsetting virtual objects in a predetermined axial direction within AR systems, the issue of flickering caused by occlusion is resolved, maintaining a stable and realistic display of virtual objects in relation to real space surfaces.

WO2024232383A1PCT designated stage expired Publication Date: 2024-11-14SQUARE ENIX EUROPE
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017072
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2024-05-08
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Augmented Reality (AR) display systems experience flickering issues when virtual objects are placed in contact with real space surfaces due to slight errors or vibrations in coordinate measurements, causing frequent changes in the context between detected surfaces and virtual objects, leading to unrealistic occlusion effects.

Method used

Implementing an offset function that shifts virtual objects in a predetermined axial direction based on their coordinates, allowing for stable display superimposition on real space without appearing to float or flicker.

Benefits of technology

This method effectively suppresses flickering and maintains a realistic display by ensuring virtual objects do not appear to float above real space surfaces, enhancing the overall AR experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017072_14112024_PF_FP_ABST
    Figure JP2024017072_14112024_PF_FP_ABST
Patent Text Reader

Abstract

[Problem] To suppress flickering originating from an occlusion function, using a simple method. [Solution] This object display program causes a computer to achieve: an offsetting function for offsetting a virtual object in a predetermined axial direction from an object in a real space, in accordance with coordinates of a viewpoint relative to the virtual object in the axial direction; and a display function for displaying the offset virtual object in an overlapping manner in the real space.
Need to check novelty before this filing date? Find Prior Art

Description

OBJECT DISPLAY PROGRAM, OBJECT DISPLAY SYSTEM, AND OBJECT DISPLAY METHOD

[0001] At least one embodiment of the present invention relates to an object display program, an object display system, and an object display method.

[0002] There are AR display systems that display content superimposed on real space. For example, Patent Literature 1 describes an AR additional information content display system. The AR additional information content display system displays spatial points, which are objects indicating predetermined coordinate points in three-dimensional space, on a display screen of a terminal device on which an AR marker is displayed. The AR additional information content display system includes a spatial point generation unit that generates a group of spatial points on the display screen based on the AR marker, and a spatial point and two-dimensional information link unit that links the specified spatial points with two-dimensional information displayed in two dimensions on the display screen and makes the spatial points linked to the two-dimensional information appear different from spatial points not linked to the two-dimensional information.

[0003] Japanese Patent Application Laid-Open No. 2021-144407

[0004] There is an occlusion function when displaying AR. Occlusion refers to a state in which an object in the foreground hides an object behind it. By using the occlusion function, the display system can grasp the front-to-back relationship of objects, making it possible to display AR in a state in which an object in front hides an object in the background, resulting in a more realistic AR display.

[0005] However, when an AR display system places a virtual object so that it is in contact with a surface such as a detection surface detected from real space, flickering due to the occlusion function can occur. This occurs because the coordinates of the object detected from real space and the coordinates of the virtual object match, and if there is a slight error or fluctuation in the measurement, the front-to-back relationship between the surface such as the detection surface detected from real space and the virtual object frequently changes. There was a latent need for a method to suppress this flickering due to the occlusion function as simply as possible.

[0006] An object of at least one embodiment of the present invention is to solve the above problem and to suppress flickering caused by the occlusion function in a simple manner.

[0007] From a non-limiting perspective, an object display program according to one embodiment of the present invention causes a computer to realize an offset function that offsets a virtual object from an object in real space in a predetermined axial direction according to a coordinate in the axial direction of a viewpoint, the coordinate being based on the virtual object, and a display function that displays the offset virtual object superimposed on real space.

[0008] From a non-limiting perspective, an object display system according to one embodiment of the present invention is an object display system equipped with at least a user terminal, and includes: offset means for offsetting a virtual object from an object in real space in a predetermined axial direction in accordance with a coordinate of a viewpoint in the axial direction, the coordinate being based on the virtual object; and display means for displaying the offset virtual object superimposed on real space.

[0009] From a non-limiting perspective, an object display method according to one embodiment of the present invention is a computer-based object display method, which includes: an offset process for offsetting a virtual object from an object in real space in a predetermined axial direction in accordance with a coordinate in the axial direction of a viewpoint, the coordinate being based on the virtual object; and a display process for displaying the offset virtual object superimposed on real space.

[0010] Each embodiment of the present application addresses one or more of the deficiencies.

[0011] FIG. 1 is a block diagram showing an example of the configuration of an object display system corresponding to at least one of the embodiments of the present invention. FIG. 2 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. FIG. 3 is a flowchart showing an example of processing of an object display program corresponding to at least one of the embodiments of the present invention. FIG. 4 is a diagram showing an example of display by a display device. FIG. 5 is a conceptual diagram illustrating offset processing by an offset unit.

[0012] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that the various components in the examples of the embodiments described below can be combined as appropriate to the extent that no inconsistencies or the like arise. Furthermore, content described as an example of one embodiment may be omitted in other embodiments. Furthermore, the content of operations and processes unrelated to the characteristic parts of each embodiment may be omitted. Furthermore, the order of various processes constituting the various flows and sequences described below may be in any order to the extent that no inconsistencies or the like arise in the processing content.

[0013] [Embodiment] An outline of an embodiment of the present invention will be described below. In the embodiment, an object display program executed on a server, which is an example of a computer, will be described as an example.

[0014] 1 is a block diagram showing an example of the configuration of an object display system 100 corresponding to at least one embodiment of the present invention. The object display system 100, which is an example of an object display system, includes a server 10 and a user terminal 20 used by a user. Each of the user terminals 20A, 20B, and 20C is an example of the user terminal 20. The configuration of the object display system 100 is not limited to this. For example, the object display system 100 may be configured so that multiple users use a single user terminal. The object display system 100 may also include multiple servers.

[0015] The server 10 and the user terminal 20 are examples of computers. The server 10 and the user terminal 20 are each communicatively connected to a communication network 30 such as the Internet. The connection between the communication network 30 and the server 10 and the connection between the communication network 30 and the user terminal 20 may be a wired connection or a wireless connection. For example, the user terminal 20 may connect to the communication network 30 by performing data communication with a base station managed by a telecommunications carrier via a wireless communication line.

[0016] The object display system 100 includes a server 10 and a user terminal 20, thereby realizing various functions for executing various processes in response to user operations.

[0017] The object display system 100 may also function as a game processing system. The game processed by the game processing system may be a video game, or may be a game that utilizes real space, with or without XR technology. When the object display system 100 also functions as a game processing system, the server 10 controls the progress of the game. The server 10 may control the progress of the game in accordance with the player's operations. The server 10 is managed by an administrator of the object display system 100, and has various functions for providing information regarding various processes to multiple user terminals 20. The server 10 may be configured by an information processing device such as a game server capable of rendering game images. Note that the object display system and the game processing system may be configured by multiple servers.

[0018] The server 10 includes a processor 11, a memory 12, and a storage device 13. The processor 11 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various calculations and control. If the server 10 includes a GPU (Graphics Processing Unit), some of the calculations and control may be performed by the GPU. The server 10 uses the data read into the memory 12 to execute various information processing operations using the processor 11, and stores the obtained processing results in the storage device 13 as necessary.

[0019] The storage device 13 functions as a storage medium for storing various types of information. There are no particular limitations on the configuration of the storage device 13, but from the viewpoint of reducing the processing load on the user terminal 20, it is preferable that the storage device 13 be configured to be able to store all of the various types of information necessary for the control performed by the object display system 100. Examples of such a storage device include an HDD and an SSD. However, the storage device that stores the various types of information only needs to have a storage area accessible by the server 10, and may be configured to have a dedicated storage area outside the server 10, for example.

[0020] The user terminal 20 is connected to the communication network 30 and includes hardware (for example, a display device that displays a browser screen or a game screen according to coordinates) and software for executing various processes by communicating with the server 10. Note that the multiple user terminals 20 may be configured to be able to communicate directly with each other without going through the server 10.

[0021] The user terminal 20 may be configured as a communication terminal managed by a user and capable of playing a network distribution type game. Examples of communication terminals include a mobile phone terminal, a PDA (Personal Digital Assistant), a portable game device, VR goggles, AR glasses, smart glasses, AR contacts, and so-called wearable devices. The configuration of the user terminal that can be included in the object display system 100 is not limited to these. Other examples of the configuration of the user terminal include a combination of various communication terminals, a personal computer, and a stationary game device.

[0022] A display device may be built into the user terminal 20. The display device may be connected wirelessly or by wire to the user terminal 20. Note that the display device is an extremely common configuration, and is therefore not shown here.

[0023] When a user terminal is used for gaming, the game screen is displayed by a display device as, for example, a composite image, and the user perceives this composite image. The game screen is displayed, for example, on a display, which is an example of a display device provided in the user terminal, or on a display, which is an example of a display device connected to the user terminal. Examples of display devices include a holographic display device capable of displaying holograms, a projection device that projects images (including game screens) onto a screen, and a device that performs XR display. XR includes virtual reality (VR), augmented reality (AR), mixed reality (MR), and substitutional reality (SR).

[0024] The user terminal 20 includes a processor 21, a memory 22, and a storage device 23. The processor 21 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various calculations and control. Furthermore, if the user terminal 20 includes a GPU (Graphics Processing Unit), some of the various calculations and control may be performed by the GPU. The user terminal 20 uses the data read into the memory 22 to execute various information processing operations with the processor 21, and stores the obtained processing results in the storage device 23 as necessary. The storage device 23 functions as a storage medium for storing various information.

[0025] The user terminal 20 may have a built-in input device. The input device may also be connected to the user terminal 20 wirelessly or via a wire. The input device accepts operational input from the user. In response to the operational input from the user, a processor provided in the server 10 or a processor provided in the user terminal 20 executes various control processes. Examples of input devices include a touch panel screen provided in a mobile phone terminal, a keyboard, a mouse, a game pad, a joystick, and other controllers. A camera built into or connected to the user terminal 20 may also correspond to an input device. The user performs operational input by making gestures such as moving their hand in front of the camera (gesture input).

[0026] Additionally, the user terminal 20 may be equipped with other output devices such as a speaker, etc. The other output devices output sounds, vibrations, and various other types of information to the user.

[0027] 2 is a block diagram showing the configuration of a server corresponding to at least one embodiment of the present invention. Server 10A, which is an example of the configuration of server 10, includes at least an offset unit 101 and a display unit 102. A processor included in server 10A references an object display program stored in a storage device and executes the program to functionally realize the offset unit 101 and the display unit 102.

[0028] The offset unit 101 has a function of offsetting a virtual object from an object in real space in an axial direction in accordance with a coordinate in a predetermined axial direction of a viewpoint, with the virtual object as a reference.

[0029] The display unit 102 has a function of displaying the offset virtual object superimposed on the real space.

[0030] Next, a program execution process according to an embodiment of the present invention will be described below: Fig. 3 is a flowchart showing an example of processing of an object display program corresponding to at least one of the embodiments of the present invention.

[0031] The offset unit 101 offsets the virtual object from the object in real space in an axial direction according to a coordinate in a predetermined axial direction of the viewpoint, with the virtual object as a reference (St11). The display unit 102 displays the offset virtual object superimposed on the real space (St12).

[0032] A virtual object is an imaginary object that is to be displayed overlaid on real space. Examples of virtual objects include a player character, an enemy character, an item such as a sword, and a vehicle such as a tank. However, virtual objects are not limited to these.

[0033] The viewpoint refers to the position of a virtual camera or the viewpoint of a user acquired by a user terminal such as VR goggles, AR glasses, smart glasses, or AR contacts.

[0034] The predetermined axis direction may typically be the upward direction in a predetermined space. For example, the Y axis direction in a Y-up coordinate space or the Z axis direction in a Z-up coordinate space is the predetermined axis direction. However, a direction other than the upward direction may also be the predetermined axis direction.

[0035] The coordinates of a predetermined axis direction of the viewpoint, with a virtual object as the reference, refer to the coordinates of the viewpoint when a predetermined position of the virtual object is assumed to have a predetermined coordinate value (for example, zero coordinate, which is the origin). For example, if the virtual object is a humanoid figure, the coordinates of the figure's feet are assumed to have a coordinate value of 0. The coordinates of the viewpoint in this case correspond to the coordinates of the viewpoint with the virtual object as the reference here. The predetermined position of the object may be determined as appropriate by the system designer or the like.

[0036] The object in the real space means an object in the real space such as a desk, a chair, or a floor.

[0037] To offset in the axial direction means to shift the display position in the axial direction.

[0038] [Specific Example of Offset] Fig. 4 is a diagram showing an example of display by a display device. In this example, a smartphone is used as the user terminal and display device. Based on an image captured by the user terminal's built-in camera, the object display system 100 detects a shelf and a mug as objects in real space. The object display system 100 recognizes the horizontal plane of the shelf as a detection plane PL and displays a virtual object OBJ on the detection plane PL using AR. AR display of a virtual object refers to displaying a virtual object superimposed on real space.

[0039] In this example, the virtual object OBJ is a figure. When the occlusion function is used, flickering may occur at the feet of the figure. This is because the figure's feet are in contact with the detection plane PL of the object in real space, and the position of the figure's feet or the detection plane PL frequently changes.

[0040] FIG. 5 is a conceptual diagram illustrating an example of offset processing by the offset unit.

[0041] The object display system 100 detects an object in real space and AR-displays a virtual object OBJ on the detected surface of the object in real space.

[0042] 5, reference symbol V denotes a viewpoint of a virtual camera, etc. As the viewpoint positions, P1 and P2 are shown as examples.

[0043] 5 illustrates an example of a Y-up coordinate space. In this case, coordinates C1 and C2 are coordinates in the Y-axis direction with respect to the viewpoint V, based on the feet of the virtual object OBJ, i.e., the part that contacts the detection plane PL.

[0044] In the object display system 100 of the present disclosure, the offset unit 101 offsets the virtual object OBJ in an axial direction from the object in real space according to the coordinate of a predetermined axial direction (the Y-axis direction in Figure 5) of the viewpoint, based on the virtual object OBJ.

[0045] For example, when viewpoint V is at position P1, the virtual object OBJ is viewed from above, which makes it easy for flickering due to the occlusion function to occur. To prevent this flickering, the offset unit 101 offsets the virtual object OBJ in the axial direction. In other words, the offset unit 101 slightly lifts the virtual object OBJ from the detection plane PL. The offset amount may be, for example, approximately 1.5 cm when converted into a length in real space. However, the offset amount may be a value other than 1.5 cm and may be set appropriately depending on the environment.

[0046] When the viewpoint V is at position P2, the virtual object OBJ is viewed from the side, so flickering caused by the occlusion function hardly occurs. Therefore, the offset unit 101 does not offset the virtual object OBJ in the axial direction. In other words, the offset unit 101 grounds the virtual object OBJ so that it does not float above the detection plane PL. Note that, as an internal process, the offset process may be performed with the offset amount set to 0.

[0047] [Example of Processing for Offset Amount] The offset amount may be processed as a binary value of 0 or a predetermined value. The offset amount may also be processed so as to take on more than two values.

[0048] For example, the offset unit 101 may decrease the offset amount as the absolute value of the coordinate in the predetermined axis direction of the viewpoint decreases. In other words, if the predetermined axis direction is upward, the offset amount may increase as the viewpoint moves to a position where the virtual object OBJ is viewed from further above, and the offset amount may decrease as the viewpoint moves to a position where the virtual object OBJ is viewed from further to the side. More specifically, the offset unit 101 may set the offset amount to zero when the absolute value of the coordinate in the predetermined axis direction of the viewpoint is smaller than a predetermined value. This prevents the object OBJ from appearing to float above the detection plane PL.

[0049] The offset unit 101 may set the offset amount in stages based on one or more threshold values, for example, if the coordinate value is b or greater, the offset amount = Y1; if the coordinate value is a or greater but less than b, the offset amount = Y2; if the axis coordinate value is less than a, the offset amount = 0.

[0050] The offset unit 101 may also set the offset amount using a function that receives a value based on the coordinate value as input and outputs a value from which the offset amount can be derived.

[0051] An upper limit may be set for the offset amount.

[0052] According to one aspect of the embodiment, flickering caused by the occlusion function can be suppressed by a simple method.

[0053] According to one aspect of the embodiment, it is possible to prevent flickering and display a virtual object so that the virtual object does not appear to float above the detection plane of the object in real space.

[0054] According to one aspect of the embodiment, when a virtual object is placed on an object in real space, flickering caused by an occlusion function can be suppressed by a simple method.

[0055] As described above, each embodiment of the present application solves one or more deficiencies. Note that the effects of each embodiment are non-limiting effects or examples of effects.

[0056] In each of the above-described embodiments, the user terminal 20 and the server 10 execute the above-described various processes in accordance with various control programs (e.g., object display programs) stored in their own storage devices. Also, other computers, not limited to the user terminal 20 and the server 10, may execute the above-described various processes in accordance with various control programs (e.g., object display programs) stored in their own storage devices.

[0057] Furthermore, the configuration of object display system 100 is not limited to the configurations described as examples of each of the above-mentioned embodiments. For example, a configuration may be adopted in which a server executes some or all of the processes described as processes executed by a user terminal, or a configuration may be adopted in which a user terminal executes some or all of the processes described as processes executed by a server. Furthermore, a configuration may be adopted in which the user terminal includes some or all of the storage unit (storage device) included in the server. In other words, a configuration may be adopted in which the user terminal or the server in object display system 100 includes some or all of the functions included in the other.

[0058] Furthermore, the program may be configured to cause a part or all of the functions described as examples of each of the above-mentioned embodiments to be realized by a single device that does not include a communication network.

[0059] [Notes] The above-described embodiments have been described so as to enable at least the following inventions to be implemented by a person having ordinary skill in the art to which the invention pertains. [1] An object display program that causes a computer to realize: an offset function that offsets a virtual object from an object in real space in a predetermined axial direction relative to a viewpoint, the virtual object being used as a reference, according to the coordinate of the predetermined axial direction; and a display function that displays the offset virtual object superimposed on real space. [2] The object display program according to [1], in which the offset function reduces the offset amount as the absolute value of the coordinate decreases. [3] The object display program according to [2], in which the offset function sets the offset amount to zero when the absolute value is smaller than a predetermined value. [4] The object display program according to any one of [1] to [3], in which the predetermined axial direction is an upward direction in a predetermined space. [5] An object display system having at least a user terminal, comprising: an offset means for offsetting a virtual object from an object in real space in a predetermined axial direction according to a coordinate of a viewpoint with the virtual object as a reference, the coordinate being the coordinate of the virtual object, and a display means for displaying the offset virtual object superimposed on real space. [6] An object display method implemented by a computer, comprising: an offset process for offsetting the virtual object from an object in real space in the axial direction according to a coordinate of a viewpoint with the virtual object as a reference, the coordinate being the coordinate of the virtual object, and a display process for displaying the offset virtual object superimposed on real space.

[0060] According to one embodiment of the present invention, an object display program and an object display system are useful that can suppress flickering caused by an occlusion function in a simple manner.

[0061] 10, 10A Server 11 Processor 12 Memory 13 Storage device 20, 20A, 20B User terminal 21 Processor 22 Memory 23 Storage device 30 Communication network 100 Object display system 101 Offset unit 102 Display unit

Claims

1. An object display program that causes a computer to realize: an offset function that offsets a virtual object from an object in real space in a predetermined axial direction according to coordinates of a viewpoint in the predetermined axial direction, with the virtual object as a reference; and a display function that displays the offset virtual object superimposed on the real space.

2. The object display program according to claim 1, wherein in the offset function, the smaller the absolute value of the coordinates, the smaller the offset amount.

3. The object display program according to claim 2, wherein the offset function sets the offset amount to zero when the absolute value is smaller than a predetermined value.

4. The object display program according to any one of claims 1 to 3, wherein the predetermined axis direction is an upward direction in a predetermined space.

5. An object display system having at least a user terminal, comprising: an offsetting means for offsetting a virtual object from an object in real space in a predetermined axial direction in accordance with a coordinate in the predetermined axial direction of a viewpoint, the coordinate being based on the virtual object; and a displaying means for displaying the offset virtual object superimposed on the real space.

6. A method for displaying an object by a computer, comprising: an offset process for offsetting a virtual object from an object in real space in a predetermined axial direction according to coordinates in the axial direction of a viewpoint with respect to the virtual object as a reference; and a display process for displaying the offset virtual object superimposed on the real space.

Citation Information

Patent Citations

  • Image generation system and information storage medium

    JP2001084398A