Display Control System

Abstract

A server 10 according to one embodiment of the present invention is provided with: a file storage unit 13 that stores, for a first virtual object, a first file having a first file size and a second file having a second file size larger than the first file size; and a display control unit 14 that controls display of the first virtual object. The display control unit 14 executes: a first process for displaying the first file on a display unit 24 by preferentially transmitting the first file to a user terminal 20; a second process for transmitting the second file to the user terminal 20 after the transmission of the first file or in parallel with the transmission of the first file; and a third process for, when the first file is being displayed on the display unit 24 and the transmission of the second file is completed, switching the first file being displayed on the display unit 24 to the second file.

Description

[Technical Field]

[0001] One aspect of the present invention relates to a display control system that controls the display of a virtual object. [Background technology]

[0002] There are known technologies (hereinafter referred to as "XR technologies") that present virtual objects to users, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR). For example, Patent Document 1 discloses a mechanism for providing augmented reality or mixed reality to a user by superimposing a virtual object on a real-space image displayed on a display device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-203382 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned XR technology, in order to enhance the comfort of the user's virtual experience, it is necessary to display virtual objects as quickly as possible. For example, one system that applies the above-mentioned XR technology is a system that downloads and displays virtual objects, which are 3D objects generated by scanning objects existing in real space, to a user's device. However, such virtual objects based on real objects generally have large file sizes, which can take a long time to download and can result in a long time until the virtual object is displayed on the display unit of the user's device. On the other hand, if the file size of the virtual object is reduced by methods such as file reduction to shorten the download time, another problem can arise: the virtual object cannot be delivered to the user at its original quality (resolution), thereby degrading the quality of the user's virtual experience.

[0005] Therefore, one aspect of the present invention aims to provide a display control system that can shorten the time it takes for a virtual object to be displayed on a user terminal while maintaining the quality of the user's virtual experience. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a display control system for controlling the display of virtual objects displayed on a display unit of a user terminal, the display control system including: a file storage unit configured to store, for a first virtual object, a first file having a first file size and a second file having a second file size larger than the first file size; and a display control unit configured to control the display of the first virtual object on the user terminal. The display control unit performs the following operations: a first process for transmitting the first file to the user terminal with priority over the second file, thereby displaying the first virtual object based on the first file on the display unit; a second process for transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file; and a third process for switching the first virtual object based on the first file displayed on the display unit to a first virtual object based on the second file when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal has been completed.

[0007] In a display control system according to one aspect of the present disclosure, a first file having a small file size (first file size) is preferentially transmitted to a user terminal. This configuration shortens the time until a first virtual object is displayed on a display unit of the user terminal. After the first file is transmitted, or in parallel with the transmission of the first file, a second file is transmitted to the user terminal. After the user terminal receives the second file, the first virtual object based on the first file displayed on the display unit is replaced with a first virtual object based on the second file. This configuration allows the first virtual object based on the second file, which has a larger file size and higher quality than the first file, to be presented to the user later, thereby maintaining the quality of the user's virtual experience. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to provide a display control system that can maintain the quality of the user's virtual experience while shortening the time it takes for a virtual object to be displayed on a user terminal. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a functional configuration of a system including a server and a user terminal, which is a display control system according to an embodiment. [Figure 2] FIG. 10 illustrates an example of a file generation process. [Figure 3] FIG. 10 is a diagram illustrating an example of display control of a virtual object. [Figure 4] FIG. 10 is a sequence diagram illustrating an example of the operation of the server and the user terminal. [Figure 5] FIG. 2 illustrates an example of the hardware configuration of a server and a user terminal. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and redundant description will be omitted.

[0011] FIG. 1 is a diagram illustrating an example of a system including a server 10, which is a display control system according to an embodiment, and a user terminal 20. The server 10 is a device that controls the display of virtual objects displayed on the display unit 24 of the user terminal 20. For example, the server 10 may provide the user with an AR or MR experience by displaying virtual objects together with a real-space image on the display unit 24 of the user terminal 20. Alternatively, the server 10 may provide the user with a VR experience by displaying virtual objects together with a virtual background (virtual space image) on the display unit 24 of the user terminal 20. The user terminal 20 is a computer terminal used by a user. The type of the user terminal 20 is not limited. For example, the user terminal 20 may be a personal computer such as a desktop PC or a laptop PC, a terminal device such as a smartphone or tablet held by the user, or a wearable device attached to a part of the user's body, such as the head. Examples of wearable devices include head-mounted displays such as goggles, glasses (eyeglasses), and hats.

[0012] The server 10 and the user terminal 20 are connected to each other via a wired or wireless communication network so as to be able to communicate data with each other. The server 10 and the user terminal 20 may be connected via one or more other devices. For example, if the user terminal 20 is a head-mounted display, each of the server 10 and the user terminal 20 may be configured to be able to communicate with another device such as a user's smartphone. In this case, the server 10 is configured to be able to communicate with the user terminal 20 via the other device.

[0013] 1, the server 10 (display control system) includes a receiving unit 11 (terminal information acquiring unit), a file generating unit 12, a file storage unit 13, and a display control unit 14. The user terminal 20 includes an imaging unit 21, a transmitting unit 22, a receiving unit 23, and a display unit 24.

[0014] First, each functional configuration of the user terminal 20 will be described. The imaging unit 21 has a function of capturing an image of the real space around (for example, in front of) the user terminal 20 to acquire a real space image (captured data). The imaging unit 21 is configured by, for example, a camera. The real space image may be a still image or a moving image (video). When an AR experience or an MR experience is provided to the user, the real space image may be displayed on the display unit 24. Furthermore, in order to generate a virtual object, which is a 3D object, by the file generation unit 12 described later, the captured data may be composed of a plurality of image data sets obtained by capturing images of an object (real object) existing in real space from a plurality of different directions.

[0015] The transmission unit 22 transmits the imaging data acquired by the imaging unit 21 to the server 10. In this embodiment, the transmission unit 22 also transmits terminal information relating to the attributes of the user terminal 20 to the server 10. Examples of the attributes of the user terminal 20 include application type, communication speed, processing performance, etc.

[0016] The "application type" is information indicating the type of application used to display a virtual object transmitted from the server 10. The application type can be identified, for example, by a combination of the type of user terminal 20 and the type of application running on the user terminal 20. Examples of application types include a dedicated application running on a PC, a web browser running on a PC, a web browser running on a smartphone, and an application built into a head-mounted display such as smart glasses. The transmission unit 22 transmits terminal information indicating the application type to the server 10, for example, when starting execution of a service that displays a virtual object transmitted from the server 10 (for example, during a login operation).

[0017] The "communication speed" is the downstream speed (download speed) when downloading data from the server 10 to the user terminal 20. For example, when the user terminal 20 is a terminal (e.g., a head-mounted display, a smartphone, etc.) that moves with the user and the server 10 and the user terminal 20 communicate wirelessly, the communication speed may change because the strength of the wireless radio waves varies depending on the location of the user terminal 20. The transmission unit 22, for example, periodically transmits terminal information indicating the communication speed to the server 10.

[0018] "Processing performance" is information indicating the specifications of the processor, memory, etc. provided in the user terminal 20, the processor usage rate, the memory usage rate, etc. For example, the terminal information indicating the processing performance at a certain point in time is information indicating the amount of computing resources (processor, memory, etc.) available on the user terminal 20 at that point in time. The transmission unit 22, for example, periodically transmits the terminal information indicating the processing performance to the server 10.

[0019] The receiving unit 23 receives information for displaying a virtual object on the display unit 24 from the server 10 (a first file, a second file, etc., which will be described later).

[0020] The display unit 24 displays a virtual object in a display area 24a provided in the display unit 24, based on an instruction from the display control unit 14 of the server 10. Specifically, the display unit 24 displays a virtual object in the display area 24a, based on information for displaying the virtual object (first file, second file, etc.) received from the server 10. If the user terminal 20 is a PC, the display unit 24 may be configured by a display monitor. If the user terminal 20 is a mobile terminal such as a smartphone, the display unit 24 may be configured by a touch panel display. If the user terminal 20 is a head-mounted display, the display unit 24 may be configured by a display placed in front of both eyes of the user.

[0021] Next, a description will be given of each functional configuration of the server 10. The receiving unit 11 receives various data transmitted by the user terminal 20 (transmitting unit 22). In this embodiment, the receiving unit 11 receives imaging data acquired by the imaging unit 21. The receiving unit 11 also functions as a terminal information acquiring unit that acquires the above-mentioned terminal information (application type, communication speed, processing performance, etc.).

[0022] The file generation unit 12 acquires the imaging data received by the receiving unit 11. If the imaging data includes a real object that exists in real space, the file generation unit 12 generates a first file and a second file of a virtual object corresponding to the real object based on the imaging data. The file generation unit 12 generates a file representing a virtual object corresponding to the real object included as a subject in the imaging data by, for example, performing at least one of image recognition (2D image recognition), photogrammetry, and 3D scanning on the imaging data. Note that the method for generating a virtual object by the file generation unit 12 is not limited to the generation method based on imaging data of a real object as described above. For example, the file generation unit 12 may generate a file representing a virtual object by modeling or the like. In other words, the virtual object used in this embodiment is not limited to an object based on a real object, but may also be an object created by a creator such as a CG designer.

[0023] An example of file generation processing by file generation unit 12 will be described with reference to FIG. 2. FIG. 2 shows an example in which a rabbit kept as a pet by a user is captured as a real object on user terminal 20 (imaging unit 21). In this example, file generation unit 12 generates three types of files 50A, 50B, and 50C as data representing a virtual object (first virtual object) corresponding to this real object. Files 50A, 50B, and 50C are all data used to display a virtual object on display unit 24 of user terminal 20, and may include drawing information for drawing the virtual object in display area 24a. In this embodiment, the virtual object is represented as a 3D object.

[0024] File 50A is data that has not been compressed (reduced) and that retains the original quality (resolution). That is, file 50A is an original file generated based on imaging data of a real object. File 50B is data obtained by compressing file 50A by 30%. That is, the file size (file capacity) of file 50B is 70% of the file size of file 50A. File 50C is data obtained by compressing file 50A by 60%. That is, the file size of file 50C is 40% of the file size of file 50A.

[0025] Due to the data compression, files 50B and 50C have lower quality than file 50A. As an example, as shown in FIG. 2, the surface shape of the virtual object (rabbit) represented by file 50B is coarser than the surface shape of the virtual object represented by original file 50A. Furthermore, the surface shape of the virtual object represented by file 50C is even coarser than the surface shape of the virtual object represented by file 50B. In other words, files 50B and 50C have lower quality (resolution) than original file 50A by the amount that their file sizes are smaller than the original file size.

[0026] In this embodiment, as an example, the file generation unit 12 generates two types of compressed files (files 50B and 50C). However, the file generation unit 12 may generate only one type of compressed file (one specific compression ratio), or may generate three or more types of compressed files (three or more different compression ratios). For example, the file generation unit 12 may determine the compression ratio of the compressed file to be generated depending on the file size of the original file 50A. For example, if the file size of the original file 50A is not very large, the file size can be sufficiently small without increasing the compression ratio very much. In this case, the file generation unit 12 may generate only the file 50B.

[0027] The file storage unit 13 acquires and stores, for a first virtual object that is one virtual object, a first file having a first file size and a second file having a second file size larger than the first file size.

[0028] In the present embodiment, as an example, the first virtual object is a virtual object corresponding to the real object (rabbit) shown in FIG. 2. That is, in the present embodiment, the first virtual object is a virtual object generated based on imaging data of the real object. In the example of FIG. 2, file 50C corresponds to a first file having a first file size. Furthermore, each of files 50A and 50B corresponds to a second file having a second file size larger than the first file size. That is, in the example of FIG. 2, file storage unit 13 stores a plurality of (here, two) second files (files 50A and 50B) having mutually different file sizes. Note that, when file generation unit 12 generates only files 50A and 50B, file 50B functions as the first file and file 50A functions as the second file.

[0029] In this embodiment, the file storage unit 13 stores the files 50A and 50B (second files) and the file 50C (first file) generated by the file generation unit 12 in association with the virtual objects corresponding to the real objects. That is, the file storage unit 13 stores (registers) the first file and the second file corresponding to each virtual object.

[0030] The file storage unit 13 manages multiple virtual objects (i.e., multiple virtual objects each corresponding to a different real object) using, for example, unique identification information (object IDs). For example, when virtual objects are operated on the web (i.e., when displayed by a web browser or the like on the user terminal 20), the file storage unit 13 may assign an identifier such as a unique URI (Uniform Resource Identifier) ​​to a file representing each virtual object.

[0031] Furthermore, the first file and the second file (three files 50A, 50B, and 50C in this embodiment) for the same virtual object can be managed as follows.

[0032] In the first example, files 50A, 50B, and 50C are managed by file names. For example, file storage unit 13 manages (stores) file 50A (original file) as “usagi.3d,” file 50B (file compressed by 30%) as “usagi_a.3d,” and file 50C (file compressed by 60%) as “usagi_b.3d.” In this example, “usagi” is character string information that uniquely identifies a virtual object (here, as an example, a virtual object corresponding to the real object (rabbit) shown in FIG. 2). “.3d” is an example of an identifier assigned to each file (data). “_a” and “_b” added after “usagi” are information corresponding to the compression rate with respect to the original file. According to the first example, it is possible to grasp the virtual object and the compression rate from the file name.

[0033] In the second example, files 50A, 50B, and 50C are managed by embedding information about the compression ratio (the compression ratio of the original file is 0) in the header of each file, etc. In this case, file storage unit 13 may assign a common file name (for example, "usagi.3d") to files 50A, 50B, and 50C by storing files 50A, 50B, and 50C in different locations.

[0034] In a third example, multiple files 50A, 50B, and 50C corresponding to the same virtual object are managed by a management table. For example, multiple files 50A, 50B, and 50C corresponding to the same virtual object are associated with each other in the management table by a common key that indicates the virtual object.

[0035] In any of the above first to third example methods, when it becomes necessary to transmit data (files) of a certain virtual object to the user terminal 20 (for example, when a request to display a virtual object is received from the user terminal 20), it becomes possible to access the first file and the second file corresponding to the virtual object from among the multiple files stored in the file storage unit 13. Note that the file management method is not limited to the above first to third examples, and management may be performed by methods other than these.

[0036] The display control unit 14 controls the display of a first virtual object on the user terminal 20. For example, when it becomes necessary to display a first virtual object on the display unit 24 of the user terminal 20, the display control unit 14 performs the above-described display control by controlling a process of providing files (first file, second file) corresponding to the first virtual object to the user terminal 20. Specifically, the display control unit 14 performs a first process, a second process, and a third process described below. An example of the process performed by the display control unit 14 will be described below with reference to FIG. 3.

[0037] The first process is a process of transmitting the first file (file 50C in the example of FIG. 2) to the user terminal 20 with priority over the second file (files 50A and 50B in the example of FIG. 2), thereby displaying a first virtual object based on the first file on the display unit 24. That is, the display control unit 14 provides the user terminal 20 with priority over file 50C, which has a reduced file size compared to the original file. According to the above configuration, as shown in the upper part of FIG. 3, the time from when a download request is issued from the user terminal 20 to the server 10 to when the first virtual object based on file 50C is displayed on the display unit 24 (display area 24a) of the user terminal 20 can be shortened compared to when the original file (file 50A) is transmitted to the user terminal 20.

[0038] The second process is a process of transmitting a second file (file 50A or file 50B) to the user terminal 20 after transmitting a first file (file 50C) to the user terminal 20, or in parallel with transmitting the first file to the user terminal 20. In the second process, the display control unit 14 may execute either a first control pattern in which transmission of the second file to the user terminal 20 is started after transmitting the first file, or a second control pattern in which transmission of the second file to the user terminal 20 is performed in parallel with transmitting the first file to the user terminal 20.

[0039] Because the file size of the second file is larger than the file size of the first file, the transmission process of the first file also completes before the transmission process of the second file in the second control pattern. However, because the transmission process of the second file is executed in parallel with the transmission process of the first file in the second control pattern, the time when the transmission of the second file in the second control pattern is completed is earlier than the time when the transmission of the second file is completed in the first control pattern. In other words, the second control pattern is configured to start the transmission process of the second file before the transmission of the first file is completed, thereby enabling the transmission of the second file to be completed earlier than in the first control pattern, which starts the transmission process of the second file after the transmission of the first file is completed. On the other hand, the time when the transmission of the first file in the second control pattern is completed is later than the time when the transmission of the first file is completed in the first control pattern. In other words, while the second control pattern allows the second file to be sent more quickly than the first control pattern, the time required to send the first file is longer than with the first control pattern because the second file is also sent while the first file is being sent (i.e., the communication resources are shared between the first file and the second file).

[0040] Therefore, from the perspective of shortening the time until the first virtual object is displayed on the display unit 24 of the user terminal 20 as much as possible, the first control pattern, which downloads the first file and the second file in sequence, is more advantageous than the second control pattern. On the other hand, from the perspective of shortening the waiting time until the first virtual object is displayed (the waiting time experienced by the user) and providing the user with high-quality data (the second file) as soon as possible, the second control pattern, which downloads the first file and the second file in parallel, is more advantageous than the first control pattern.

[0041] Therefore, the display control unit 14 may be configured to be able to switch between the first control pattern and the second control pattern. For example, the display control unit 14 may determine whether to execute the first control pattern or the second control pattern based on a user setting. That is, the display control unit 14 may allow the user to select whether to execute the first control pattern or the second control pattern. Alternatively, the display control unit 14 may determine whether to execute the first control pattern or the second control pattern based on the terminal information (e.g., communication speed). For example, the display control unit 14 may determine to execute the second control pattern when it is expected that the download of the first file will be completed in a sufficiently short time even if the first file and the second file are downloaded in parallel (e.g., when the communication speed is equal to or higher than a predetermined threshold). Note that even when the user terminal 20 has a configuration for performing parallel processing at high speed (e.g., multithreading) or a configuration supporting divided downloads, it is expected that the download of the first file will be completed in a sufficiently short time. Therefore, the display control unit 14 may determine whether to execute the first control pattern or the second control pattern depending on the communication method implemented in the user terminal 20. According to the above configuration, it is possible to appropriately switch between the first control pattern and the second control pattern described above depending on the use case.

[0042] Furthermore, in the second process, the display control unit 14 may determine, based on the terminal information, a second file to be transmitted to the user terminal 20 from among a plurality of second files (files 50A and 50B in this embodiment). For example, the display control unit 14 may determine the second file to be provided to the user terminal 20 according to the attributes of the user terminal 20 described above (application type, communication speed, processing performance, etc.).

[0043] For example, if an application running on the user terminal 20 cannot smoothly display the file 50A having the original quality (i.e., if the file 50A has a quality that exceeds the range that can be processed by the application), or if the user terminal 20 does not have the processing performance (e.g., specifications of the processor, memory, etc.) required to perform display processing using the file 50A, then even if the file 50A is sent to the user terminal 20, the user terminal 20 cannot perform display processing that makes the most of the quality of the file 50A. Therefore, the display control unit 14 may determine, based on the terminal information including information regarding at least one of the application type and processing performance of the user terminal 20, as the second file to be provided to the user terminal 20, a second file having a quality that does not exceed the maximum quality at which display processing can be performed on the user terminal 20.

[0044] For example, in the example of Figure 2, if file 50A has a quality higher than the maximum quality at which display processing can be performed on user terminal 20, and file 50B has a quality equal to or lower than the maximum quality, display control unit 14 may select file 50B as the second file to be sent to user terminal 20.

[0045] Furthermore, even if the user terminal 20 has sufficient specifications (e.g., processor, memory, and other performance) to perform display processing using the file 50A, if the communication speed of the user terminal 20 is slow, it will take a very long time to complete the download of the file 50A. Therefore, the display control unit 14 may determine, for example, based on terminal information including information about the communication speed of the user terminal 20, a second file whose download is expected to be completed within a predetermined threshold time as the second file to be provided to the user terminal 20. For example, in the example of FIG. 2, if the estimated download time of the file 50A estimated from the communication speed of the user terminal 20 exceeds the threshold time and the estimated download time of the file 50B is equal to or shorter than the threshold time, the display control unit 14 may select the file 50B as the second file to be transmitted to the user terminal 20.

[0046] According to the above configuration, when a plurality of second files (files 50A and 50B in the example of FIG. 2) having different file sizes (i.e., different qualities) are stored in the file storage unit 13, it becomes possible to provide (transmit) a second file having an appropriate quality to the user terminal 20 based on the attributes of the user terminal 20. Note that in the example of FIG. 3, the file 50B is determined as the second file to be transmitted to the user terminal 20 by the above-described determination process.

[0047] The third process is a process of switching the first virtual object based on file 50C displayed on the display unit 24 to a first virtual object based on file 50B when a first virtual object based on a first file (file 50C) is displayed on the display unit 24 (display area 24a) and transmission of a second file (here, as an example, file 50B) to the user terminal 20 has been completed (see arrow P1 in Figure 3).

[0048] Switching from file 50C to file 50B can be performed, for example, by changing the identifier designated as the display target on the user terminal 20 (here, the identifier corresponding to file 50C (e.g., the above-mentioned URI, object ID, etc.)) to the identifier corresponding to file 50B. Furthermore, information such as the angle of view, position, and distance (depth) of the first virtual object based on file 50C that was displayed on the display unit 24 immediately before the switching may be configured to be carried over after the switching. In this case, it is possible to smoothly switch from file 50C to file 50B while maintaining the way the first virtual object appears to the user.

[0049] Furthermore, when a first virtual object based on the second file (file 50B) is displayed on the display unit 24 (display area 24a) (see the lower part of FIG. 3 ), the display control unit 14 may execute display control to switch back the first virtual object based on file 50B displayed on the display unit 24 to a first virtual object based on the first file (file 50C) depending on the occurrence status of a processing delay in the user terminal 20. For example, the display control unit 14 may determine whether a processing delay is occurring in the user terminal 20 based on information including information related to the processing performance of the user terminal 20 (for example, the terminal information described above). For example, the display control unit 14 may determine that a processing delay is occurring when the processor utilization rate of the user terminal 20 is equal to or higher than a threshold. Then, when determining that a processing delay is occurring, the display control unit 14 may cause the user terminal 20 to execute a process of switching back from the second file (file 50B) to the first file (file 50C) (see arrow P2 in FIG. 3 ). Since file 50C has a smaller file size than file 50B, the amount of calculation and memory usage required for the display process (drawing process) is smaller than those for file 50B. Therefore, according to the above display control, by executing the cutback process when a processing delay occurs, the processing delay can be resolved.

[0050] Furthermore, when a first virtual object based on the second file (file 50B) is displayed on the display unit 24 (display area 24a) and a plurality of virtual objects including the first virtual object are displayed on the display unit 24, the display control unit 14 may execute display control to switch back the first virtual object based on file 50B displayed on the display unit 24 to the first virtual object based on the first file (file 50C) in accordance with the number of virtual objects displayed on the display unit 24. For example, the display control unit 14 may acquire information indicating the number of virtual objects displayed on the display unit 24. For example, the display control unit 14 may acquire such information from the user terminal 20, or may know the number of virtual objects displayed on the display unit 24 based on a history of transmission of virtual object files to the user terminal 20. Then, for example, when the display control unit 14 determines that the number of displayed virtual objects is equal to or greater than a predetermined threshold, the display control unit 14 may cause the user terminal 20 to execute processing to switch back from the second file (file 50B) to the first file (file 50C) (see arrow P2 in FIG. 3 ). According to the above display control, by performing a revert process in a situation where a processing delay is likely to have occurred (or is likely to occur in the future) due to a large number of displayed virtual objects, it is possible to eliminate or avoid the processing delay.

[0051] Next, an example of the operation of the server 10 and the user terminal 20 will be described with reference to Fig. 4. In step S1, the imaging unit 21 captures an image of a real object. In step S2, the transmission unit 22 transmits the imaging data acquired in step S1 to the server 10. The transmission unit 22 also transmits terminal information (information including, as an example in this embodiment, application type, communication speed, processing performance, etc.) to the server 10. The imaging data and terminal information are received by the reception unit 11 of the server 10.

[0052] In step S3, the file generation unit 12 generates a file 50A (second file), a file 50B (second file), and a file 50C (first file) based on the imaging data (see FIG. 2).

[0053] In step S4, the file storage unit 13 stores the files 50A, 50B, and 50C generated by the file generation unit 12.

[0054] Subsequently, the processes of steps S5 and S6 (first process) are executed, for example, when the server 10 receives a display request for the first virtual object from the user terminal 20. That is, in step S5, the display control unit 14 transmits the first file (file 50C in the example of FIG. 3) to the user terminal 20 with priority over the second file (file 50B in the example of FIG. 3). As a result, in step S6, the first virtual object based on the first file (file 50C) is displayed on the display unit 24 (see the upper part of FIG. 3).

[0055] Furthermore, in step S7, the display control unit 14 executes a second process. That is, after transmitting the first file (file 50C), or in parallel with transmitting the first file, the display control unit 14 transmits the second file (file 50B) to the user terminal 20. Because the file size of the second file (second file size) is larger than the file size of the first file (first file size), the second file is transmitted to the user terminal 20 after step S6. That is, the reception of the second file at the user terminal 20 is completed at a point after step S6.

[0056] Subsequently, in step S8, the display control unit 14 executes a third process. That is, when a first virtual object based on the first file (file 50C) is displayed on the display unit 24 (display area 24a) and transmission of the second file (file 50B) to the user terminal 20 is completed, the display control unit 14 switches the first virtual object based on file 50C displayed on the display unit 24 to a first virtual object based on file 50B (see arrow P1 in FIG. 3). For example, the display control unit 14 can cause the user terminal 20 to execute the above-described switching process by transmitting to the user terminal 20, together with the second file, instruction information instructing the user terminal 20 to "switch from the first file to the second file after receiving the second file."

[0057] Next, in step S9, the display control unit 14 determines whether to perform display control to switch back the first virtual object based on file 50B displayed on the display unit 24 to the first virtual object based on the first file (file 50C). This determination process can be performed periodically or irregularly based on terminal information (e.g., information related to processing performance) received from the user terminal 20, information indicating the number of virtual objects displayed on the display unit 24, etc.

[0058] For example, as described above, when the display control unit 14 determines that a processing delay has occurred in the user terminal 20, or when the display control unit 14 determines that the number of virtual objects displayed on the display unit 24 is equal to or greater than a threshold, the display control unit 14 instructs the user terminal 20 to execute a cutback process (step S10). As a result, in step S11, a process of cutting back from the second file (file 50B) to the first file (file 50C) is executed (see arrow P2 in FIG. 3). Note that after the cutback process has been executed (i.e., after step S11), the display control unit 14 may execute the process of step S8 again if it determines that the reason for the cutback (for example, the above-mentioned processing delay has occurred, the number of displayed virtual objects is equal to or greater than a threshold, etc.) has been resolved.

[0059] In the server 10 described above, the first file (file 50C) having a small file size (first file size) is transmitted to the user terminal 20 first. The above configuration shortens the waiting time until the first virtual object is displayed on the display unit 24 of the user terminal 20. After the first file is transmitted, or in parallel with the transmission of the first file, the second file (file 50B) is transmitted to the user terminal 20. Then, after the user terminal 20 receives the second file, the first virtual object based on the first file displayed on the display unit 24 is switched to the first virtual object based on the second file (see arrow P1 in FIG. 3). The above configuration allows the first virtual object based on the second file, which has a larger file size and higher quality than the first file, to be presented to the user later, thereby maintaining the quality of the user's virtual experience.

[0060] Furthermore, in this embodiment, the file generation unit 12 generates a first file and a second file of a virtual object corresponding to a real object based on the imaging data, and the file storage unit 13 stores the first file and the second file generated by the file generation unit 12 in association with the virtual object. According to the above configuration, when a virtual object corresponding to a real object is generated, the first file and the second file corresponding to the virtual object can be automatically generated and stored.

[0061] In the above embodiment, the display control system is configured by the server 10. However, the display control system may be configured by the server 10 alone, or may be configured by the server 10 and the user terminal 20. That is, some of the functions of the server 10 described above may be executed by the user terminal 20. For example, in the above embodiment, the display control unit 14 of the server 10 controls the display content of the display unit 24 of the user terminal 20. However, the display control unit 14 may be provided in the user terminal 20, not in the server 10. Furthermore, some of the functions of the server 10 described above may be omitted. For example, in the above embodiment, the server 10 (file generation unit 12) generates a file of a virtual object, but the file of a virtual object may be generated by a device other than the server 10.

[0062] The display control system of the present disclosure (the server 10 in the above embodiment) has the following configuration.

[0063] [1] A display control system that controls the display of a virtual object displayed on a display unit of a user terminal, a file storage unit that stores, for a first virtual object that is one of the virtual objects, a first file having a first file size and a second file having a second file size larger than the first file size; a display control unit that controls display of the first virtual object on the user terminal, The display control unit a first process of transmitting the first file to the user terminal with priority over the second file, thereby displaying the first virtual object based on the first file on the display unit; a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file; and executing a third process of switching the first virtual object based on the first file displayed on the display unit to the first virtual object based on the second file when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal has been completed. Display control system.

[0064] [2] further comprising a file generation unit that acquires imaging data obtained by imaging a real object that is an object existing in real space, and generates the first file and the second file of the virtual object corresponding to the real object based on the imaging data; the file storage unit stores the first file and the second file generated by the file generation unit in association with the virtual object corresponding to the real object; [1] The display control system according to [1].

[0065] [3] When the first virtual object based on the second file is displayed on the display unit, the display control unit switches back the first virtual object based on the second file displayed on the display unit to the first virtual object based on the first file according to an occurrence status of a processing delay of the user terminal. [1] or [2]. A display control system according to [1] or [2].

[0066] [4] When the first virtual object based on the second file is displayed on the display unit and a plurality of virtual objects including the first virtual object are displayed on the display unit, the display control unit switches back the first virtual object based on the second file displayed on the display unit to the first virtual object based on the first file, depending on the number of the virtual objects displayed on the display unit. A display control system that is one of [1] to [3].

[0067] [5] The display control unit is configured to be able to switch between a first control pattern in which transmission of the second file to the user terminal is started after transmission of the first file, and a second control pattern in which transmission of the second file to the user terminal is performed in parallel with transmission of the first file, in the second process. A display control system that is one of [1] to [4].

[0068] [6] A terminal information acquisition unit that acquires terminal information indicating attributes of the user terminal, the file storage unit stores a plurality of the second files having different file sizes; the display control unit determines, based on the terminal information, the second file to be transmitted to the user terminal from among the plurality of second files. A display control system that is one of [1] to [5].

[0069] Furthermore, the block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, using wires, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or the multiple devices with software.

[0070] Functions include, but are not limited to, judging, determining, calculating, computing, processing, deriving, investigating, searching, verifying, receiving, transmitting, outputting, accessing, resolving, selecting, choosing, establishing, comparing, assuming, expecting, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning.

[0071] For example, the server 10 according to an embodiment of the present disclosure may function as a computer that performs the display control method of the present disclosure. Fig. 5 is a diagram illustrating an example of a hardware configuration of the server 10 according to an embodiment of the present disclosure. The server 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. The user terminal 20 may also have a hardware configuration similar to that of the server 10.

[0072] In the following description, the term "device" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the server 10 may be configured to include one or more of the devices shown in Fig. 5, or may be configured to exclude some of the devices.

[0073] Each function of the server 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0074] The processor 1001 controls the entire computer by running, for example, an operating system, and may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc.

[0075] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with the programs. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, each functional unit of the server 10 (e.g., the display control unit 14, etc.) may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be made for other functional blocks. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

[0076] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a display control method according to an embodiment of the present disclosure.

[0077] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray disc), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0078] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0079] The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (for example, a touch panel).

[0080] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0081] The server 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0082] Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

[0083] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0084] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0085] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0086] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, notification of predetermined information (e.g., notification that "X is true") is not limited to being done explicitly, but may be done implicitly (e.g., by not notifying the predetermined information).

[0087] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0088] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0089] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0090] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0091] The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the mathematical formulas and the like that use these parameters may differ from those explicitly disclosed in this disclosure. The various information elements may be identified by any suitable names, and the various names assigned to these various information elements are not intended to be limiting in any way.

[0092] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0093] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0094] When used in this disclosure, the terms "include," "including," and variations thereof are intended to be inclusive, similar to the term "comprising." Furthermore, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0095] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0096] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]

[0097] 10...Server (display control system), 11...Receiving unit, 12...File generation unit, 13...File storage unit, 14...Display control unit, 20...User terminal, 24...Display unit, 24a...Display area, 50A...File (second file), 50B...File (second file), 50C...File (first file).

Claims

1. A display control system that controls the display of a virtual object displayed on a display unit of a user terminal, a file storage unit that stores, for a first virtual object that is one of the virtual objects, a first file having a first file size and a second file having a second file size larger than the first file size; a display control unit that controls display of the first virtual object on the user terminal, The display control unit a first process of transmitting the first file to the user terminal with priority over the second file, thereby displaying the first virtual object based on the first file on the display unit; a second process of transmitting the second file to the user terminal after transmitting the first file or in parallel with transmitting the first file; a third process of switching the first virtual object based on the first file displayed on the display unit to the first virtual object based on the second file when the first virtual object based on the first file is displayed on the display unit and transmission of the second file to the user terminal has been completed; The display control unit is configured to be able to switch, in the second process, between a first control pattern in which transmission of the second file to the user terminal is started after transmission of the first file, and a second control pattern in which transmission of the second file to the user terminal is performed in parallel with transmission of the first file, and to determine to execute the second control pattern when a communication speed of the user terminal is equal to or higher than a predetermined threshold. Display control system.

2. a file generation unit that acquires imaging data obtained by imaging a real object that is an object existing in a real space, and generates the first file and the second file of the virtual object corresponding to the real object based on the imaging data, the file storage unit stores the first file and the second file generated by the file generation unit in association with the virtual object corresponding to the real object; The display control system according to claim 1 .

3. the display control unit, when the first virtual object based on the second file is displayed on the display unit, switches the first virtual object based on the second file displayed on the display unit back to the first virtual object based on the first file in accordance with an occurrence status of a processing delay of the user terminal; The display control system according to claim 1 .

4. when the first virtual object based on the second file is displayed on the display unit and a plurality of virtual objects including the first virtual object are displayed on the display unit, the display control unit switches back the first virtual object based on the second file displayed on the display unit to the first virtual object based on the first file, in accordance with the number of the virtual objects displayed on the display unit; The display control system according to claim 1 .

5. a terminal information acquisition unit that acquires terminal information indicating attributes of the user terminal; the file storage unit stores a plurality of the second files having different file sizes; the display control unit determines, based on the terminal information, the second file to be transmitted to the user terminal from among the plurality of second files; The display control system according to claim 1 .

Images