Display processing apparatus, display processing method, storage medium, and information processing apparatus

Abstract

[Problem] To provide a display processing device, a display processing method, a storage medium, and an information processing device capable of reducing delays in cases of receiving and displaying data of a virtual object created by an external device. [Solution] This display processing device comprises: a transmission unit that transmits first operation information of a mobile terminal to an external device; a reception unit that receives, at a first time instant, data of a first display region of a virtual object rendered by the external device on the basis of the first operation information, and receives, at a second time instant after the first time instant, data of a second display region of the virtual object rendered by the external device on the basis of the first operation information, the second display region being adjacent to the first display region; and a display control unit that performs control to display, on a display device of the mobile terminal, the received data of the first display region and then display the data of the second display region together with the data of the first display region as a single frame.

Description

[Technical Field]

[0001] The present disclosure relates to a display processing device, a display processing method, a storage medium, and an information processing device. [Background technology]

[0002] In recent years, various technologies have been developed for viewing by merging real space and virtual space. For example, AR (Augmented Reality) technology displays images of a virtual space (hereinafter referred to as a virtual object) superimposed on the real space while the real space is directly visible, and VR (Virtual Reality) technology allows a user to view a generated virtual space from any viewpoint. Such AR and VR technologies can be provided using, for example, a head-mounted display (HMD), which is a display device worn on the user's head. More specifically, an optically transparent HMD is an example of a device that realizes AR. Furthermore, a non-transparent HMD is an example of a device that realizes VR, which covers the user's field of view with a display unit to enhance the sense of immersion.

[0003] Furthermore, with regard to technology for displaying virtual objects using an HMD, for example, Patent Document 1 below describes that when a virtual object is superimposed on real space using an optically transparent HMD, the position and posture of the virtual object are made to follow the movement of the user wearing the HMD or an object in real space, thereby reducing the sense of discomfort caused by a deviation in the display position of the virtual object relative to real space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 181263 Summary of the Invention [Problem to be solved by the invention]

[0005] The above prior art does not take into consideration the delay that occurs when receiving and displaying data of a virtual object generated by an external device.

[0006] Therefore, the present disclosure proposes a display processing device, a display processing method, a storage medium, and an information processing device that are capable of reducing delays when receiving and displaying data of a virtual object generated by an external device. [Means for solving the problem]

[0007] According to the present disclosure, a display processing device is proposed that includes a transmitting unit that transmits first motion information of a mobile terminal to an external device; a receiving unit that receives data of a first display area of a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area adjacent to the first display area of the virtual object drawn by the external device based on the first motion information at a second time after the first time; and a display control unit that displays the received data of the first display area and then controls the display of the data of the second display area together with the data of the first display area as one frame on a display device of the mobile terminal.

[0008] According to the present disclosure, a display processing method is proposed, including: a processor transmitting first motion information of a mobile terminal to an external device; receiving, at a first time, data of a first display area of a virtual object drawn by the external device based on the first motion information; receiving, at a second time after the first time, data of a second display area adjacent to the first display area of the virtual object drawn by the external device based on the first motion information; and, after displaying the received data of the first display area, controlling the display device of the mobile terminal to display the data of the second display area together with the data of the first display area as one frame.

[0009] According to the present disclosure, a program for causing a computer to function as a transmission unit that transmits first operation information of a mobile terminal to an external device, a reception unit that receives data of a first display area of a virtual object drawn based on the first operation information by the external device at a first time, and receives data of a second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time, and a display control unit that performs control to display the data of the second display area together with the data of the first display area as one frame on a display device of the mobile terminal after displaying the received data of the first display area, is stored in a storage medium.

[0010] According to the present disclosure, a reception unit that receives first operation information of a mobile terminal, and based on the first operation information, transmits data of a first display area of a virtual object drawn as a part of one frame to a display processing device that performs control to display the data of the first display area on a display device of the mobile terminal at a first time, and after drawing the data of the first display area based on the first operation information, a transmission unit that transmits data of a second display area adjacent to the first display area of the virtual object drawn as another part of the one frame to the display processing device at a second time after the first time, is provided.

Brief Description of Drawings

[0011] [Figure 1] It is a block diagram showing an example of a basic configuration of an information processing apparatus according to an embodiment of the present disclosure. [Figure 2] It is a diagram for explaining the length of a delay when performing drawing processing in an external device. [Figure 3] It is a diagram for explaining each slice in a scanning display. [Figure 4] It is a diagram for explaining a series of flows of processing in units of slices according to the present embodiment. [Figure 5]It is a sequence diagram showing an example of the flow of display processing in slice units of the display processing system according to the present embodiment. [Figure 6] It is a diagram for explaining the importance of each slice according to the gaze point in the present embodiment. [Figure 7] It is a diagram for explaining the display order of each slice in the priority processing according to the present embodiment. [Figure 8] It is a diagram for explaining a series of flows for preferentially performing processing in slice units according to the present embodiment in an arbitrary order. [Figure 9] It is a sequence diagram showing an example of the flow of display processing for preferentially performing processing in slice units according to the present embodiment in an arbitrary order. [Figure 10] It is a diagram for explaining other display orders of each slice in the priority processing according to the present embodiment. [Figure 11] It is a diagram for explaining a series of flows for processing slices with high importance first in the processing in slice units according to the present embodiment. [Figure 12] It is a block diagram showing an example of the functional configuration of the image deformation unit according to the present embodiment. [Figure 13] It is a diagram showing an example of an image generated in slice units by the image generation unit according to the present embodiment. [Figure 14] It is a diagram showing an example of the phenomenon of tearing that may occur during display. [Figure 15] It is a diagram for explaining the image deformation of the contour line of the virtual object according to the present embodiment. [Figure 16] It is a diagram for specifically explaining the deformation of the contour line of the virtual object according to the present embodiment. [Figure 17] It is a diagram for explaining the adjustment of the dividing line for dividing one frame into a plurality of slices according to the present embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0012] With reference to the accompanying drawings, preferred embodiments of the present disclosure will be described in detail. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0013] Also, the description will be made in the following order. 1. Configuration of a display processing system according to an embodiment of the present disclosure 1-1. Configuration example of the mobile terminal 10 1-2. Configuration example of the server 20 2. Sorting out problems 3. Technical features 3-1. Processing in slice units 3-2. Priority processing 3-3. Suppression of tearing (3-3-1. Black insertion period) (3-3-2. Image deformation) (3-3-3. Adjustment of dividing lines) 4. Supplementary notes

[0014] <<1. Configuration of a display processing system according to an embodiment of the present disclosure>> FIG. 1 is a diagram showing the configuration of a display processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the display processing system according to an embodiment of the present disclosure includes a mobile terminal 10 and a server 20.

[0015] The mobile terminal 10 and the server 20 are communicatively connected via a network 30 and perform data transmission and reception. When the server 20 is a cloud server, the network 30 is assumed to be the Internet including wireless or wired (for example, 5G (fifth-generation mobile communication system)). When the server 20 exists in the same space as the mobile terminal 10, such as a smartphone or a notebook PC, short-range wireless communication such as Wi-Fi (registered trademark) or wired (such as a USB cable) is assumed.

[0016] The mobile terminal 10 is an example of an information processing terminal that controls the display of virtual objects in VR or AR. The mobile terminal 10 is realized by, for example, an HMD worn on the user's head, a wearable device such as a smartphone, a tablet terminal, or a glasses-type display device. The mobile terminal 10 receives display data of virtual objects from the server 20 and controls the display on the display unit 160 of the mobile terminal 10. The server 20 is an example of an information processing device that generates display data of virtual objects in VR or AR. Specifically, the server 20 receives information on the position and orientation from the mobile terminal 10, performs drawing of virtual objects based on the received position and orientation information, and transmits the drawing result to the mobile terminal 10.

[0017] Note that since image data (data of the drawing result of virtual objects) is continuously sent from the server 20 to the mobile terminal 10, it may be assumed to use a data link layer that can secure bandwidth. Also, since the transmission from the mobile terminal 10 to the server 20 has a small data volume and is greatly affected by communication delay, a lower-latency data link layer may be assumed. For example, when the server 20 is a cloud server and uses 5G for the network 30, the transmission from the server 20 to the mobile terminal 10 may use eMBB (enhanced Mobile Broadband), and URLLC (Ultra-Reliable and Low Latency Communications) may be used for the transmission from the mobile terminal 10 to the server 20. Also, when the server 20 is a smartphone, a notebook PC, etc., and can be carried by a user wearing the mobile terminal 10 (for example, an HMD), Bluetooth (registered trademark) or Wi-Fi (registered trademark) may be used for the network 30, or a wired connection such as USB or Gigabit Ethernet may be used.

[0018] <1-1. Configuration example of the mobile terminal 10> As shown in FIG. 1, the mobile terminal 10 includes a control unit 100, a sensor unit 110, a communication unit 120, a clock signal generation unit 130, a timestamp counter 140, a memory unit 150, and a display unit 160.

[0019] The communication unit 120 communicatively connects to a server 20, which is an external device, via a network 30, either wired or wirelessly, and transmits and receives data. For example, the communication unit 120 transmits the position and orientation information of the mobile terminal 10. Also, the communication unit 120 receives data on the rendering result of virtual objects from the server 20. The communication unit 120 may be, for example, a wired / wireless LAN (Local Area Network), or Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 3G (third-generation mobile communication system), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system)), etc.

[0020] More specifically, for example, the communication unit 120 may have a function of packetizing data to be transmitted according to a communication protocol under the control of the control unit 100 and transmitting it to the server 20. Also, the communication unit 120 may have a function of distributing packets sent according to a communication protocol to destinations within the mobile terminal 10. The communication unit 120 can receive not only data on the rendering result of virtual objects but also a clock signal from the server 20 for time synchronization.

[0021] The sensor unit 110 is various sensors that detect the operation information of the mobile terminal 10. For example, the sensor unit 11 may be a camera or an IMU (Inertial Measurement Unit). More specifically, the sensor unit 110 acquires sensing data for estimating the head position and orientation and the eye position and orientation of the user who uses the mobile terminal 10. For example, when the mobile terminal 10 is realized by an HMD and is worn on the user's head, the position and orientation of the mobile terminal 10 are estimated to be the position and orientation of the user's head. The eye position and orientation are detected by sensors (such as a camera and an infrared sensor) fixed in the direction of the user's eyes inside the HMD.

[0022] The control unit 100 functions as an arithmetic processing unit and a control device, and controls the overall operations within the mobile terminal 10 according to various programs. The control unit 100 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example. Further, the control unit 100 may include a ROM (Read Only Memory) that stores programs, arithmetic parameters, etc. to be used, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0023] The control unit 100 according to this embodiment is also a display processing device that functions as a position and orientation estimation unit 101, a data decoding unit 102, an image deformation unit 103, a position and orientation prediction unit 104, and a display control unit 105. The position and orientation estimation unit 101 estimates the position and orientation of the user based on various sensing data detected by the sensor unit 110. Specifically, the position and orientation estimation unit 101 has a so-called Head Tracker function that estimates the position and orientation of the user's head based on camera images of the surroundings captured by the sensor unit 110 and sensing data of the IMU. The position and orientation of the head may be the position and orientation on a coordinate system defined with respect to the surrounding real space. Note that the Head Tracker has an outside-in method that uses sensors installed outside, an inside-out method that uses sensors mounted on the object to be measured itself (the mobile terminal 10 (HMD)), and a hybrid method that is a combination of these. In addition, the position and orientation estimation unit 101 also has a so-called Eye Tracker function that estimates the position and orientation of the user's eyeballs (such as the position of the pupils and irises) based on infrared camera images of the user's eyes captured by the sensor unit 110. The position and orientation of the user's eyeballs is one type of gaze information, and the Eye Tracker may further estimate the fixation point (coordinates) on the display screen of the display unit 160. The sensor unit 110 continuously performs sensing, and the control unit 100 continuously transmits the position and orientation information (an example of the first operation information) estimated by the position and orientation estimation unit 101 from the communication unit 120 to the server 20. On the server 20 side, the image generation unit 201 described later performs rendering processing of virtual objects based on the position and orientation information (the first operation information) received from the mobile terminal 10, and transmits the data of the rendering result to the mobile terminal 10. The data to be transmitted is encoded by the data encoding unit 203 described later.

[0024] The data decoding unit 102 decodes the data of the rendering result (including the image data of the virtual object) transmitted from the server 20, and outputs the decoded data to the image deformation unit 103. The image deformation unit 103 receives, as inputs, the output from the data decoding unit 102 (including the image data of the virtual object) and the prediction result based on the latest position and orientation information (an example of the second motion information) by the position and orientation prediction unit 104, and performs image deformation so that the virtual object appears to be located at a more accurate position at the time when the image of the virtual object is displayed on the display unit 160. On the mobile terminal 10 side, delay compensation can be performed by performing image deformation on the image data of the virtual object received from the server 20 using the prediction result of the position and orientation information. The result of the image deformation is output as a pixel column to the display unit 160 realized by a scanning display. Details of the image deformation will be described later.

[0025] In the position and orientation prediction unit 104, based on the latest position and orientation information (an example of the second motion information) estimated by the position and orientation estimation unit 101, the position and orientation information after a predetermined time is predicted. The time after the predetermined time assumes the predicted display time on the display unit 160 of the slice for which image deformation is to be performed using this prediction. For example, the position and orientation prediction unit 104, based on the head position and orientation information h estimated by the position and orientation estimation unit 101 at time t w T h (t h ), calculates the head position and orientation information w T h (t) at a future time t. Also, the position and orientation prediction unit 104, based on the eyeball position and orientation information e estimated by the position and orientation estimation unit 101 at time t h T e (t e ), calculates the eyeball position and orientation information h T e (t) at a future time t.

[0026] The display control unit 105 controls the display unit 160 to display the image data appropriately deformed by the image deformation unit 103. Details of the display process will be described later.

[0027] The clock signal generating unit 130 acquires the clock signal (the clock signal that serves as the reference for timing control of the entire system) generated by the clock signal generating unit 230 of the server 20 using a clock propagation mechanism within the network 30, and references the clock signal to generate a clock signal synchronized with the server 20 using a PLL (phase-locked loop). Whether a wired or wireless communication connection is used between the mobile terminal 10 and the server 20, the data link layer of the network 30 is equipped with a means for transmitting and receiving a reference clock signal. For example, LTE, 5G, Bluetooth, and Wi-Fi, which can be used for wireless connections, can all define time slots with a time axis resolution of 1 ms or less and perform communication control, enabling clock synchronization using the server 20 as the reference clock. Even in the case of a wired connection, clock synchronization using the server 20 as the reference clock is possible in a system in which a host controller, such as USB, controls the timing of the entire bus. In addition, in the case of GbE (Gigabit Ethernet), it is possible to synchronize (between the timestamp counters of the mobile terminal 10 and the server 20) with sufficient precision by using PTP (Precision Time Protocol) in combination.

[0028] The time stamp counter 140 is a time counter for controlling the timing of the entire system. The time stamp counter 140 may increase monotonically in principle, except in special cases such as during initialization. At the time of system initialization, after initializing the time stamp counter 140 of the mobile terminal 10 according to the value of the time stamp counter 240 of the server 20, the counter is advanced using the reference clock signal generated by the clock signal generation unit 230 or the clock signal generation unit 130, so that the values of the time stamp counter 140 and the time stamp counter 240 are synchronized with sufficient accuracy (for example, at least an error of 1 ms or less). The value of the time stamp counter 140 is referred to, for example, when the control unit 100 captures sensing data from the sensor unit 110 and is used as the time stamp of the sensing data. Also, regarding the display control of the display unit 160 realized by the scanning type display, the time stamp counter value is referred to at the timing of the start of scanning of each frame (commonly known as VSync: Vertical Synchronizing signal), and is used for estimating the next VSync scheduled time.

[0029] Note that the above-described timing synchronization method and its use are merely examples, and the present disclosure is not limited thereto.

[0030] The storage unit 150 is realized by a ROM (Read Only Memory) that stores programs, arithmetic parameters, etc. used in the processing of the control unit 100, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0031] The display unit 160 is a display device that displays an image of a virtual object. Also, the display unit 160 according to the present embodiment is realized by a scanned display. The scanned display sequentially performs display on each slice (display area) that is divided into a plurality of parts in a direction perpendicular to the scanning direction. That is, the scanned display has a plurality of adjacent display areas with different display timings (the lighting times are different depending on the position on the screen). Each slice will be described later with reference to FIG. 3. More specifically, the scanned display displays the pixel column output from the image deformation unit 103. As types of the scanned display, for example, the following two types are assumed.

[0032] One is a type in which the line to be scanned can be arbitrarily selected (that is, the scanning order can be arbitrarily changed). Specifically, displays using OLED (Organic Light Emitting Diode) or LED can be mentioned. In this type, the lighting time (hold time) of each pixel can also be changed. The other is a type in which the line to be scanned cannot be arbitrarily selected (that is, the scanning order cannot be arbitrarily changed). Specifically, an LBS (laser scanning type) display that combines a laser light source and a MEMS mirror can be mentioned.

[0033] Also, the display unit 160 realized by the scanned display may be an optical see-through display that can directly deliver the light in the real space to the user's eyes, or a non-transmissive display in which the light in the real space does not directly reach the user's eyes. Also, the display unit 160 may be a display that is switchable between an optical see-through display and a non-transmissive display. Also, the display unit 160 may be provided in an HMD, or may be provided in a smartphone, a tablet terminal, or the like. Also, the display unit 160 may be provided in a separate device that is communicatively connected to the mobile terminal 10.

[0034] The configuration of the mobile terminal 10 has been specifically described above. However, the configuration of the mobile terminal 10 according to the present disclosure is not limited to the example shown in FIG. 1. For example, the mobile terminal 10 may be implemented by a plurality of devices. Specifically, it may be a configuration including a display device (corresponding to at least the display unit 160) implemented by an HMD or the like, and an information processing terminal (a display processing device corresponding to at least the control unit 100) implemented by a smartphone, a tablet terminal, a PC, or the like.

[0035] <1-2. Configuration Example of Server 20> As shown in FIG. 1, the server 20 includes a control unit 200, a communication unit 220, a clock signal generation unit 230, a timestamp counter 240, and a storage unit 250.

[0036] The communication unit 220 communicatively connects to the mobile terminal 10 via the network 30 by wire or wirelessly, and transmits and receives data. For example, the communication unit 220 receives the position and orientation information from the mobile terminal 10. Also, the communication unit 220 transmits the data of the rendering result of the virtual object generated by the control unit 200 to the mobile terminal 10. The communication unit 220 may be, for example, a wired / wireless LAN (Local Area Network), or Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 3G (third-generation mobile communication system), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system)), or the like.

[0037] The control unit 200 functions as an arithmetic processing device and a control device, and controls the overall operation within the server 20 according to various programs. The control unit 200 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example. Also, the control unit 200 may include a ROM (Read Only Memory) that stores programs and arithmetic parameters to be used, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0038] Also, the control unit 200 according to this embodiment also functions as an image generation unit 201, a position and orientation prediction unit 202, and a data encoding unit 203. The image generation unit 201 has a function of referring to the position and orientation information (head position and orientation information or eyeball position and orientation information) received from the mobile terminal 10, drawing a virtual object, and outputting it as an image. In the case of VR, the image generation unit 201 draws a virtual space as seen from the viewpoint position (user viewpoint in the virtual space) corresponding to the position and orientation information. Further, the drawing result output from the image generation unit 201 may include distance information of the virtual object and speed information of the virtual object in addition to the RGB image. Also, the image generation unit 201 may draw a virtual object with reference to the latest position and orientation prediction information by the position and orientation prediction unit 202. The position and orientation prediction unit 202 predicts the position and orientation information after a predetermined time based on the position and orientation information received from the mobile terminal 10. The time after the predetermined time assumes the display prediction time on the mobile terminal 10. The drawing process based on the position and orientation prediction information can be said to be one of the delay compensations.

[0039] The data encoding unit 203 encodes the drawing result output from the image generation unit 201. The control unit 200 transmits the data encoded by the data encoding unit 203 from the communication unit 220 to the mobile terminal 10.

[0040] The clock signal generation unit 230 has a function of generating a clock signal that serves as a reference for the timing control of the entire system. The server 20 transmits the clock signal generated by the clock signal generation unit 230 from the communication unit 220 to the mobile terminal 10.

[0041] The timestamp counter 240 is a time counter for the timing control of the entire system. The timestamp counter 240 advances the counter using the reference clock signal generated by the clock signal generation unit 230.

[0042] The memory unit 250 is realized by a ROM (Read Only Memory) that stores programs, arithmetic parameters, etc. used in the processing of the control unit 200, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0043] As described above, the configuration of the server 20 has been specifically described, but the configuration of the server 20 according to the present disclosure is not limited to the example shown in FIG. 1. For example, the server 20 may be realized by a plurality of devices.

[0044] <<2. Organization of problems>> Here, since more high-definition drawing processing increases power consumption, leading to an increase in the size of the heat dissipation mechanism and the battery capacity of the terminal performing the drawing processing, by performing the drawing of virtual objects on the server 20 as in this embodiment, miniaturization and weight reduction of the mobile terminal 10 can be achieved. However, when performing the drawing processing of virtual objects on an external device such as the server 20, compared with the case of performing it on the mobile terminal 10 alone, the delay increases due to communication and image compression / expansion, and there is a risk that the sense of localization of the virtual object when the virtual object is displayed on the mobile terminal 10 is impaired.

[0045] FIG. 2 is a diagram for explaining the length of delay in the case of performing drawing processing by an external device. In the example shown in FIG. 2, first, the position and orientation estimation of the user's eyeballs and head is performed by various sensors (Eye Tracker and Head Tracker) provided in a mobile terminal such as an HMD worn on the user's head, and the position and orientation information is transmitted to a server which is an external device (Upstream transfer). The server performs drawing processing of virtual objects (that is, generation of images of virtual objects) based on the received position and orientation information (Render function), encodes (compresses) this (Video Encode is performed), and transmits it to the mobile terminal. When the mobile terminal receives the data (Downstream transfer), it decodes (decompresses) the encoded data (Video Decode is performed). So far, a delay due to communication and encoding / decoding of data (the passage of time from the acquisition time of the position and orientation information used for drawing the virtual object) has already occurred. Since time has passed since the time when the position and orientation information used for drawing the virtual object was acquired, if there is movement on the mobile terminal side during that time, the sense of localization of the virtual object displayed on the mobile terminal may be impaired. That is, in the case of AR, a deviation occurs in the display position and orientation between the real space and the virtual object to be superimposed, and in the case where a video see-through display is used, a deviation also occurs in the display position and orientation between the video of the real space and the virtual object to be superimposed. Also, in the case of VR, the video of the virtual space does not follow (or follows with a delay) the user's movement, resulting in a sense of discomfort and unnaturalness, and the immersion feeling in the virtual space is impaired.

[0046] In addition, when the display unit of the mobile terminal is assumed to be a Scanned Display, the mobile terminal sequentially performs display on each slice (display area) that is divided into a plurality of parts in a direction perpendicular to the scanning direction. FIG. 3 is a diagram for explaining each slice in the scanned display. As shown in FIG. 3, the scanned display is divided into, for example, five slices (display areas) in a direction perpendicular to the scanning direction (the x-axis direction in the figure) (the y-axis direction in the figure). The boundaries between the slices are provided substantially parallel to the scanning direction. Each slice is sequentially subjected to display processing according to the scanning. That is, the scanned display has a plurality of adjacent display areas with different display timings (the lighting times are different depending on the position on the screen). For example, the display processing is sequentially performed from slice 1 (S1) to slice 5 (S5). The mobile terminal divides the data of the drawing result of the virtual object received from the server into five slices, sequentially scans them, and performs display output. At this time, the mobile terminal can reduce the display delay felt by the user by performing image deformation for each slice based on the position and orientation information of the mobile terminal acquired immediately before performing the display output for each slice. Examples of the image deformation include Time Warp and Distortion Correction.

[0047] However, when performing rendering processing on an external device, the delay due to communication and data encoding / decoding is significant, and it is not sufficient to only compensate for the delay of image deformation during the above display output. For example, assuming that the estimation of eye position and orientation (Eye tracker) operates at 300 frames per second (fps) with a processing time of 3.33 ms, and the estimation of head position and orientation (Head tracker) has a processing time of 2 ms. Assuming that the communication delay (Uplink latency) in transmitting data from the mobile terminal to the server is 1 ms for 5G URLLC, and the communication delay (Downlink latency) in receiving data from the server of the mobile terminal is 4 ms for 5G eMBB. Assuming that the processing delay of compression, network transfer, and decompression is 16.7 ms assuming a throughput of 60 fps, and the Vsync period of the scanning display is 16.7 ms, the delay from the acquisition (detection by the sensor) of the position and orientation information of the mobile terminal to the display (Motion-to-photon latency) is approximately 95 ms (5.7 V). Note that the "display" time in "until display" is, for example, represented by the center point of one frame (when the scanning display is evenly divided into five parts, the center of the slice 3 located in the center corresponds), and the display timing of the representative point is used as the reference. In the example shown in FIG. 2, the rendering and data transmission of the rendering result are performed in units of one frame, and it is possible to shorten it to approximately 18 ms (1.1 V) by Time Warp. However, the compensation period (approximately 95 ms (5.7 V) in this example) is long, and distortion is likely to occur.

[0048] Therefore, according to the present disclosure, a display processing system is proposed that can reduce the delay when receiving and displaying data of virtual objects generated by an external device.

[0049] Hereinafter, the technical features of the display processing system according to an embodiment of the present disclosure will be described.

[0050] <<3. Technical Features>> The display processing system according to this embodiment has the configuration shown in FIG. 1, and realizes a mechanism for more appropriately reducing the delay when the mobile terminal 10 receives and displays the data of virtual objects generated by the server 20 which is an external device. Specifically, in the image generation unit 201 of the server 20, one frame is divided into a plurality of display areas (referred to as slices in this specification), drawing is performed for each slice, and when the drawing of each slice is completed, they are sequentially compressed and transmitted. That is, in the example shown in FIG. 2, drawing, encoding, and transmission were performed in units of one frame, but in the display processing system according to this embodiment, drawing, encoding, and transmission are performed in units of slices. Also, in the mobile terminal 10, decoding, image transformation, and display output are sequentially performed in units of the received slices. In this way, in the display processing system according to this embodiment, by performing pipeline processing of drawing, transmission, transformation, and display output in units of slices, the delay time (Motion-to-photon latency) from when the position and orientation information is acquired to when the display is made can be shortened. Hereinafter, it will be specifically described with reference to FIG. 4.

[0051] <3-1. Processing in units of slices> FIG. 4 is a diagram for explaining a series of processes of processing in units of slices according to this embodiment. As shown in FIG. 4, in the display processing system according to this embodiment, in the image generation (drawing process of virtual objects in the image generation unit 201) of the server 20, one frame is divided into a plurality of slices and sequential drawing is performed, sequential encoding and transmission (reception at the mobile terminal 10) are performed in units of slices, and in the mobile terminal 10, image transformation and display output in units of slices are also performed. The number and size of the slices to be divided are not particularly limited. As an example, the image generation unit 201 of the server 20 may evenly divide one frame. For example, as shown in FIG. 3, the image generation unit 201 divides one frame into five display areas of slice 1 to slice 5 and sequentially draws from slice 1.

[0052] The image generation unit 201 refers to the prediction result of the position information predicted by the position and orientation prediction unit 202 based on the position and orientation information (an example of the first operation information) received from the mobile terminal 10, and performs drawing of the virtual object for each slice.

[0053] The communication unit 120 of the mobile terminal 10 receives the data of slice 1 (the first display area) at the first time t1. Further, the communication unit 120 of the mobile terminal 10 receives the data of slice 2 (the second display area) adjacent to slice 1 (the first display area) at the second time t2 after the first time. The display control unit 105 of the mobile terminal 10 performs control to start display output of the data of slice 2 (the second display area) as one frame together with the data of slice 1 after starting the display output of the data of slice 1 (the first display area) to the display unit 160.

[0054] Also, as shown in FIG. 4 , the communication unit 120 of the mobile terminal 10 receives data of a third display area (e.g., slice 5) corresponding to a portion of one frame rendered by the server 20 at a third time t3 that is later than the second time t2. The display control unit 105 of the mobile terminal 10 starts displaying data of slice 1 (first display area) at a time before the third time t3, and starts displaying data of slice 5 (third display area) after starting display of data of slice 2 (second display area). In this way, the first display area starts displaying at least before the third time, and the mobile terminal 10 can start displaying data without waiting for the entire frame to be rendered or received, thereby reducing delay time. Here, "slice 5" is given as an example of the "third display area," but the present embodiment is not limited thereto. The "third display area" refers to a display area other than the first display area corresponding to a portion of one frame rendered by the server 20 and the second display area adjacent to the first display area. 4, the "third display area" may be "slice 4" or "slice 3." In addition, although the first time to the third time are here defined as the reception times at which the rendering result data is received from the server 20, the present embodiment is not limited to this. For example, the first time to the third time may be the times at which the server 20 transmits the rendering result data to the mobile terminal 10, or the times at which the mobile terminal 10 decodes the rendering result data received from the server 20, or the times at which the image is transformed.

[0055] The image generation unit 201 of the server 20 may draw a virtual object for each slice based on the result of prediction of position information by the position and orientation prediction unit 202 based on position and orientation information (an example of first motion information) received from the mobile terminal 10. The drawing result for each slice by the image generation unit 201 includes an RGB image, distance information of the virtual object, and speed information of the virtual object. The distance information of the virtual object and the speed information of the virtual object may be used in image deformation (a type of delay compensation) by the image deformation unit 103 performed in the mobile terminal 10, which will be described later.

[0056] In addition, in the image deformation by the image deformation unit 103 of the mobile terminal 10, for each slice, by referring to the position and orientation information (an example of the second operation information) acquired immediately before and the prediction result based on the position and orientation information, image deformation is performed, thereby reducing the display delay felt by the user. Examples of image deformation include Time Warp and Distortion Correction.

[0057] Using more specific numerical values for the reduction of the delay time by executing pipeline processing in slice units, for example, assuming the various numerical values described above with reference to FIG. 2, the delay from the acquisition (detection by the sensor) of the position and orientation information of the mobile terminal 10 to the display is about 42 ms (2.7 V). In this case, for example, even if it is shortened to about 18 ms (1.1 V) by Time Warp, since the compensation period is shorter than the example shown in FIG. 2 (about 42 ms (2.7 V) in this example), distortion is less likely to occur. That is, it can be said that when the ratio of the period shortened by image deformation to the compensation period exceeds a predetermined value, distortion is less likely to occur.

[0058] FIG. 5 is a sequence diagram showing an example of the flow of display processing in slice units of the display processing system according to the present embodiment. As shown in FIG. 5, first, the position and orientation estimation unit 101 of the mobile terminal 10 estimates the position and orientation of the user's head and eyes based on the sensing data of the sensor unit 110 (step S103), and transmits the estimated position and orientation information to the server 20 (step S106).

[0059] Next, the image generation unit 201 of the server 20 generates an image (drawing process) of the virtual object in slice units based on the received position and orientation information (the first operation information) (step S109). At this time, the image generation unit 201 may perform the drawing of the virtual object with reference to the prediction result of the position and orientation information for a predetermined time predicted by the position and orientation prediction unit 202 based on the received position and orientation information.

[0060] Next, the server 20 sequentially transmits the data of the rendering results in slice units to the mobile terminal 10 (step S112). Specifically, the server 20 sequentially encodes the data of the rendering results in slice units output from the image generation unit 201 with the data encoding unit 203 and transmits it from the communication unit 220 to the mobile terminal 10. At the mobile terminal 10, the received data of the rendering results in slice units is sequentially decoded.

[0061] Next, the image deformation unit 103 of the mobile terminal 1 transforms the image data (image data of virtual objects in slice units) included in the received rendering results in slice units based on the most recent position and orientation information (second motion information) (step S115). At this time, the image deformation unit 103 may perform image deformation with reference to the position and orientation information prediction result for a predetermined time (display scheduled time) predicted by the position and orientation prediction unit 104 based on the most recent position and orientation information.

[0062] Then, the display control unit 105 of the mobile terminal 1 starts displaying the image data of the deformed slices in units of slices (step S118). Data of the rendering results in slice units is continuously transmitted sequentially from the server 20, and the mobile terminal 10 performs decoding, image deformation, and display processing in slice units.

[0063] In this way, in the present embodiment, by performing pipeline processing of rendering, transmission / reception, deformation, and display output in slice units, the delay time from when the position and orientation information is acquired to when it is displayed can be shortened.

[0064] <3-2. Priority processing> In the example described above, the case where slices 1 to 5 obtained by dividing one frame are sequentially processed from the top (in the order of slice 1, slice 2, slice 3, slice 4, slice 5) has been described, but the processing order is not limited to this. For example, it is also possible to preferentially execute processing from an area important to the user, such as near the fixation point on the screen, to reduce the perceived delay.

[0065] The importance of an area can be determined from the content displayed on the screen and the user's line-of-sight direction (or fixation point on the screen). For example, the image generation unit 201 of the server 20 determines the importance distribution within one frame based on the content to be displayed in one frame and the user's line-of-sight direction (or fixation point on the screen), and determines the division of the area and the drawing order. For example, the image generation unit 201 divides the display area into each slice centered on the fixation point being gazed at by the user within the screen, and further determines the drawing priority of each slice. The fixation point is obtained from the position and orientation information (such as the position and orientation information of the eyeballs) transmitted from the mobile terminal 10. FIG. 6 is a diagram for explaining the importance of each slice according to the fixation point in the present embodiment. For example, as shown in FIG. 6, the image generation unit 201 divides the display area into five slices centered on the fixation point P, and sets the importance of slice 3 including the fixation point P to be the highest.

[0066] (When the scanning line can be arbitrarily changed) When the display unit 160 of the mobile terminal 10 is a type of display where the scanning line can be arbitrarily changed, the image generation unit 201 sets a higher demand level for the vicinity of the fixation point P, and determines, for example, in the order of slice 3, slice 4, slice 2, slice 5, slice 1. Then, drawing, encoding, transmission / reception, decoding, image transformation, and display processing are performed in this order.

[0067] FIG. 7 is a diagram for explaining the display order of each slice in the prioritized processing according to the present embodiment. Since each process from drawing to display output according to the present embodiment is executed by pipeline processing, when the processing is performed in the above order, in the display unit 160, as shown in FIG. 7, the display processing is performed in the order of slice 3, slice 4, slice 2, slice 5, slice 1. In this case, it is possible to shorten the delay of the important area (the area of slice 3 in this example) compared to simply dividing and displaying from the upper part of the screen.

[0068] FIG. 8 is a diagram for explaining a series of processes that preferentially perform processing in slice units according to an arbitrary order according to the present embodiment. As shown in FIG. 8, for example, based on the level of importance according to the gaze point, drawing, encoding, reception, decoding, image transformation, and display output are performed in the order of slice 3, slice 4, slice 2, slice 5, and slice 1.

[0069] The communication unit 120 of the mobile terminal 10 receives the data of slice 3 (an example of the first display area) located in the area including the gaze point at the first time t1. Further, the communication unit 120 of the mobile terminal 10 receives the data of slice 4 (an example of the second display area) adjacent to slice 3 at the second time t2 after the first time. The display control unit 105 of the mobile terminal 10 starts the display output of the data of slice 3 (the first display area) to the display unit 160, and then controls to start the display output of the data of slice 4 (the second display area) as one frame together with the data of slice 1.

[0070] Also, as shown in FIG. 8, the communication unit 120 of the mobile terminal 10 receives the data of the third display area (here, for example, slice 1) corresponding to a part of one frame drawn by the server 20 at the third time t3 after the second time t2. The display control unit 105 of the mobile terminal 10 starts displaying the data of slice 3 (the first display area) at a time before the third time t3, and starts displaying the data of slice 1 (the third display area) after starting the display of the data of slice 4 (the second display area). In this way, the first display area starts to be displayed at least before the third time, and the mobile terminal 10 can start displaying without waiting for all the drawing and reception of one frame, and can shorten the delay time.

[0071] At this time, the delay from the acquisition (detection by the sensor) of the position and orientation information of the mobile terminal 10 to the display (display of slice 3 which is an important area) is, for example, about 28 ms (1.7 V) when assuming various numerical values described above with reference to FIG. 2, and the delay is further reduced compared to the example described with reference to FIG. 4. Also, for example, it can be shortened to about 10 ms (0.6 V) with Time Warp.

[0072] Note that the display control unit 105 of the mobile terminal 10 may shorten the hold time of each pixel in the display output in the case of a display of a type in which the scanning line can be arbitrarily changed. For example, in the example shown in FIG. 8, a duty ratio of 20% (= 3.3 ms) is assumed. Thereby, it becomes possible to make less visible the phenomenon called tearing that occurs at the boundary between slices. Tearing is a phenomenon in which the video shown on one screen looks as if it is shifted halfway (at the slice division line), and is also called screen tearing. Although it will be described in detail later, for example, when performing display output in the order as shown in FIG. 7, when drawing adjacent slices in the reverse order of the scanning direction, such as slice 2 displayed after slice 3, after ensuring the time for turning off the previously displayed slice 3 (shortening the hold time) and inserting black, slice 2 is lit, so that the tearing at the slice boundary can be made less visible.

[0073] FIG. 9 is a sequence diagram showing an example of the flow of display processing that preferentially performs processing in units of slices according to this embodiment in an arbitrary order. As shown in FIG. 9, first, the position and orientation estimation unit 101 of the mobile terminal 10 estimates the position and orientation of the user's head and eyes based on the sensing data of the sensor unit 110 (step S123), and transmits the estimated position and orientation information to the server 20 (step S126).

[0074] Next, the image generating unit 201 of the server 20 determines the divided areas and the drawing order (drawing order of slices) according to the importance (step S127). The importance may be calculated according to the point of interest on the screen identified from the position and orientation information transmitted from the mobile terminal 10, for example. Note that the information on the point of interest on the screen may be identified by the mobile terminal 10 and transmitted to the server 20.

[0075] Next, the image generation unit 201 performs image generation (rendering process) of the virtual object in slice units in the determined order based on the received position and orientation information (first operation information) (step S129). At this time, the image generation unit 201 may render the virtual object by referring to the position and orientation information prediction result for a predetermined time predicted by the position and orientation prediction unit 202 based on the received position and orientation information.

[0076] Next, the server 20 sequentially transmits the rendering result data in slice units to the mobile terminal 10 (step S132). Specifically, the server 20 sequentially encodes the rendering result data in slice units output from the image generation unit 201 using the data encoding unit 203, and transmits the encoded data from the communication unit 220 to the mobile terminal 10. The mobile terminal 10 sequentially decodes the received rendering result data in slice units. Note that all of the data is pipeline processed in a determined order.

[0077] Next, the image deformation unit 103 of the mobile terminal 10 deforms the image data (image data of the virtual object in slice units) included in the received rendering result in slice units based on the most recent position and orientation information (second operation information) (step S135). At this time, the image deformation unit 103 may perform the image deformation by referring to the position and orientation information prediction result for a predetermined time (expected display time) predicted by the position and orientation prediction unit 104 based on the most recent position and orientation information.

[0078] Then, the display control unit 105 of the mobile terminal 10 starts displaying the image data in units of slices after the image transformation (step S138). The server 20 continues to sequentially transmit the rendering result data in units of slices, and the mobile terminal 10 performs decoding, image transformation, and display processing in units of slices.

[0079] In this way, in this embodiment, when drawing, transmitting / receiving, transforming, and display output are pipelined on a slice-by-slice basis, drawing on a slice-by-slice basis begins in an order determined according to importance based on the gaze point, etc., so that slices with higher importance can be displayed first, and the display delay can be further reduced as perceived by the user.

[0080] (When the scan line cannot be changed arbitrarily) If the display unit 160 of the mobile terminal 10 is a type of display in which the scan line cannot be arbitrarily changed, the image generation unit 201 sets the slice of the area with the highest importance according to, for example, the gaze point P to the beginning of the rendering process. As a result, if, for example, slice 3 has the highest importance, it becomes possible to start display output with slice 3 at the beginning (priority) as shown in FIG. 10. The display process can then be performed in the order of slice 4, slice 5, slice 1, and slice 2. After slice 2, slice 3 of the next frame is displayed.

[0081] In this way, by giving priority to displaying slices with high importance, it is possible to further reduce the display delay perceived by the user. This will be specifically described below with reference to FIG.

[0082] FIG. 11 is a diagram for explaining a series of processes in which slices with high importance are processed first in slice units according to the present embodiment. As shown in FIG. 11, the server 20 performs rendering processing on the slices 3, 4, 5, 1, and 2 in this order, with the slice 3 having a high importance according to, for example, the gaze point, at the head, based on the position and orientation information of the eyes and head transmitted from the mobile terminal 10. Also, subsequent processes of encoding, receiving, decoding, image transformation, and display output are also pipeline-processed in the order of slice 3, slice 4, slice 5, slice 1, and slice 2. As a result, the delay from the acquisition (detection by the sensor) of the position and orientation information of the mobile terminal 10 to the display (display of slice 3, which is an important area) is, for example, about 30 ms (1.8 V) when assuming the various numerical values described above with reference to FIG. 2, and the delay is further reduced compared to the example described with reference to FIG. 4. Also, for example, it can be shortened to about 8.3 ms (0.5 V) by Time Warp.

[0083] Here, when any slice after slice 2 among the plurality of slices forming one frame is set at the head in the order of slice 3, slice 4, slice 5, slice 1, and slice 2, tearing may occur at the slice boundary. In particular, when performing rendering, transmission, and display processes in order from an area important to the user, such as near the gaze point, tearing may be easily noticeable. In the example shown in FIG. 11, since slice 3 is set at the head of the rendering process, there may be a case where tearing occurs at the boundary between slice 2 of the i-th frame i and slice 3 of the (i + 1)-th frame i+1 . In this case, although details will be described later, it is possible to make the tearing between slices less visible by deforming the image so that the outline of the virtual object straddling the boundary is seen continuously. In the example shown in FIG. 11, in the image deformation unit 103 of the mobile terminal 10, based on slice 2 of the i-th frame i and slice 3 of the (i + 1)-th frame i+1 , the slice 2 i is deformed so that the outline of the virtual object straddling the boundary of these slices is seen continuously.

[0084] Hereinafter, the suppression of tearing according to this embodiment will be specifically described.

[0085] <3-3. Suppression of Tearing> (3-3-1. Black Insertion Period) As a first countermeasure, there is control in which the display control unit 105 of the mobile terminal 10 shortens the lighting time of each pixel corresponding to the area of each slice (shortens the hold time). Thereby, when displaying a slice (the third slice) adjacent to the previously displayed slice (the first slice) in the direction opposite to the scanning order (for example, from the upper part to the lower part of the screen), a period for black insertion by turning off the light is secured. Specifically, when the display control unit 105 determines the display order, it is possible to create a period for black insertion by separating the update timings of adjacent slices.

[0086] For example, in the example shown in FIG. 8, when displaying slice 2 (the third slice) adjacent to slice 3 (the first slice) in the direction opposite to the scanning order, a period for black insertion is provided. Each slice to be displayed is subjected to image transformation by the image transformation unit 103 based on the latest position and orientation information and prediction for each slice. Therefore, for example, if image transformation and display are performed in the order of slice 3, slice 4, slice 2, slice 5, slice 1, the contour of the virtual object straddling the boundary between slice 2 and slice 3 may shift. In such a case, if there is a period during which slice 3 and slice 2 are displayed simultaneously, the shift of the contour may be conspicuous. Also, since slice 3 is an important area for the user, especially in the vicinity of the fixation point, tearing may be easily noticeable.

[0087] Therefore, as described above, by shortening the lighting time (hold time) of each pixel, for example, a period during which slice 3 and slice 2 are displayed simultaneously can be avoided, and a period for black insertion can be provided in between. Thereby, it becomes possible to make it difficult for the user to perceive tearing such as a shift in the contour of the virtual object, and it is possible to provide a more comfortable viewing experience for the user.

[0088] (3-3-2. Image Deformation) As a second countermeasure, the image deformation unit 103 of the mobile terminal 10 may deform the contour line of the virtual object. Specifically, based on the image data of two adjacent slices, the image deformation unit 103 performs image deformation so that the contour line of the virtual object straddling the boundary between these slices is continuous.

[0089] In particular, in the case of a display of a type where scanning lines cannot be arbitrarily selected as described with reference to FIG. 11, when an important slice determined according to the fixation point or the like is set at the head of the drawing process, the slice (for example, slice 2 i ) displayed at the end of the i-th frame and the slice (for example, slice 3 i+1 ) displayed at the beginning of the (i + 1)-th frame, tearing is likely to occur at the boundary therebetween. In this case, the image deformation unit 103 can reduce tearing by deforming the contour line of the virtual object of slice 2 i to be continuous with the contour line of slice 3 i+1 .

[0090] Also, the method of deforming the contour line of the virtual object described here may be applied when each slice is displayed in an arbitrary order according to the importance distribution according to the fixation point or the like in the case of a display of a type where scanning lines can be arbitrarily selected as described with reference to FIG. 8.

[0091] Hereinafter, a specific description will be given with reference to FIGS. 12 to 16.

[0092] FIG. 12 is a block diagram showing an example of the functional configuration of the image deformation unit 103 according to the present embodiment. As shown in FIG. 12, the image deformation unit 103 functions as a virtual object motion prediction deformation unit 1031, a position and orientation prediction deformation unit 1032, and an inter-slice contour deformation unit 1033. Note that the virtual object motion prediction deformation unit 1031, the position and orientation prediction deformation unit 1032, and the inter-slice contour deformation unit 1033 all perform deformation in units of slices.

[0093] First, the image deformation unit 103 receives the image data (RGB image) included in the drawing result transmitted from the server 20 in slice units, as well as the distance information and speed information of the virtual object included in the drawing result. Here, FIG. 13 shows an example of an image generated by the image generation unit 201 of the server 20 in slice units.

[0094] As shown in FIG. 13, the image generation unit 201 draws the i-th frame in slice units considering the movement of the user U and the movement of the virtual object 30, and then draws the (i + 1)-th frame in slice units considering the newly acquired movement of the user U and the movement of the virtual object 30. The movement of the user U can be considered by referring to the position and orientation information transmitted from the mobile terminal 10 as described above, and the position and orientation information at a predetermined time (display prediction time) predicted by the position and orientation prediction unit 202 based on the position and orientation information. Further, when drawing the virtual object 30 in slice units, the image generation unit 201 may consider the speed information of the virtual object 30 and the distance information from the user U to the virtual object 30.

[0095] In the example shown on the right side of FIG. 13, in order to reduce the display delay, slice 3 including the fixation point P is set at the head of the drawing process, and is drawn sequentially in slice units in the order of slice 3, slice 4, slice 5, slice 1, and slice 2. Also, when the image generation unit 201 performs sequential drawing for each slice, since it draws according to the movement of the virtual object 30 at the drawing time, the drawing position of the virtual object changes for each slice. And the drawing result in slice units includes the generated image (RGB image), and the distance information and speed information of the virtual object 50.

[0096] Based on the input speed information and distance information of the virtual object, the virtual object motion prediction deformation unit 1031 predicts the position of the virtual object at a predetermined time (display prediction time), and deforms the generated image in slice units so that the virtual object is displayed at the predicted position.

[0097] The position and orientation prediction deformation unit 1032 acquires the prediction result of the latest position and orientation information from the position and orientation prediction unit 104, and further deforms the generated image in slice units so that a virtual object is displayed at a position viewed from the viewpoint position after a predetermined time (display prediction time) based on the prediction result.

[0098] In this way, by predicting the position of the virtual object and the position and orientation of the user on the mobile terminal 10 side and performing image deformation, it is possible to minimize the display delay of the virtual object.

[0099] However, when preferentially drawing, transmitting-receiving, and displaying high-importance slices including the fixation point, as shown in FIG. 14, slice 2 displayed at the end of the i-th frame i and slice 3 displayed at the beginning of the (i + 1)-th frame i+1 there may be a tearing phenomenon between them. Such tearing may also occur due to prediction errors. In particular, since slice 3 i+1 is at the frame switching part, it is likely to cause tearing with the previous slice. Also, since slice 3 i+1 is a high-importance display area including the fixation point P, the tearing is likely to be noticeable. Also, in a display of a type where the scanning lines can be controlled in an arbitrary order, if the lighting time is lengthened to increase the luminance and no black insertion period is created, tearing may similarly occur.

[0100] Therefore, in the present embodiment, the slice-interval contour deformation unit 1033 performs a process of deforming the contour of the virtual object that straddles the slices so as to be continuous. FIG. 15 is a diagram for explaining the image deformation of the contour line of the virtual object according to the present embodiment.

[0101] First, when drawing slice 3 (of the (i + 1)-th frame) including the fixation point, the image generation unit 201 of the server 20 draws with the width of the slice widened slightly upward, and slice 2 (of the i-th frame) displayed immediately before i+1 ​i Create an overlapping area (hereinafter referred to as the overlapping area 52) with it. Next, the image deformation unit 103 of the mobile terminal 10 undergoes deformation based on the prediction of the virtual object's movement by the virtual object movement prediction deformation unit 1031 and deformation based on the prediction of the position and orientation by the position and orientation prediction deformation unit 1032, and then the inter-slice contour deformation unit 1033 deforms the contour line of the virtual object between slices. Specifically, image deformation is performed so that the deviation of the contour line of the virtual object in the overlapping area 52 shown in FIG. 15 is minimized (the slice 2 for which image deformation has been performed i’ is obtained). That is, the inter-slice contour deformation unit 1033 makes the contour line of the virtual object in the overlapping area 52 that overlaps with slice 3 i among slices 2 i+1 approach the contour line of the virtual object in slice 3 i+1 so that the contour line can be seen continuously between slice 2 i and slice 3 i+1 . The deformation of the contour line can be performed using, for example, a vertex set (polygon mesh).

[0102] FIG. 16 is a diagram for specifically explaining the deformation of the contour line of the virtual object according to the present embodiment. As shown in FIG. 16, first, the inter-slice contour deformation unit 1033 extracts the contour for the overlapping area 52 of adjacent slices. The overlapping area 52 of adjacent slices is, for example, the overlapping area 52 i in slice 2 i of the i-th frame and the overlapping area 52 i+1 in slice 3 i+1 of the (i + 1)-th frame.

[0103] Next, the inter-slice contour deformation unit 1033 finds corresponding points (corresponding vertices) between the extracted contours and makes each corresponding point into a polygon mesh. Then, the two polygon meshes (the polygon mesh of the overlapping area 52 i and the polygon mesh of the overlapping area 52 i+1 ) are combined. And the inter-slice contour deformation unit 1033 is the overlapping area 52 iApply the composite mesh to obtain an image with minimized contour deviation.

[0104] In the composition of two polygon meshes, in order to make the contour line appear continuous across slices, at the upper part of the overlapping area, the area of the slice located above (overlapping area 52 i ) and at the lower part, the area of the slice located below (overlapping area 52 i+1 ), it is necessary to select the control points of the mesh so as to approach them. More specifically, for example, the following implementation examples can be cited.

[0105]

Number

[0106] Note that the method of deforming the contour line of the virtual object spanning between the slices described above is just an example, and the present embodiment is not limited thereto.

[0107] (3-3-3. Adjustment of the division line) As a third countermeasure, the image generation unit 201 of the server 20 adjusts the division line that divides one frame into a plurality of regions. Since tearing is likely to be noticeable if there is a slice boundary (division line) near the user's fixation point, the image generation unit 201 adjusts each division line so that the fixation point is near the center of the slice (center in the width direction). Hereinafter, it will be described with reference to FIG. 17.

[0108] FIG. 17 is a diagram for explaining the adjustment of the division line that divides one frame into a plurality of slices according to the present embodiment. As shown on the left side of FIG. 17, the image generation unit 201 usually divides the height h of one frame into a predetermined number of slices sWhen dividing equally (so that the display time is equal), as shown in the center of FIG. 17, each dividing line is adjusted (in other words, the width of each slice is adjusted) so that the fixation point is near the center in the width direction (height direction) of slice S3. Further, considering that the fixation point may move slightly up and down, the image generation unit 201 may further reduce tearing by randomly varying the dividing line up and down for each frame as shown on the right side of FIG. 17 even when the fixation point does not move. That is, the image generation unit 201 changes the width (height) of the slice (the first display area) including the fixation point for each frame and makes it different at least between consecutive frames.

[0109] More specifically, for example, the y - coordinate y of the upper side of slice j (the j - th slice) including the fixation point s [j] is obtained by the following formula. However, j is an integer such that 1 ≦ j ≦ N.

[0110]

Equation

[0111] Next, as shown in the center of FIG. 17, when the y - coordinate of the fixation point is y gaze the y - coordinate y of the upper side of slice j in this case s [j] is obtained by the following formula. However, this is when j is an integer such that 2 ≦ j ≦ N, and when j = 1, y s [1]=0.

[0112]

Equation

[0113] Also, when randomly shifting the dividing line as shown on the right side of FIG. 17, a dither term is introduced as shown in the following formula to obtain y s [j]. However, this is when j is an integer such that 2 ≦ j ≦ N. Here, the dither term [i, j] is the j - th slice in the i - th frame. Also, when j = 1, y s [1]=0.

[0114]

number

[0115] <<4. Supplementary Information>> As described above, in the embodiment of the present disclosure, it is possible to reduce delays when receiving and displaying data of a virtual object generated by an external device.

[0116] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0117] For example, in the above-described embodiment, it has been described that the position and orientation estimation unit 101 estimates the head position and orientation and the eyeball position and orientation, but the present disclosure is not limited to this. For example, it may estimate only the head position and orientation and transmit the head position and orientation information to the server 20. Furthermore, in the above-described embodiment, it has been described that the position and orientation prediction unit 104 predicts the head position and orientation and the eyeball position and orientation, but the present disclosure is not limited to this. For example, it may predict only the head position and orientation and output the prediction result to the image deformation unit 103. The image deformation unit 103 may perform image deformation using only the head position and orientation information or the head position and orientation prediction result. Furthermore, the head position and orientation and the eyeball position and orientation are examples of user movement information, and the present disclosure is not limited to this.

[0118] It is also possible to create a computer program for causing hardware such as a CPU, ROM, and RAM built into the above-described mobile terminal 10 or server 20 to perform the functions of the mobile terminal 10 or server 20. A computer-readable storage medium storing the computer program is also provided.

[0119] Moreover, the effects described in this specification are illustrative or exemplary only and not limiting. That is, the technology according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or instead of the above effects.

[0120] Note that the present technology can also have the following configurations. (1) A transmission unit that transmits first operation information of a mobile terminal to an external device, receives data of a first display area of a virtual object drawn based on the first operation information by the external device at a first time, a receiving unit that receives data of a second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time; a display control unit that, after displaying the received data of the first display area, performs control to display the data of the second display area together with the data of the first display area as one frame on a display device of the mobile terminal; A display processing device comprising: (2) The receiving unit receives data of a third display area of the virtual object corresponding to a part of the one frame, drawn based on the first operation information by the external device, at a third time after the second time, The display control unit controls the display device to display the data of the first display area at a time before the third time and to display the data of the third display area after displaying the data of the second display area, the display processing device according to (1) above. (3) The display processing device further includes a deformation unit that deforms the received data of the second display area based on second operation information acquired after the first operation information, The display control unit performs control to display the deformed data of the second display area, the display processing device according to (1) or (2) above. (4) The first display area is a display area located in an area including a user's gaze point on the display device among the display areas obtained by dividing the area of one frame displayed on the display device, and is the display processing device according to (3) above. (5) The second display area is a display area located in an area closer to the gaze point than a third display area received by the receiving unit at a third time after the second display area, and is the display processing device according to (4) above. (6) The deformation unit deforms the outline of a virtual object that straddles the boundaries of adjacent display areas so that the outline is continuous among the plurality of display areas forming one frame displayed on the display device, and is the display processing device according to (4) or (5) above. (7) The display control unit is adjacent to the first display area on the side opposite to the second display area, and controls to display the data of the third display area received at a third time after the second time after the display of the data of the first display area ends after a certain period of time has elapsed, and is the display processing device according to any one of (4) to (6) above. (8) The first display area is generated such that the gaze point is located at the center in the width direction of the first display area, and is the display processing device according to any one of (4) to (7) above. (9) The width of the first display area is different between consecutive frames, and is the display processing device according to (8) above. (10) The first operation information includes information on the position and orientation detected by a motion sensor provided in the mobile terminal, and is the display processing device according to any one of (1) to (9) above. (11) The second operation information includes information on the position and orientation detected by a motion sensor provided in the mobile terminal, and the deformation unit deforms the data of the second display area based on a result predicted based on the second operation information, and is the display processing device according to (3) above. (12) The processor transmits the first operation information of the mobile terminal to an external device, receives data of a first display area of a virtual object drawn based on the first operation information by the external device at a first time, receives data of a second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time, and after displaying the received data of the first display area, performs control to display the data of the second display area together with the data of the first display area as one frame on a display device of the mobile terminal. A display processing method including the above. (13) A computer is caused to function as a transmission unit that transmits first operation information of a mobile terminal to an external device, a reception unit that receives data of a first display area of a virtual object drawn based on the first operation information by the external device at a first time, and receives data of a second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time, and a display control unit that performs control to display the data of the second display area together with the data of the first display area as one frame on a display device of the mobile terminal after displaying the received data of the first display area. A storage medium storing a program for causing the above to function. (14) A reception unit that receives first operation information of a mobile terminal, and based on the first operation information, transmits data of a first display area of a virtual object drawn as part of one frame to a display processing device that performs control to display the data of the first display area on a display device of the mobile terminal at a first time. After rendering the data of the first display area based on the first operation information, a transmission unit that transmits the data of the second display area adjacent to the first display area of the virtual object rendered as another part of the one frame to the display processing device at a second time after the first time An information processing apparatus comprising the same

Explanation of Signs

[0121] 10 Mobile terminal 100 Control unit 101 Position and orientation estimation unit 102 Data decoding unit 103 Image deformation unit 104 Position and orientation prediction unit 105 Display control unit 110 Sensor unit 120 Communication unit 130 Clock signal generation unit 140 Timestamp counter 150 Storage unit 160 Display unit 20 Server 200 Control unit 201 Image generation unit 202 Position and orientation prediction unit 203 Data encoding unit 220 Communication unit 230 Clock signal generation unit 240 Timestamp counter 250 Storage unit

Claims

1. A transmitting unit that transmits first operation information of a mobile terminal to an external device; receives data of a first display area of a virtual object drawn based on the first operation information by the external device at a first time; a receiving unit that receives data of a second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time; a display control unit that, after displaying the received data of the first display area, performs control to display the data of the second display area together with the data of the first display area as one frame on a display device of the mobile terminal; comprising: The first display area is a display area located in an area including a user's gaze point on the display device among the display areas obtained by dividing the area of one frame displayed on the display device, and is generated such that the gaze point is located at the center in the width direction of the first display area. A display processing device.

2. The receiving unit receives data of a third display area of the virtual object corresponding to a part of the one frame, drawn based on the first operation information by the external device, at a third time after the second time; The display control unit controls the display device to display the data of the first display area at a time before the third time and to display the data of the third display area after displaying the data of the second display area. The display processing device according to claim 1.

3. The display processing device further comprises: a deforming unit that deforms the received data of the second display area based on second operation information acquired after the first operation information; The display control unit performs control to display the deformed data of the second display area. The display processing device according to claim 1.

4. The second display area is a display area located closer to the gaze point than a third display area received by the receiving unit at a third time after the second display area. The display processing device according to claim 1.

5. The deforming unit deforms such that the contour line of the virtual object straddling the boundaries of adjacent display areas is continuous among a plurality of display areas forming one frame displayed on the display device. The display processing device according to claim 3.

6. The display control unit is adjacent to the first display area on the side opposite to the second display area, and controls to display the data of the third display area received at a third time after the second time after a lapse of a certain time after the display of the data of the first display area is finished. The display processing apparatus according to claim 1.

7. The width of the first display area is different between consecutive frames. The display processing apparatus according to claim 1.

8. The first operation information includes information on the position and orientation detected by a motion sensor provided in the mobile terminal. The display processing apparatus according to claim 1.

9. The second operation information includes information on the position and orientation detected by a motion sensor provided in the mobile terminal. The deformation unit deforms the data of the second display area based on a result predicted based on the second operation information. The display processing apparatus according to claim 3.

10. A processor transmits the first operation information of the mobile terminal to an external device; receives the data of the first display area of the virtual object drawn based on the first operation information by the external device at a first time; receives the data of the second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time; after displaying the received data of the first display area, controls to display the data of the second display area together with the data of the first display area as one frame on the display device of the mobile terminal; including The first display area is a display area located in an area including the user's gaze point on the display device among the display areas obtained by dividing the area of one frame displayed on the display device, and is generated such that the gaze point is located at the center in the width direction of the first display area. Display processing method.

11. A computer a transmission unit that transmits the first operation information of the mobile terminal to an external device; receives the data of the first display area of the virtual object drawn based on the first operation information by the external device at a first time, a receiving unit that receives the data of the second display area adjacent to the first display area of the virtual object drawn based on the first operation information by the external device at a second time after the first time A display control unit that, after displaying the received data of the first display area, performs control to display the data of the second display area together with the data of the first display area as one frame on the display device of the mobile terminal; A program for causing the mobile terminal to function as claimed in claim 1 is stored, The first display area is a display area located in an area including the user's gaze point on the display device among the display areas obtained by dividing the area of one frame displayed on the display device, and is generated such that the gaze point is located at the center in the width direction of the first display area. A computer-readable storage medium.

12. A receiving unit that receives first operation information of a mobile terminal; Based on the first operation information, data of a first display area of a virtual object drawn as a part of one frame is transmitted at a first time to a display processing device that performs control to display the data of the first display area on the display device of the mobile terminal; Based on the first operation information, after drawing the data of the first display area, data of a second display area adjacent to the first display area of the virtual object drawn as another part of the one frame is transmitted to the display processing device at a second time after the first time by a transmitting unit; Comprising: The first display area is a display area located in an area including the user's gaze point on the display device among the display areas obtained by dividing the area of one frame displayed on the display device, and is generated such that the gaze point is located at the center in the width direction of the first display area. An information processing device.

Images