Display processing device, display processing method, storage medium, and information processing device

Abstract

[Problem] To provide a display processing device, a display processing method, a storage medium, and an information processing device capable of reducing delay in a case where data of a virtual object created by an external device is received and displayed. [Solution] The display processing device includes: a transmission unit that transmits first operation information of a mobile terminal to an external device; a reception unit that receives, at a first time, data of a first display area of a virtual object rendered by the external device based on the first operation information, and receives, at a second time after the first time, data of a second display area of the virtual object rendered by the external device based on the first operation information, the second display area being adjacent to the first display area; and a display control unit that performs control to display the received data of the first display area on a display device of the mobile terminal, and then display the data of the second display area together with the data of the first display area as a single frame.

Description

Technical Field

[0001] This disclosure relates to display processing apparatus, display processing method, storage medium, and information processing apparatus. Background Technology

[0002] In recent years, various technologies have been developed for viewing real and virtual spaces in a fusion manner. For example, augmented reality (AR) and virtual reality (VR) technologies have been developed. In AR, images of virtual spaces (hereinafter referred to as virtual objects) are overlaid and displayed in the real space while the real space is being viewed directly. In VR, the generated virtual space is viewed from any viewpoint. Such AR and VR can be provided using, for example, head-mounted displays (HMDs) worn on the user's head. More specifically, examples of devices implementing AR include transmissive HMDs. Furthermore, examples of devices implementing VR include non-transmissive HMDs that utilize display units to cover the user's field of view to further enhance immersion.

[0003] Furthermore, regarding the technology of using HMD to display virtual objects, for example, the following Patent Document 1 describes that when a virtual object is superimposed and displayed in real space by an optically transmissive HMD, the position and orientation of the virtual object follow the movement of the user wearing the HMD or the object in real space, thereby reducing discomfort caused by the deviation of the display position of the virtual object from the real space.

[0004] Reference List

[0005] Patent documents

[0006] Patent Document 1: International Patent Application Publication No. 2019 / 181263 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] The aforementioned conventional techniques do not take into account latency in receiving and displaying data of virtual objects generated by external devices.

[0009] Therefore, this disclosure provides a display processing apparatus, display processing method, storage medium, and information processing apparatus that can reduce latency when receiving and displaying data of virtual objects generated by external devices.

[0010] Solution to the problem

[0011] According to this disclosure, a display processing apparatus is proposed, comprising: a transmitting unit that transmits first motion information of a mobile terminal to an external device; a receiving unit that receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area; and a display control unit that, after displaying the received data of the first display area, executes control to display the data of the second display area together with the data of the first display area as a frame on the display device of the mobile terminal.

[0012] According to this disclosure, a display processing method is proposed, comprising the following steps by a processor: sending first motion information of a mobile terminal to an external device; receiving data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time; and receiving data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area.

[0013] After displaying the received data in the first display area, control is executed to display the data in the second display area together with the data in the first display area as a single frame on the display device of the mobile terminal.

[0014] According to this disclosure, a storage medium for a stored program is proposed, the program being used by a computer as: a sending unit that sends first motion information of a mobile terminal to an external device; a receiving unit that receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, the second display area being adjacent to the first display area; and

[0015] The display control unit, after displaying the data received in the first display area, performs control to display the data in the second display area together with the data in the first display area as a frame on the display device of the mobile terminal.

[0016] According to this disclosure, an information processing apparatus is proposed, comprising: a receiving unit for receiving first motion information from a mobile terminal; and a sending unit for sending data of a first display area of ​​a virtual object, which is drawn as part of a frame based on the first motion information, to a display processing device at a first time, wherein the display processing device performs control to display the data of the first display area on the display device of the mobile terminal; and after the data of the first display area is drawn based on the first motion information, the sending unit sends data of a second display area of ​​a virtual object, which is drawn as part of a frame, to the display processing device at a second time later than the first time, wherein the second display area is adjacent to the first display area. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating an example of the basic configuration of an information processing apparatus according to one embodiment of the present disclosure.

[0018] Figure 2 It is a graph used to describe the length of delay when drawing processing is performed by an external device.

[0019] Figure 3 It is a diagram used to describe a slice in a scanned display.

[0020] Figure 4 This is a diagram illustrating the sequential flow of processing in units of slices, according to this embodiment.

[0021] Figure 5 This is a sequence diagram illustrating an example of a display processing flow of a display processing system in slices, according to this embodiment.

[0022] Figure 6 This is a diagram illustrating the importance of slices based on the fixation point, according to this embodiment.

[0023] Figure 7 This is a diagram illustrating the display order of slices in priority processing according to this embodiment.

[0024] Figure 8 This is a diagram illustrating the sequential flow of processing on a slice-by-slice basis, according to this embodiment.

[0025] Figure 9 This is a sequence diagram illustrating an example of a processing flow that prioritizes the execution of slice-based processes in any order, according to this embodiment.

[0026] Figure 10 This is a diagram illustrating another display order of slices in priority processing according to this embodiment.

[0027] Figure 11 This is a diagram illustrating the sequential flow of processing highly important slices at the beginning of a slice-based process, according to this embodiment.

[0028] Figure 12 This is a block diagram illustrating an example of the functional configuration of the image deformation unit according to this embodiment.

[0029] Figure 13 This is a diagram illustrating an example of an image generated by the image generation unit in slices, according to this embodiment.

[0030] Figure 14 This is an example diagram illustrating a tearing phenomenon that may occur during display.

[0031] Figure 15 This is a diagram illustrating the image deformation used to describe the outline of a virtual object according to this embodiment.

[0032] Figure 16 This is a diagram illustrating the deformation of the outline of a virtual object according to a specific description of this embodiment.

[0033] Figure 17 This is a diagram illustrating the adjustment of the dividing lines for dividing a frame into multiple slices according to this embodiment. Detailed Implementation

[0034] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals, and redundant descriptions will be omitted.

[0035] In addition, the descriptions will be given in the following order.

[0036] 1. Configuration of a display processing system according to one embodiment of the present disclosure

[0037] 1-1. Configuration Example of Mobile Terminal 10

[0038] 1-2. Configuration Example for Server 20

[0039] 2. Overview of the Problem

[0040] 3. Technical Features

[0041] 3-1. Processing by slice

[0042] 3-2. Prioritize processing

[0043] 3-3. Inhibit tearing

[0044] (3-3-1 Black Insertion Period)

[0045] (3-3-2. Image distortion)

[0046] (Adjustment of the 3-3-3 dividing line)

[0047] 4. Supplement

[0048] 1. <<Configuration of a display processing system according to one embodiment of this disclosure>>

[0049] Figure 1 This is a diagram illustrating the configuration of a display processing system according to one embodiment of the present disclosure. (See diagram for example.) Figure 1 As shown, a display processing system according to one embodiment of the present disclosure includes a mobile terminal 10 and a server 20.

[0050] Mobile terminal 10 and server 20 are communicatively connected via network 30 to send and receive data. In the case where server 20 is a cloud server, it is assumed that network 30 is the Internet, including wireless or wired networks (e.g., 5G (fifth-generation mobile communication system)). Furthermore, if server 20 exists in the same space as mobile terminal 10, such as a smartphone or laptop PC, it is assumed to be short-range wireless communication such as Wi-Fi (registered trademark) or wired communication (USB cable, etc.).

[0051] Mobile terminal 10 is an example of an information processing terminal that performs display control of virtual objects in VR or AR. Mobile terminal 10 is implemented as, for example, an HMD worn on a user's head, a smartphone, a tablet terminal, a wearable device such as an eyeglass display device, etc. Mobile terminal 10 receives display data of virtual objects from server 20 and performs control to display the data on display unit 160 of mobile terminal 10. Server 20 is an example of an information processing device that generates display data of virtual objects in VR or AR. Specifically, server 20 receives position and orientation information from mobile terminal 10, draws virtual objects based on the received position / orientation information, and sends the drawing results to mobile terminal 10.

[0052] Note that since image data (data representing the rendering results of virtual objects) is continuously transmitted from server 20 to mobile terminal 10, it can be assumed that a data link layer capable of securing the bandwidth is used. Furthermore, since the transmission from mobile terminal 10 to server 20 involves a small data volume and is largely affected by communication latency, a low-latency data link layer can be assumed. For example, if server 20 is a cloud server and 5G is used for network 30, enhanced mobile broadband (eMBB) can be used for transmission from server 20 to mobile terminal 10, and ultra-reliable and low-latency communication (URLLC) can be used. Additionally, if server 20 is a smartphone, laptop PC, etc., that can be carried by a user wearing mobile terminal 10 (e.g., HMD), Bluetooth (trademark) or Wi-Fi (trademark) can be used for network 30, or a wired connection such as USB or Gigabit Ethernet can be used.

[0053] <1-1. Configuration Example of Mobile Terminal 10>

[0054] like Figure 1 As shown, the mobile terminal 10 includes a control unit 100, a sensor unit 110, a communication unit 120, a clock signal generation unit 130, a timestamp counter 140, a storage unit 150, and a display unit 160.

[0055] The communication unit 120 is communicatively connected to the server 20, which is an external device, via the network 30 in a wired or wireless manner to send and receive data. For example, the communication unit 120 sends the location / orientation information of the mobile terminal 10. Furthermore, the communication unit 120 receives data from the server 20 showing the rendering results of virtual objects. The communication unit 120 can be, for example, a wired / wireless local area network (LAN), Wi-Fi, Bluetooth, a mobile communication network (LTE, 3G, 4G, or 5G), etc.

[0056] More specifically, for example, the communication unit 120 may have the following functions: under the control of the control unit 100, it may packetize the data to be sent according to the communication protocol and send the packetized data to the server 20. Furthermore, the communication unit 120 may have the function of transferring the packets sent according to the communication protocol to a delivery destination in the mobile terminal 10. In addition to receiving data of the rendering results of virtual objects from the server 20, the communication unit 120 may also receive clock signals for time synchronization.

[0057] Sensor unit 110 comprises various sensors that detect operational information of mobile terminal 10. For example, sensor unit 110 may be a camera device or an inertial measurement unit (IMU). More specifically, sensor unit 110 acquires sensing data for estimating the position and orientation of the user's head and the position and orientation of the eyes. For example, in the case where mobile terminal 10 is implemented by an HMD and worn on the user's head, the position and orientation of mobile terminal 10 are estimated as the position and orientation of the user's head. The position and orientation of the eyes are detected by sensors (camera device, infrared sensor, etc.) fixed within the HMD in the direction of the user's eyes.

[0058] The control unit 100 serves as an arithmetic processing device and a control device, and controls the overall operation of the mobile terminal 10 according to various programs. The control unit 100 is implemented by, for example, electronic circuitry (e.g., a central processing unit (CPU) or a microprocessor). Furthermore, the control unit 100 may include a read-only memory (ROM) for storing programs to be used, operating parameters, etc., and a random access memory (RAM) for temporarily storing parameters that change appropriately.

[0059] The control unit 100 according to this embodiment is also a display processing device that serves as a position / orientation estimation unit 101, a data decoding unit 102, an image deformation unit 103, a position / orientation prediction unit 104, and a display control unit 105. The position / orientation estimation unit 101 estimates the user's position and orientation based on various types of sensing data detected by the sensor unit 110. Specifically, the position / orientation estimation unit 101 has a so-called head tracker function, which estimates the user's head position and orientation based on images obtained by imaging the surrounding environment from a camera device and sensing data from the IMU acquired by the sensor unit 110. The head position and orientation can be position and orientation in a coordinate system defined relative to the surrounding real space. Note that the head tracker includes an outside-in method using externally mounted sensors, an inside-out method using sensors mounted on the measurement target itself (mobile terminal 10 (HMD)), and a hybrid method combining these. Furthermore, the position / orientation estimation unit 101 also functions as an eye tracker, which estimates the position and orientation of the user's eyeballs (position of the pupil, iris, etc.) based on an infrared camera image obtained by imaging the user's eyes acquired by the sensor unit 110. The position and orientation of the user's eyeballs is a form of gaze information, and the eye tracker can also estimate the gaze point (coordinates) on the display screen of the display unit 160. The sensor unit 110 continuously performs sensing, and the control unit 100 continuously transmits the position / orientation information estimated by the position / orientation estimation unit 101 (an example of first motion information) from the communication unit 120 to the server 20. On the server 20 side, the image generation unit 201, described later, performs processing to draw virtual objects based on the position / orientation information (first motion information) received from the mobile terminal 10, and sends the drawing result data to the mobile terminal 10. The data to be transmitted is encoded by the data encoding unit 203, described later.

[0060] Data decoding unit 102 decodes the rendering result data (including image data of virtual objects) sent from server 20 and outputs the decoded data to image warping unit 103. Image warping unit 103 receives the output from data decoding unit 102 (including image data of virtual objects) and the prediction result based on the latest position / orientation information (an example of second motion information) from position / orientation prediction unit 104 as input, and performs image warping so that when the image of the virtual object is displayed on display unit 160, the virtual object appears to be positioned in a more accurate location. On the mobile terminal 10 side, delay compensation can be performed by using the prediction result of position / orientation information to perform image warping on the image data of the virtual object received from server 20. The result of image warping is output as a pixel column to display unit 160 implemented by the scanned display. Details of image warping will be described later.

[0061] Position / orientation prediction unit 104 predicts position / orientation information after a predetermined time based on the latest position / orientation information (an example of second motion information) estimated by position / orientation estimation unit 101. "After a predetermined time" is assumed to be the predicted display time of the slice to be subjected to image deformation on display unit 160. For example, position / orientation prediction unit 104 predicts position / orientation information after a predetermined time based on the latest position / orientation information estimated by position / orientation estimation unit 101 at time t. h Estimated head position / orientation information w T h (t h To calculate head position / orientation information at a future time t. w T h (t). Furthermore, the position / orientation prediction unit 104 is based on the position / orientation estimation unit 101 at time t. e Estimated eye position / orientation information h T e (t e To calculate eye position / orientation information at future time t. h T e (t).

[0062] The display control unit 105 performs control to display image data appropriately distorted by the image warping unit 103 on the display unit 160. Details of the display processing will be described later.

[0063] The clock signal generation unit 130 uses the clock propagation mechanism in network 30 to acquire the clock signal generated by the clock signal generation unit 230 of server 20 (a clock signal used as a reference for timing control of the entire system), and references this clock signal to generate a clock signal synchronized with server 20 in the phase-locked loop (PLL). Note that regardless of whether a wired or wireless communication connection is used between mobile terminal 10 and server 20, devices for sending and receiving the reference clock signal are provided in the data link layer of network 30. For example, in the case of a wireless connection, any of LTE, 5G, Bluetooth, and Wi-Fi can be used, and communication can be controlled by defining time slots with a time axis resolution of 1 ms or less, enabling clock synchronization using the server 20 side as a reference clock. Furthermore, in the case of a wired connection, in systems such as USB where the main controller performs timing control of the entire bus, clock synchronization using server 20 as a reference clock can be performed. Additionally, in the case of Gigabit Ethernet (GbE), sufficiently accurate synchronization (between the timestamp counters of mobile terminal 10 and server 20) can be achieved by combining the use of the Precision Time Protocol (PTP).

[0064] Timestamp counter 140 is a time counter used for timing control of the entire system. Except for special cases such as initialization, timestamp counter 140 can, in principle, monotonically increase. During system initialization, after initializing the timestamp counter 140 of the mobile terminal 10 according to the value of timestamp counter 240 of the server 20, the counter is advanced using a reference clock signal generated by clock signal generation unit 230 or clock signal generation unit 130. The values ​​of timestamp counter 140 and timestamp counter 240 are synchronized with each other with sufficient accuracy (at least an error of 1 ms or less). For example, when the control unit 100 acquires sensing data from the sensor unit 110, it references the value of timestamp counter 140 and uses the value of timestamp counter 140 as the timestamp of the sensing data. Furthermore, regarding the display control of the display unit 160 implemented by the scanned display, the timestamp counter value (typically referred to as Vsync (vertical synchronization signal)) is referenced at the start timing of each frame's scan and used to estimate the next planned VSync time.

[0065] Note that the above-described timed synchronization methods and their usage are merely examples, and this disclosure is not limited thereto.

[0066] The storage unit 150 is implemented by a read-only memory (ROM) for storing programs, operating parameters and the like for processing the control unit 100, and a random access memory (RAM) for temporarily storing parameters and the like that which change appropriately.

[0067] The display unit 160 is a display device for displaying an image of a virtual object. Furthermore, according to this embodiment, the display unit 160 is implemented by a scanned display. The scanned display sequentially displays multiple slices (display areas) divided in a direction perpendicular to the scanning direction. That is, the scanned display has multiple adjacent display areas, and these multiple adjacent display areas have different display timings (the light-up time varies depending on the position of the screen). This will be referred to later. Figure 3 The slice is described. Furthermore, more specifically, the scanned display shows the pixel column output from the image warping unit 103. For example, assume the following two types are the types of the scanned display.

[0068] One type allows for the arbitrary selection of lines to be scanned (i.e., the scanning order can be arbitrarily changed). Specific examples of displays include organic light-emitting diode (OLED) displays and displays using LEDs. In this type, the light-emitting time (hold time) of each pixel can also be varied. Another type does not allow for the arbitrary selection of lines to be scanned (i.e., the scanning order cannot be arbitrarily changed). Specific examples of displays include laser scanning (LBS) displays that combine a laser light source and MEMS mirrors.

[0069] Furthermore, the display unit 160 implemented by the scanned display can be an optically transparent display capable of directly delivering light from real space to the user's eyes, or it can be a non-transmissive display in which light from real space does not directly reach the user's eyes. Additionally, the display unit 160 can be a display capable of switching between an optically transparent display and a non-transmissive display. Furthermore, the display unit 160 can be disposed in an HMD, or in a smartphone, tablet terminal, etc. Furthermore, the display unit 160 can be disposed in a separate device communicatively connected to the mobile terminal 10.

[0070] Although the configuration of the mobile terminal 10 has been described in detail above, the configuration of the mobile terminal 10 according to this disclosure is not limited to... Figure 1 The example shown. For example, mobile terminal 10 can be implemented by multiple devices. Specifically, it may include a display device implemented by HMD or the like (corresponding to at least display unit 160) and an information processing terminal implemented by smartphone, tablet terminal, PC or the like (corresponding to at least display processing device of control unit 100).

[0071] <1-2. Configuration Example for Server 20>

[0072] like Figure 1 As shown, server 20 includes control unit 200, communication unit 220, clock signal generation unit 230, timestamp counter 240 and storage unit 250.

[0073] Communication unit 220 is communicatively connected to mobile terminal 10 via network 30 in a wired or wireless manner to send and receive data. For example, communication unit 220 receives location / orientation information from mobile terminal 10. Furthermore, communication unit 220 sends data representing the rendering results of virtual objects generated by control unit 200 to mobile terminal 10. Communication unit 220 can be, for example, a wired / wireless local area network (LAN), Wi-Fi, Bluetooth, a mobile communication network (LTE, 3G, 4G, or 5G), etc.

[0074] The control unit 200 serves as an arithmetic processing and control device, and controls the overall operation of the server 20 according to various programs. The control unit 200 is implemented, for example, by electronic circuitry (e.g., a central processing unit (CPU) or a microprocessor). Furthermore, the control unit 200 may include a read-only memory (ROM) storing programs to be used, operating parameters, etc., and a random access memory (RAM) temporarily storing appropriately changing parameters, etc.

[0075] Furthermore, the control unit 200 according to this embodiment also functions as an image generation unit 201, a position / orientation prediction unit 202, and a data encoding unit 203. The image generation unit 201 has the following functions: drawing virtual objects based on position / orientation information (head position / orientation information and eye position / orientation information) received from the mobile terminal 10, and outputting the virtual objects as images. In the case of VR, the image generation unit 201 draws a virtual space viewed from a viewpoint position (the user's viewpoint in the virtual space) corresponding to the position / orientation information. Furthermore, the drawing result output from the image generation unit 201 may include, in addition to an RGB image, distance information and speed information of the virtual objects. Furthermore, the image generation unit 201 may draw virtual objects based on the latest position / orientation prediction information from the position / orientation prediction unit 202. The position / orientation prediction unit 202 predicts position / orientation information after a predetermined time based on the position / orientation information received from the mobile terminal 10. Assume that "after a predetermined time" is the predicted display time on the mobile terminal 10. It can be said that the drawing process based on position / orientation prediction information is one form of delay compensation.

[0076] The data encoding unit 203 encodes the rendering result output from the image generation unit 201. The control unit 200 transmits the data encoded by the data encoding unit 203 from the communication unit 220 to the mobile terminal 10.

[0077] The clock signal generation unit 230 has the function of generating a clock signal that serves as a reference for timing control of the entire system. The server 20 sends the clock signal generated by the clock signal generation unit 230 from the communication unit 220 to the mobile terminal 10.

[0078] The timestamp counter 240 is a time counter used for timing control of the entire system. The timestamp counter 240 uses a reference clock signal generated by the clock signal generation unit 230 to advance the counter.

[0079] The storage unit 250 is implemented by a read-only memory (ROM) for storing programs, operating parameters, etc. for processing by the control unit 200, and a random access memory (RAM) for temporarily storing parameters that change appropriately.

[0080] Although the configuration of server 20 has been described in detail above, the configuration of server 20 according to this disclosure is not limited to... Figure 1 The example shown. For example, server 20 can be implemented by multiple devices.

[0081] 2. <<Overview of the Problem>>

[0082] Here, since higher-resolution rendering increases power consumption and leads to an increase in the size of the heat dissipation mechanism and battery capacity of the terminal performing the rendering process, as in this embodiment, the size and weight of the mobile terminal 10 can be reduced by having the server 20 render the virtual objects. However, when the rendering of virtual objects is performed by an external device such as the server 20, compared to the case where the processing is performed solely by the mobile terminal 10, latency increases due to the compression and expansion of communication or images, and there is a possibility that the positioning of virtual objects may be compromised when displayed on the mobile terminal 10.

[0083] Figure 2 This is a graph used to describe the length of the delay when drawing processing is performed by an external device. Figure 2In the example shown, firstly, the position and orientation of the user's eyes and head are estimated using various sensors (eye trackers and head trackers) placed in a mobile terminal, such as an HMD worn on the user's head, and the position / orientation information is sent to a server (uplink transmission) as an external device. Based on the received position / orientation information (rendering function), the server performs virtual object rendering processing (i.e., virtual object image generation), encodes (compresses) the virtual object image (performs video encoding), and sends the data to the mobile terminal. When data is received (downlink transmission), the mobile terminal decodes (decompresses) the encoded data (performs video decoding). At this point, a delay has occurred due to communication and data encoding and decoding (the elapsed time from the time the position / orientation information used to render the virtual object). Since time has elapsed since the acquisition of the position / orientation information for rendering the virtual object, movement on the mobile terminal side during this time may impair the perceived positioning of the virtual object displayed on the mobile terminal. That is, in the case of AR, the display position and orientation of the virtual objects to be overlaid and displayed are offset from their display position and orientation in real space, and even when using a video perspective display, the display position and orientation of the virtual objects to be overlaid and displayed are offset from their display position and orientation in the video in real space. Furthermore, in the case of VR, the video in the virtual space does not follow (or follows with a delay) the user's movement, resulting in discomfort or unnaturalness, thereby impairing the immersion in the virtual space.

[0084] Note that, assuming the display unit included in the mobile terminal is the scanned display, the mobile terminal sequentially displays multiple slices (display areas) divided in a direction perpendicular to the scanning direction. Figure 3 It is a diagram used to describe a slice of the screen being scanned. For example... Figure 3As shown, the scanned display is divided into, for example, five slices (display areas) in a direction perpendicular to the scanning direction (x-axis direction in the figure) (y-axis direction in the figure). The boundaries between the slices are set to be substantially parallel to the scanning direction. The slices undergo display processing in the scanning sequence. That is, the scanned display has multiple adjacent display areas, and these multiple adjacent display areas have different display timings (the emission time varies depending on the screen position). For example, display processing is performed sequentially from slice 1 (S1) to slice 5 (S5). The mobile terminal divides the data of the rendering results of virtual objects received from the server into five slices, scans the slices sequentially, and displays and outputs the data. At this time, the mobile terminal can reduce the display delay perceived by the user by performing image warping on each slice based on the position / orientation information of the mobile terminal obtained immediately before the display output of each slice. Examples of image warping include time warping and distortion correction.

[0085] However, when rendering is performed by an external device, the latency caused by communication and data encoding and decoding is large, and the latency compensation for image distortion at the display output is insufficient. For example, assuming that eye position / orientation estimation (eye tracker) operates at 300 frames per second (fps) and the processing time is 3.33ms, assuming that head position / orientation estimation (head tracker) takes 2ms to process, assuming that the communication latency (uplink latency) of the mobile terminal when sending to the server is 1ms for 5G URLLC, assuming that the communication latency (downlink latency) of the mobile terminal when receiving from the server is 4ms for 5G eMBB, assuming that the processing latency of compression, network transmission and expansion at a throughput of 60fps is 16.7ms, and assuming that the Vsync period of the scanned display is 16.7ms, the latency from acquiring the position / orientation information of the mobile terminal (detected by the sensor) to display (motion photon latency) is approximately 95ms (5.7V). Note that the "display" time in the phrase "to display" is based on the display timing of a representative point, for example, using the center point of a frame as the representative point (corresponding to the center of slice 3 in the case where the scanned display is equally divided into five parts). Figure 2 In the example shown, the drawing result data is drawn and sent in units of one frame, and although the time can be shortened to about 18ms (1.1V) by time warping, the compensation period (about 95ms (5.7V) in this example) is long and distortion may occur.

[0086] Therefore, this disclosure proposes a display processing system capable of reducing latency when receiving and displaying data of virtual objects generated by an external device.

[0087] The technical features of a display processing system according to one embodiment of the present disclosure will be described below.

[0088] 3. <<Technical Features>>

[0089] The display processing system according to this embodiment has Figure 1 The configuration shown implements a mechanism for more appropriately reducing latency when the mobile terminal 10 receives and displays data of virtual objects generated by the server 20, which is an external device. Specifically, in the image generation unit 201 of the server 20, a frame is divided into multiple display areas (referred to as slices in this specification), drawing is performed for each slice, and the slices are sequentially compressed and sent at the time points when the drawing of each slice is completed. That is, although in Figure 2 The example shown performs drawing, encoding, and transmission on a frame-by-frame basis, but in the display processing system according to this embodiment, drawing, encoding, and transmission are performed on a slice-by-slice basis. Furthermore, in the mobile terminal 10, decoding, image warping, and display output are also performed sequentially on a slice-by-slice basis. As described above, in the display processing system according to this embodiment, by performing pipelined processing for drawing, transmission, warping, and display output on a slice-by-slice basis, the delay time (motion photon delay) from acquiring position / orientation information to display can be shortened. In the following text, reference will be made to... Figure 4 Provide a detailed description.

[0090] <3-1. Processing on a slice-by-slice basis>

[0091] Figure 4 This is a diagram illustrating the sequential flow of processing on a slice-by-slice basis according to this embodiment. Figure 4 As shown, in the display processing system according to this embodiment, during image generation on server 20 (virtual object drawing processing in image generation unit 201), a frame is divided into multiple slices for sequential drawing, sequential encoding and transmission (received by mobile terminal 10) are performed on a slice-by-slice basis, and image deformation and display output on a slice-by-slice basis are also performed on mobile terminal 10. The number and size of the slices to be divided are not particularly limited. As an example, image generation unit 201 of server 20 can divide a frame equally. For example, as... Figure 3 As shown, the image generation unit 201 divides a frame into five display areas, from slice 1 to slice 5, and performs drawing sequentially starting from slice 1.

[0092] Note that the image generation unit 201 refers to the position information prediction result predicted by the position / orientation prediction unit 202 based on the position / orientation information (an example of the first motion information) received from the mobile terminal 10, and draws a virtual object for each slice.

[0093] The communication unit 120 of the mobile terminal 10 receives data from slice 1 (first display area) at a first time t1. Furthermore, the communication unit 120 of the mobile terminal 10 receives data from slice 2 (second display area), which is adjacent to slice 1 (first display area), at a second time t2, which is later than the first time. After starting to output the data of slice 1 (first display area) to the display unit 160, the display control unit 105 of the mobile terminal 10 executes control to start the display output of data from slice 2 (second display area), combining it with the data from slice 1 as a single frame.

[0094] In addition, such as Figure 4 As shown, the communication unit 120 of the mobile terminal 10 receives data of a third display area (e.g., slice 5) corresponding to a portion of a frame drawn by the server 20 at a third time t3, which is later than the second time t2. The display control unit 105 of the mobile terminal 10 starts displaying data of slice 1 (first display area) at a time earlier than the third time t3, and also starts displaying data of slice 5 (third display area) after starting to display data of slice 2 (second display area). In this way, the first display area starts to be displayed at least before the third time, and the mobile terminal 10 can start displaying without waiting for the entire drawing or reception of a frame. Therefore, the delay time can be shortened. Here, although "slice 5" is described as "third display area" as an example, this embodiment is not limited to this. "Third display area" corresponds to a display area different from the first display area and the second display area, which correspond to a portion of a frame drawn by the server 20, and the second display area is adjacent to the first display area. For example, in Figure 4 In the example shown, the "third display area" can be "slice 4" or "slice 3". Furthermore, although in this example, the first time to the third time refers to the time when the drawing result data is received from server 20, this embodiment is not limited to this. For example, the first time to the third time could be the time when server 20 sends the drawing result data to mobile terminal 10, the time when mobile terminal 10 decodes the drawing result data received from server 20, or the time of image deformation.

[0095] Note that the image generation unit 201 of server 20 can draw virtual objects for each slice based on the position information prediction result predicted by the position / orientation prediction unit 202 based on the position / orientation information (an example of first motion information) received from mobile terminal 10. Furthermore, the drawing result of image generation unit 201 on a slice-by-slice basis includes an RGB image, distance information of the virtual object, and velocity information of the virtual object. The distance information and velocity information of the virtual object can be used for image deformation (one of the delay compensation methods) performed by image deformation unit 103 in mobile terminal 10, as described later.

[0096] Furthermore, in the image warping performed by the image warping unit 103 of the mobile terminal 10, the display delay perceived by the user can be reduced by referring to the previously acquired position / orientation information (an example of second motion information) and the prediction result based on the position / orientation information and performing image warping for each slice. Examples of image warping include temporal warp and distortion correction.

[0097] When applying more specific numerical values ​​to reduce latency by performing pipelined processing on a slice-by-slice basis, for example, assuming the above reference... Figure 2 Under the various numerical conditions described, the delay from acquiring the location / orientation information of the mobile terminal 10 (detected by a sensor) to display is approximately 42 ms (2.7V). In this case, even if the delay is reduced to approximately 18 ms (1.1V) through time warping, for example, the compensation period is still significantly longer than expected. Figure 2 The compensation period in the example shown is short (approximately 42ms (2.7V) in this example), so almost no distortion occurs. That is, it can also be said that when the period shortened by image deformation relative to the compensation period exceeds a predetermined value, distortion is unlikely to occur.

[0098] Figure 5 This is a sequence diagram illustrating an example of a slice-based display processing flow of a display processing system according to this embodiment. Figure 5 As shown, firstly, the position / orientation estimation unit 101 of the mobile terminal 10 estimates the position and orientation of the user's head and eyes based on the sensing data of the sensor unit 110 (step S103), and sends the estimated position / orientation information to the server 20 (step S106).

[0099] Next, the image generation unit 201 of server 20 performs image generation (drawing processing) of the virtual object in slices based on the received position / orientation information (first motion information) (step S109). At this time, the image generation unit 201 can draw the virtual object with reference to the position / orientation information prediction result of the position / orientation prediction unit 202 based on the received position / orientation information at a predetermined time.

[0100] Next, server 20 sequentially sends the drawing results data, in units of slices, to mobile terminal 10 (step S112). Specifically, server 20 sequentially encodes the drawing results data, in units of slices, output from image generation unit 201 via data encoding unit 203, and sends the encoded data from communication unit 220 to mobile terminal 10. Mobile terminal 10 sequentially decodes the received drawing results data, in units of slices.

[0101] Next, the image deformation unit 103 of the mobile terminal 10 deforms the image data (image data of virtual objects in slices) included in the received slice-based rendering results based on the latest position / orientation information (second motion information) (step S115). At this time, the image deformation unit 103 can perform image deformation with reference to the position / orientation information prediction result of the position / orientation prediction unit 104 based on the latest position / orientation information for a predetermined time (planned display time).

[0102] Then, the display control unit 105 of the mobile terminal 10 begins to display image data in slices that have undergone image distortion (step S118). Data of the drawing results in slices is sequentially sent from the server 20, and the mobile terminal 10 performs decoding, image distortion, and display processing in slices.

[0103] As described above, in this embodiment, by performing pipelined processing for drawing, sending / receiving, deforming, and displaying output on a slice-by-slice basis, the delay time from acquiring position / orientation information to display can be shortened.

[0104] <3-2. Prioritize processing>

[0105] While the above example describes processing slices 1 through 5 obtained by dividing a frame into segments sequentially from top to bottom (in the order of slice 1, slice 2, slice 3, slice 4, and slice 5), the processing order is not limited to this. For example, perceptual latency can also be reduced by prioritizing processing from areas that are important to the user (such as near the point of gaze on the screen).

[0106] The importance of a region can be determined based on the content displayed on the screen or the user's gaze direction (or the gaze point on the screen). For example, the image generation unit 201 of server 20 obtains the importance distribution in a frame based on the content to be displayed in a frame or the user's gaze direction (or the gaze point on the screen), and determines the division and rendering order of regions. For example, the image generation unit 201 divides the display region into slices around the gaze point where the user is looking on the screen, and also determines the rendering priority of each slice. The gaze point is obtained based on position / orientation information (e.g., eye position / orientation information) sent from mobile terminal 10. Figure 6 This is a diagram according to this embodiment used to describe the importance of slices based on the fixation point. For example, such as... Figure 6 As shown, the image generation unit 201 divides the display area into five slices around the gaze point P, and sets the importance of slice 3, which includes the gaze point P, to the highest.

[0107] (The scan lines can be changed arbitrarily)

[0108] In the case that the display unit 160 of the mobile terminal 10 is a display capable of arbitrarily changing the type of scan lines, the image generation unit 201 sets the demand to be higher than the gaze point, and determines the order of the demand as, for example, slice 3, slice 4, slice 2, slice 5, and slice 1. Then, drawing, encoding, sending / receiving, decoding, image deformation, and display processing are performed in this order.

[0109] Figure 7 This is a diagram illustrating the display order of slices in priority processing according to this embodiment. Since each process from drawing to display output according to this embodiment is executed via pipelined processing, when processes are executed in the above order, as... Figure 7 As shown, the display unit 160 performs display processing in the order of slice 3, slice 4, slice 2, slice 5, and slice 1. In this case, compared to simply dividing and displaying areas from the top of the screen, the delay of important areas (the area of ​​slice 3 in this example) can be shortened.

[0110] Figure 8 This is a diagram illustrating the sequential flow of processes performed in any order, on a slice-by-slice basis, according to this embodiment. Figure 8 As shown, for example, based on the importance of the gaze point, drawing, encoding, receiving, decoding, image deformation, and display output are performed in the order of slice 3, slice 4, slice 2, slice 5, and slice 1.

[0111] The communication unit 120 of the mobile terminal 10 receives data from slice 3 (an example of a first display area) located in the region including the gaze point at a first time t1. Furthermore, the communication unit 120 of the mobile terminal 10 receives data from slice 4 (an example of a second display area) adjacent to slice 3 at a second time t2, which is later than the first time. After starting to output data of slice 3 (the first display area) to the display unit 160, the display control unit 105 of the mobile terminal 10 executes control to start displaying the data of slice 4 (the second display area), combining it with the data of slice 3 as a single frame.

[0112] In addition, such as Figure 8 As shown, the communication unit 120 of the mobile terminal 10 receives data from the server 20 at a third time t3, which is later than the second time t2, representing a third display area (here, for example, slice 1) corresponding to a portion of a frame. The display control unit 105 of the mobile terminal 10 begins displaying the data of slice 3 (the first display area) at a time earlier than the third time t3, and also begins displaying the data of slice 1 (the third display area) after beginning to display the data of slice 4 (the second display area). In this way, the first display area begins to be displayed at least before the third time, and the mobile terminal 10 can begin displaying without waiting for the entire drawing or reception of a frame. Therefore, the delay time can be shortened.

[0113] At this point, assuming the above reference... Figure 2 Under the various numerical conditions described, the delay from acquiring the location / orientation information of the mobile terminal 10 (detected by a sensor) to display (as a slice 3 of the important area) is, for example, approximately 28 ms (1.7V), and this delay is even greater than the reference delay. Figure 4 The example described has a shorter delay. Furthermore, for example, the delay can also be reduced to approximately 10 ms (0.6V) through time warping.

[0114] Note that the display control unit 105 of the mobile terminal 10 can shorten the retention time of each pixel in the display output in cases where the scan lines of this type of display can be arbitrarily changed. For example, in Figure 8 In the example shown, the duty cycle is assumed to be 20% (= 3.3ms). This makes the phenomenon known as tearing less visible; this tearing occurs at the boundaries between slices. Tearing is the phenomenon where a video image appearing on a screen seems to be shifted in the middle (at the slice dividing lines), and is also known as screen tearing. As will be described in detail later, for example, in accordance with... Figure 7When the display output is executed in the order shown, when adjacent slices are drawn in the opposite order to the scanning direction (e.g., slice 2 is displayed after slice 3), the tearing at the slice boundary can be made less noticeable by turning off the earlier displayed slice 3 (shortening the holding time) to ensure the black insertion time, and then turning on slice 2.

[0115] Figure 9 This is a sequence diagram illustrating an example of a processing flow according to this embodiment, showing the execution of slice-based processes in any order. Figure 9 As shown, firstly, the position / orientation estimation unit 101 of the mobile terminal 10 estimates the position and orientation of the user's head and eyes based on the sensing data of the sensor unit 110 (step S123), and sends the estimated position / orientation information to the server 20 (step S126).

[0116] Next, the image generation unit 201 of server 20 determines the region division and drawing order (slice drawing order) based on importance (step S127). For example, importance can be obtained based on the gaze points on the screen identified from the position / orientation information sent from mobile terminal 10. Note that information about the gaze points on the screen can be identified by mobile terminal 10 and sent to server 20.

[0117] Subsequently, the image generation unit 201 performs image generation (drawing process) of the virtual object in slices according to the determined order based on the received position / orientation information (first motion information) (step S129). At this time, the image generation unit 201 can draw the virtual object with reference to the position / orientation information prediction result of the position / orientation prediction unit 202 based on the received position / orientation information at a predetermined time.

[0118] Next, server 20 sequentially sends the drawing results data, in slice units, to mobile terminal 10 (step S132). Specifically, server 20 sequentially encodes the drawing results data, in slice units, output from image generation unit 201 via data encoding unit 203, and sends the encoded data from communication unit 220 to mobile terminal 10. Mobile terminal 10 sequentially decodes the received drawing results data, in slice units. Note that the pipelined processing executes the operations in a predetermined order.

[0119] Next, the image deformation unit 103 of the mobile terminal 10 deforms the image data (image data of virtual objects in slices) included in the received slice-based rendering results based on the latest position / orientation information (second motion information) (step S135). At this time, the image deformation unit 103 can perform image deformation with reference to the position / orientation information prediction result of the position / orientation prediction unit 104 based on the latest position / orientation information for a predetermined time (planned display time).

[0120] Then, the display control unit 105 of the mobile terminal 10 begins to display image data in slices that have undergone image distortion (step S138). Data of the drawing results in slices is sequentially sent from the server 20, and the mobile terminal 10 performs decoding, image distortion, and display processing in slices.

[0121] As described above, in this embodiment, when pipelined processing of drawing, sending / receiving, deformation, and display output is performed on a slice-by-slice basis, by starting the drawing on a slice-by-slice basis in an order determined based on the importance of the viewpoint, etc., slices with high importance can be displayed first, and the user-perceived display delay can be further reduced.

[0122] (The scan lines cannot be changed arbitrarily)

[0123] In the case where the display unit 160 of the mobile terminal 10 is a display that cannot arbitrarily change the type of scan lines, the image generation unit 201, for example, sets a slice with the highest importance according to the gaze point P as the beginning of the drawing process. Therefore, for example, in the case where slice 3 has the highest importance, such as... Figure 10 As shown, output slice 3 can be displayed at the beginning (preferred). Then, display processing can be performed in the order of slice 4, slice 5, slice 1, and slice 2. After slice 2, slice 3 of the next frame is displayed.

[0124] In this way, by prioritizing the display of highly important slices, the perceived display latency can be further reduced. This will be discussed in the following text. Figure 11 Provide a detailed description.

[0125] Figure 11 This is a diagram illustrating the sequential flow of processing highly important slices at the beginning of the processing in a slice-by-slice manner according to this embodiment. For example... Figure 11As shown, based on the eye and head position / orientation information sent from the mobile terminal 10, the server 20 performs rendering processing in the following order: for example, starting with slice 3, which has high importance according to the gaze point, followed by slice 4, slice 5, slice 1, and slice 2. Furthermore, subsequent encoding, receiving, decoding, image warping, and display output processing are performed in a pipelined process in the order of slice 3, slice 4, slice 5, slice 1, and slice 2. Therefore, assuming the above reference... Figure 2 Under the various numerical conditions described, the delay from acquiring the location / orientation information of the mobile terminal 10 (detected by a sensor) to display (as a slice 3 of the important area) is, for example, approximately 30 ms (1.8V), and this delay is even greater than the reference delay. Figure 4 The example described has a shorter delay. Furthermore, the delay can also be reduced to approximately 8.3 ms (0.5V) through time warping, for example.

[0126] Note that here, if any slice after slice 2 in a frame formed in the order of slice 3, slice 4, slice 5, slice 1, and slice 2 is set as the beginning, tearing may occur at the slice boundaries. In particular, there is a possibility that tearing is easily noticeable when the drawing, sending, and display processes are performed in order starting from areas important to the user (e.g., near the gaze point). Figure 11 In the example shown, since slice 3 is set at the beginning of the drawing process, slice 2 may be in the i-th frame. i Slice 3 of frame i+1 i+1 Tearing occurs between (at the boundary). In this case, although details will be described later, by performing image deformation to make the outline of the virtual object across the boundary appear continuous, the tear between the slices can be made less noticeable. Figure 11 In the example shown, in the image warping unit 103 of the mobile terminal 10, based on slice 2 of the i-th frame... i And slice 3 of frame i+1 i+1 , for slice 2 i The image is warped so that the outlines of virtual objects across the boundaries of these slices appear continuous.

[0127] The tear suppression according to this embodiment will be described in detail below.

[0128] <3-3. Inhibiting Tearing>

[0129] (3-3-1 Black Insertion Period)

[0130] As a first countermeasure, the following control exists: the display control unit 105 of the mobile terminal 10 shortens the light-emitting time (shortening the holding time) of each pixel corresponding to the area of ​​each slice. Therefore, when displaying a slice (third slice) adjacent to the previously displayed slice (first slice) on a side opposite to the scanning order (e.g., from the top to the bottom of the screen), the black insertion time is ensured by turning off the display. Specifically, when the display control unit 105 determines the display order, the black insertion period can be created by separating the update timing of adjacent slices.

[0131] For example, in Figure 8 In the example shown, a black insertion period is provided when slice 2 (third slice), which is adjacent to slice 3 (first slice), on the side opposite to the scanning sequence direction. The image warping unit 103 subjects each slice to be displayed to image warping based on the latest position / orientation information or a prediction for each slice. Therefore, for example, when image warping and display are performed in the order of slice 3, slice 4, slice 2, slice 5, and slice 1, there is a possibility of contour shift of the virtual object across the boundary between slice 2 and slice 3. In such a case, if there is a period where slice 3 and slice 2 are displayed simultaneously, the contour shift may be noticeable. Furthermore, since slice 3 is an important area for the user, especially near the gaze point, tearing may be easily noticed.

[0132] Therefore, as described above, for example, by shortening the emission time (holding time) of each pixel, a black insertion period can be provided between slice 3 and slice 2 without creating a period of simultaneous display of slice 3 and slice 2. This makes tearing (e.g., offset of the outline of virtual objects) less noticeable to the user and provides a more comfortable viewing experience.

[0133] (3-3-2. Image distortion)

[0134] As a second countermeasure, the image deformation unit 103 of the mobile terminal 10 can deform the outline of the virtual object. Specifically, the image deformation unit 103 performs image deformation based on the image data of two adjacent slices, so that the outline of the virtual object across the boundary between these slices is continuous.

[0135] In particular, as referenced Figure 11 As described, in a situation where the display type of scan lines cannot be arbitrarily selected, and when an important slice determined based on the foveation point, etc., is set as the beginning of the drawing process, the slice displayed at the end of the i-th frame (e.g., slice 2) is... i ) and the slice displayed at the beginning of frame i+1 (slice 3) i+1The possibility of tearing occurring between slices 2 (at the boundary) is high. In this case, image deformation unit 103 can deform slice 2 by... i The outline of the virtual object is deformed to match the slice 3 i+1 The continuous outline is used to suppress tearing.

[0136] Furthermore, the method for deforming the outline of a virtual object described herein can be applied to, for example, a reference... Figure 8 The description describes a situation where, in the case of a display with arbitrarily selected scan line type, slices are displayed in an arbitrary order according to the importance distribution of the fixation point, etc.

[0137] In the following text, reference will be made to Figures 12 to 16 Provide a detailed description.

[0138] Figure 12 This is a block diagram illustrating an example of the functional configuration of the image warping unit 103 according to this embodiment. For example... Figure 12 As shown, the image deformation unit 103 serves as the virtual object motion prediction and deformation unit 1031, the position / orientation prediction and deformation unit 1032, and the inter-slice contour deformation unit 1033. Note that the virtual object motion prediction and deformation unit 1031, the position / orientation prediction and deformation unit 1032, and the inter-slice contour deformation unit 1033 all perform deformation on a slice-by-slice basis.

[0139] First, the image data (RGB image) included in the rendering result sent from server 20 in slice form, along with the distance and velocity information of the virtual objects included in the rendering result, are input to image deformation unit 103. Here, Figure 13 An example of an image generated by the image generation unit 201 of server 20 in units of slices is shown.

[0140] like Figure 13 As shown, the image generation unit 201 draws the i-th frame in slices, taking into account the movement of the user U and the movement of the virtual object 30, and then draws the (i+1)-th frame in slices, taking into account the newly acquired movement of the user U and the movement of the virtual object 30. As described above, the movement of the user U can be considered by referring to the position / orientation information sent from the mobile terminal 10 or the position / orientation information after a predetermined time (predicted display time) predicted by the position / orientation prediction unit 202 based on the position / orientation information. Furthermore, when drawing the virtual object 30 in slices, the image generation unit 201 can consider the speed information of the virtual object 30 and the distance information from the user U to the virtual object 30.

[0141] Note that in Figure 13In the example shown on the right, slice 3, which includes the gaze point P, is placed at the beginning of the rendering process to shorten the display delay, and data is rendered sequentially in slice order: slice 3, slice 4, slice 5, slice 1, and slice 2. Furthermore, since the image generation unit 201 renders according to the movement of the virtual object 30 during rendering, the rendering position of the virtual object changes for each slice. The rendering result, slice by slice, includes the generated image (RGB image) and the distance and velocity information of the virtual object 30.

[0142] The virtual object motion prediction and deformation unit 1031 predicts the position of the virtual object after a predetermined time (predicted display time) based on the input velocity information and distance information of the virtual object, and deforms the generated image in slices so that the virtual object is displayed at the predicted position.

[0143] The position / orientation prediction and deformation unit 1032 obtains the prediction result of the latest position / orientation information from the position / orientation prediction unit 104, and based on the prediction result, further performs image deformation on the generated image in slices, so that the virtual object is displayed at the position viewed from the viewpoint position after a predetermined time (the desired display time).

[0144] As described above, by predicting the position of the virtual object and the user's position and orientation, and by also performing image deformation on the mobile terminal 10 side, the display latency of the virtual object can be minimized.

[0145] However, in such Figure 14 In cases where slices with high importance, such as those including the gaze point, are prioritized for drawing, sending, receiving, and displaying, slice 2 is displayed at the end of the i-th frame. i Slice 3 displayed at the beginning of frame i+1 i+1 The possibility of tearing occurring between sections. Such tearing could also occur due to prediction errors. In particular, due to slice 3 i+1 It's the frame swapping part, therefore in slice 3 i+1 Tearing is likely to occur between sections immediately following the previous one. Furthermore, due to section 3... i+1 This is a display area of ​​high importance, including the foveation point P, so tearing is often noticeable. Furthermore, even in displays that can control the scan lines in any order, tearing can similarly occur when the light emission time is extended to increase brightness and no black insertion period is formed.

[0146] Therefore, in this embodiment, the inter-slice contour deformation unit 1033 performs a process of deforming the contour of the virtual object across slices to make the contour of the virtual object across slices continuous. Figure 15 This is a diagram illustrating the image deformation used to describe the outline of a virtual object according to this embodiment.

[0147] First, in drawing slice 3 including the fixation point i+1 At frame (i+1), the image generation unit 201 of server 20 performs drawing by slightly increasing the width of the slice in the upward direction, and creates a slice 2 that is immediately displayed before it. i The overlapping region (hereinafter referred to as overlapping region 52) of the i-th frame. Next, the image deformation unit 103 of the mobile terminal 10 deforms the contour lines of the virtual objects between slices by the inter-slice contour deformation unit 1033 through deformation performed by the virtual object motion prediction and deformation unit 1031 based on the motion prediction of the virtual object and deformation performed by the position / orientation prediction and deformation unit 1032 based on the position and orientation prediction. Specifically, image deformation is performed such that Figure 15 Minimize the offset of the outline of the virtual object in the overlapping region 52 shown (obtain slice 2 that has undergone image deformation). i’ That is, the contour deformation unit 1033 between slices causes slice 2 to... i In slice 3 i+1 The outline of the virtual object in the overlapping region 52 and slice 3 i+1 The outlines of the virtual objects in the slice are close together, making the outlines appear to be in slice 2. i With slice 3 i+1 The lines are continuous. For example, a set of vertices (a polygonal mesh) can be used to deform the outline.

[0148] Figure 16 This is a diagram illustrating the deformation of the outline of a virtual object according to a specific description of this embodiment. For example... Figure 16 As shown, firstly, the inter-slice contour deformation unit 1033 extracts the contour of the overlapping region 52 of adjacent slices. The overlapping region 52 of adjacent slices is, for example, slice 2 of the i-th frame. i 52 overlapping regions i And slice 3 of frame i+1 i+1 52 overlapping regions i+1 .

[0149] Next, the inter-slice contour deformation unit 1033 finds corresponding points (corresponding vertices) between the extracted contours and sets these points onto the polygon mesh. Then, the two polygon meshes are synthesized (overlapping area 52). i 52 polygonal meshes and overlapping regions i+1 (The polygonal mesh). Then, the inter-slice contour deformation unit 1033 applies the combined mesh to the overlapping region 52. i To obtain an image with minimized contour offset.

[0150] Note that in the synthesis of two polygonal meshes, to make the outline appear continuous across slices, it is necessary to select the mesh control points to be closer to the area of ​​the upper slice located in the upper part of the overlapping region (overlapping region 52). i ), and closer to the lower part of the slice located in the lower part of the overlapping region (overlapping region 52) i+1 More specifically, for example, the following implementation example can be cited.

[0151] [Expression 1]

[0152] When the set of corresponding points in the slice is Each corresponding point is and

[0153] The set of corresponding points in the lower slice is Each corresponding point is

[0154] The set of vertices in the composite mesh Each vertex in middle,

[0155] The y-coordinate is a simple average. The x-coordinate is determined by the internal division ratio of the y-coordinate. weighted average

[0156] Note that the method described above for deforming the outline of a virtual object across slices is an example, and this implementation is not limited thereto.

[0157] (3-3-3. Adjustment of the dividing line)

[0158] As a third countermeasure, the image generation unit 201 of server 20 adjusts the dividing lines that divide a frame into multiple regions. If the boundaries (dividing lines) of the slices exist near the user's gaze point, tearing is easily noticeable. Therefore, the image generation unit 201 adjusts the dividing lines so that the gaze point is near the center of the slice (the center in the width direction). In the following text, reference will be made to... Figure 17 Provide a description.

[0159] Figure 17 This is a diagram illustrating the adjustment of dividing lines for dividing a frame into multiple slices, according to this embodiment. Figure 17 As shown on the left, the image generation unit 201 typically divides a frame into equal sections of height h using a predetermined number of slices. s (Display time is equal). However, as Figure 17As shown in the center, the image generation unit 201 adjusts the dividing lines (i.e., adjusts the width of the slice) so that the gaze point is near the center of the slice S3 in the width direction (height direction). Furthermore, considering the possibility that the gaze point may move slightly up or down, even if the gaze point does not move, such as... Figure 17 As shown on the right, the image generation unit 201 can further suppress tearing by randomly moving the dividing lines up and down for each frame. That is, the image generation unit 201 changes the width (height) of the slice (first display area) including the gaze point for each frame, and makes the width (height) different at least between consecutive frames.

[0160] More specifically, for example, the y-coordinate y of the upper side of slice j (the j-th slice) including the gaze point is obtained by the following formula. s [j]. Note that j is an integer 1 ≤ j ≤ N.

[0161] [Expression 2]

[0162]

[0163] Next, as Figure 17 As shown in the center, the y-coordinate at the gaze point is obtained by the following formula. gaze The y-coordinate of the upper side of slice j in the case of s [j]. Note that when j is an integer 2≤j≤N and j=1, y s [1] = 0.

[0164] [Expression 3]

[0165]

[0166] In addition, when Figure 17 When the dividing line is randomly shifted as shown on the right, y is obtained by introducing a dither term as shown in the formula below. s [j]. Note that j is an integer of 2 ≤ j ≤ N. Here, the jitter term [i, j] is the j-th slice of the i-th frame. And, when j = 1, y s [1] = 0.

[0167] [Expression 4]

[0168]

[0169] 4. <<Supplement>>

[0170] As described above, in embodiments of this disclosure, latency can be reduced when receiving and displaying data of virtual objects generated by external devices.

[0171] Preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the technology is not limited to such examples. It will be apparent to those skilled in the art that various variations or modifications can be conceived within the scope of the technical concept described in the claims. It should be understood that these naturally fall within the technical scope of the present disclosure as well.

[0172] For example, in the above embodiments, it has been described that head position / orientation estimation and eye position / orientation estimation are performed in the position / orientation estimation unit 101. However, this disclosure is not limited to this. For example, only the position and orientation of the head may be estimated, and the head position / orientation information may be sent to the server 20. Furthermore, in the above embodiments, it has been described that the position and orientation of the head and the position and orientation of the eyeballs are predicted in the position / orientation prediction unit 104. However, this disclosure is not limited to this. For example, only the position and orientation of the head may be predicted, and the prediction result may be output to the image deformation unit 103. The image deformation unit 103 may perform image deformation using only the head position / orientation information or only the prediction result of the head position and orientation. Furthermore, the head position and orientation and the eyeball position and orientation are examples of user motion information, and this disclosure is not limited to this.

[0173] Furthermore, a computer program can be created to enable hardware such as the CPU, ROM, and RAM built into the aforementioned mobile terminal 10 or server 20 to perform the functions of the mobile terminal 10 or server 20. Additionally, a computer-readable storage medium storing the computer program is provided.

[0174] Furthermore, the effects described in this specification are merely illustrative or exemplary and are not restrictive. That is, in addition to or in lieu of the effects described above, the technology according to this disclosure may exhibit other effects that would be apparent to those skilled in the art based on the description herein.

[0175] Note that this technology can also be configured as follows. (1)

[0177] A display processing device, comprising:

[0178] The transmitting unit sends the first motion information of the mobile terminal to an external device;

[0179] A receiving unit receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area; and

[0180] The display control unit, after displaying the data received in the first display area, performs control to display the data in the second display area together with the data in the first display area as a frame on the display device of the mobile terminal. (2)

[0182] According to the display processing device described in (1) above, wherein,

[0183] The receiving unit receives data of a third display area of ​​the virtual object corresponding to a portion of the frame, drawn by the external device based on the first motion information, at a third time later than the second time.

[0184] The display control unit controls the display device, which displays the data of the first display area at a time earlier than the third time, and displays the data of the third display area later than the data of the second display area. (3)

[0186] The display processing apparatus according to (1) or (2) above further includes a deformation unit, which deforms the received data of the second display area based on second motion information acquired later than the first motion information, wherein...

[0187] The display control unit performs control to display data in the distorted second display area. (4)

[0189] According to the display processing device described in (3) above, wherein,

[0190] In a display area obtained by dividing a region of a frame displayed on the display device, the first display area is a display area located in the region that includes the user's gaze point on the display device. (5)

[0192] According to the display processing device described in (4) above, wherein,

[0193] The second display area is a display area located closer to the gaze point than the third display area, which is received by the receiving unit at a third time later than the second time. (6)

[0195] According to the display processing apparatus described in (4) or (5) above, wherein,

[0196] The deformation unit deforms the outline of the virtual object to make it continuous. The outline crosses the boundaries of multiple adjacent display areas among multiple display areas, which form a frame displayed on the display device. (7)

[0198] The display processing apparatus according to any one of (4) to (6) above, wherein,

[0199] The display control unit performs the following control: after a certain period of time has elapsed since the display of data in the first display area is completed, the data in the third display area is displayed. The third display area is adjacent to the first display area on the side opposite to the second display area, and the data in the third display area is received at a third time later than the second time. (8)

[0201] The display processing apparatus according to any one of (4) to (7) above, wherein,

[0202] The first display area is generated such that the gaze point is located at the center in the width direction of the first display area. (9)

[0204] According to the display processing apparatus described in (8) above, wherein,

[0205] The width of the first display area varies between consecutive frames. (10)

[0207] The display processing apparatus according to any one of (1) to (9) above, wherein,

[0208] The first motion information includes information related to the position and orientation detected by the motion sensor located in the mobile terminal. (11)

[0210] According to the display processing device described in (3) above, wherein,

[0211] The second motion information includes information related to the position and orientation detected by the motion sensor located in the mobile terminal, and

[0212] The deformation unit deforms the data in the second display area based on the prediction result according to the second motion information. (12)

[0214] A display processing method includes the following steps by a processor:

[0215] Send the first motion information of the mobile terminal to an external device.

[0216] The system receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area.

[0217] After displaying the data received in the first display area, control is executed to display the data in the second display area together with the data in the first display area as a single frame on the display device of the mobile terminal. (13)

[0219] A stored program storage medium, the program being used by a computer as:

[0220] The transmitting unit sends the first motion information of the mobile terminal to an external device;

[0221] A receiving unit receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area; and

[0222] The display control unit, after displaying the data received in the first display area, performs control to display the data in the second display area together with the data in the first display area as a frame on the display device of the mobile terminal. (14)

[0224] An information processing apparatus, comprising:

[0225] The receiving unit receives the first motion information from the mobile terminal, and

[0226] The sending unit firstly sends data of a first display area of ​​a virtual object, which is drawn as part of a frame based on the first motion information, to a display processing device at a first time. The display processing device performs control to display the data of the first display area on the display device of the mobile terminal. After the data of the first display area is drawn based on the first motion information, the sending unit sends data of a second display area of ​​the virtual object, which is drawn as another part of the frame, to the display processing device at a second time later than the first time. The second display area is adjacent to the first display area.

[0227] Reference tag list

[0228] 10 mobile terminals

[0229] 100 control unit

[0230] 101 Position / Orientation Estimation Unit

[0231] 102 Data Decoding Unit

[0232] 103 image deformation units

[0233] 104 Position / Orientation Prediction Units

[0234] 105 Display Control Unit

[0235] 110 sensor unit

[0236] 120 communication units

[0237] 130 clock signal generation unit

[0238] 140 timestamp counter

[0239] 150 storage units

[0240] 160 display units

[0241] 20 servers

[0242] 200 control unit

[0243] 201 Image Generation Unit

[0244] 202 Position / Orientation Prediction Unit

[0245] 203 Data Encoding Unit

[0246] 220 communication units

[0247] 230 clock signal generation unit

[0248] 240 timestamp counter

[0249] 250 storage units

Claims

1. A display processing device, comprising: The transmitting unit sends the first motion information of the mobile terminal to an external device; A receiving unit receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area; as well as The display control unit, after displaying the received data from the first display area, executes control to display the data from the second display area together with the data from the first display area as a single frame on the display device of the mobile terminal. Among the display areas obtained by dividing the area of ​​a frame displayed on the display device, the first display area is a display area located in the area including the user's gaze point on the display device, and the first display area is generated such that the gaze point is located at the center in the width direction of the first display area.

2. The display processing device according to claim 1, wherein, The receiving unit receives data of a third display area of ​​the virtual object corresponding to a portion of the frame, drawn by the external device based on the first motion information, at a third time later than the second time. The display control unit controls the display device, which displays the data of the first display area at a time earlier than the third time, and displays the data of the third display area later than the data of the second display area.

3. The display processing apparatus according to claim 1 further includes a deformation unit, wherein the deformation unit deforms the received data of the second display area based on second motion information acquired later than the first motion information, wherein... The display control unit performs control to display data in the distorted second display area.

4. The display processing apparatus according to claim 1, wherein, The second display area is a display area located closer to the gaze point than the third display area, which is received by the receiving unit at a third time later than the second time.

5. The display processing apparatus according to claim 3, wherein, The deformation unit deforms the outline of the virtual object to make it continuous. The outline crosses the boundaries of multiple adjacent display areas among multiple display areas, which form a frame displayed on the display device.

6. The display processing apparatus according to claim 1, wherein, The display control unit performs the following control: after a certain period of time has elapsed since the display of data in the first display area is completed, the data in the third display area is displayed. The third display area is adjacent to the first display area on the side opposite to the second display area, and the data in the third display area is received at a third time later than the second time.

7. The display processing apparatus according to claim 1, wherein, The width of the first display area varies between consecutive frames.

8. The display processing apparatus according to claim 1, wherein, The first motion information includes information related to the position and orientation detected by the motion sensor located in the mobile terminal.

9. The display processing apparatus according to claim 3, wherein, The second motion information includes information related to the position and orientation detected by the motion sensor located in the mobile terminal, and The deformation unit deforms the data in the second display area based on the prediction result according to the second motion information.

10. A display processing method, comprising the following steps by a processor: Send the first motion information of the mobile terminal to an external device. The system receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area. After displaying the received data from the first display area, control is executed to display the data from the second display area together with the data from the first display area as a single frame on the display device of the mobile terminal. in, In a display area obtained by dividing the area of ​​a frame displayed on the display device, the first display area is a display area located in the area including the user's gaze point on the display device, and the first display area is generated such that the gaze point is located at the center in the width direction of the first display area.

11. A storage medium containing a stored program, the program being used by a computer as: The transmitting unit sends the first motion information of the mobile terminal to an external device; The receiving unit receives data of a first display area of ​​a virtual object drawn by the external device based on the first motion information at a first time, and receives data of a second display area of ​​the virtual object drawn by the external device based on the first motion information at a second time later than the first time, wherein the second display area is adjacent to the first display area; as well as The display control unit, after displaying the received data from the first display area, executes control to display the data from the second display area together with the data from the first display area as a single frame on the display device of the mobile terminal. Among the display areas obtained by dividing the area of ​​a frame displayed on the display device, the first display area is a display area located in the area including the user's gaze point on the display device, and the first display area is generated such that the gaze point is located at the center in the width direction of the first display area.

12. An information processing apparatus, comprising: The receiving unit receives the first motion information from the mobile terminal, and A sending unit firstly sends data of a first display area of ​​a virtual object, drawn as part of a frame based on the first motion information, to a display processing device, which then performs control to display the data of the first display area on the display device of the mobile terminal. After the data of the first display area is drawn based on the first motion information, the sending unit then sends data of a second display area of ​​the virtual object, drawn as another part of the frame, to the display processing device at a second time later than the first time. This second display area is adjacent to the first display area. Among the display areas obtained by dividing the area of ​​a frame displayed on the display device, the first display area is a display area located in the area including the user's gaze point on the display device, and the first display area is generated such that the gaze point is located at the center in the width direction of the first display area.

Images