Terminal equipment

The terminal device optimizes stereoscopic image processing by adjusting the number of element images based on user distance, enhancing efficiency and clarity in light field displays.

JP7589733B2Active Publication Date: 2024-11-26TOYOTA JIDOSHA KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022212673
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-12-28
Publication Date
2024-11-26
Estimated Expiration
2042-12-28

AI Technical Summary

Technical Problem

Existing light field type devices face inefficiencies in processing stereoscopic image display.

Method used

A terminal device with an imaging unit that captures user images and a display unit that outputs light rays for stereoscopic images, adjusting the number of element images based on the user's distance from the display surface to optimize processing efficiency.

Benefits of technology

Improves the efficiency of stereoscopic image processing by reducing the number of element images rendered based on user position, thereby reducing processing load and maintaining image clarity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589733000001
    Figure 0007589733000001
  • Figure 0007589733000002
    Figure 0007589733000002
  • Figure 0007589733000003
    Figure 0007589733000003
Patent Text Reader

Abstract

To provide a terminal capable of improving the efficiency of processing related to stereoscopic image display.SOLUTION: The terminal includes: an imaging unit for picking up an image of a user; a display unit that has a display plane for outputting rays that compose the element images for each of the multiple viewpoint positions for the 3D image to be displayed toward each viewpoint position; and a control unit that is configured so as to, when the distance of the user from the display plane obtained from the captured image of the user is the first distance, output first number of the element images, and when the distance is a second distance which is greater than the first distance, output rays constituting element images of the second number less than the first number to the display unit.SELECTED DRAWING: Figure 2B
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a terminal device. [Background technology]

[0002] As an example of a technology for supporting a user experience in virtual reality or augmented reality, various technologies for displaying a stereoscopic image of an object have been proposed to improve the reality of the user experience. For example, Patent Document 1 discloses a technology related to a heads-up display that displays a stereoscopic image of an object using a light field method. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-189279 Summary of the Invention [Problem to be solved by the invention]

[0004] When a light field type device is adopted, there is room for further improving the efficiency of processing related to stereoscopic image display.

[0005] The present disclosure provides a terminal device etc. that can further improve the efficiency of processing related to stereoscopic image display. [Means for solving the problem]

[0006] A terminal device in the present disclosure includes an imaging unit that images a user, a display unit having a display surface that outputs light rays that constitute elemental images for each of a plurality of viewpoint positions relative to a stereoscopic image for displaying the stereoscopic image, toward each viewpoint position, and a control unit that causes the display unit to output light rays that constitute a first number of the elemental images when a distance of the user from the display surface obtained from the captured image of the user is a first distance, and output a second number of the elemental images that is smaller than the first number when the distance is a second distance greater than the first distance. Effect of the Invention

[0007] According to the terminal device and the like of the present disclosure, it is possible to further improve the efficiency of processing related to stereoscopic image display. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a virtual event providing system. [Figure 2A] FIG. 11 is a flowchart illustrating an example of the operation of the terminal device. [Figure 2B] FIG. 11 is a flowchart illustrating an example of the operation of the terminal device. [Figure 3A] FIG. 13 is a diagram showing an example of an element image. [Figure 3B] 1A and 1B are diagrams illustrating examples of positions of a user relative to a display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1 shows an example of the configuration of a virtual event providing system including a terminal device in one embodiment. In the virtual event providing system 1, a plurality of terminal devices 12 and a server device 10 are connected to each other via a network 11 so as to be able to communicate information with each other. The virtual event providing system 1 is a system for providing an event in a virtual space in which a user can participate using the terminal device 12, i.e., a virtual event. A virtual event is an event in which a plurality of users communicate information by speech or the like in a virtual space, and each user is represented by a 3D model that represents each user.

[0010] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be composed of two or more server computers that are connected to each other so as to be able to communicate information and operate in cooperation with each other. The server device 10 transmits and receives information and processes information required to provide a virtual event.

[0011] The terminal device 12 is an information processing device equipped with a communication function, and is used by a user who participates in a virtual event provided by the server device 10. The terminal device 12 is, for example, an information processing device such as an information processing terminal, such as a smartphone or a tablet terminal, or a personal computer.

[0012] Network 11 may be, for example, the Internet, but may also include an ad-hoc network, a Local Area Network (LAN), a Metropolitan Area Network (MAN), or other network, or any combination thereof.

[0013] In this embodiment, the terminal device 12 has an imaging unit 117 that images the user, a display unit / output unit 116, and a control unit 113. The display / output unit 116 outputs light rays constituting elemental images for each of a plurality of viewpoint positions for a stereoscopic image to display the stereoscopic image, toward each viewpoint position. The control unit 113 causes the display unit to output light rays constituting a first number of elemental images when the distance of the user from the display surface obtained from the captured image of the user is a first distance, and a second number of elemental images smaller than the first number when the distance is a second distance larger than the first distance.

[0014] When multiple users participate in a virtual event held on a network by the server device 10 using each of the terminal devices 12, a virtual space image including a 3D model based on a captured image of each user is displayed on each of the terminal devices 12. The control unit 113 renders multiple element images when rendering the virtual space image, which places a corresponding processing load on the control unit 113. The control unit 113 of this embodiment can reduce the processing load by reducing the number of element images to be rendered depending on the position of the user relative to the display surface. This makes it possible to improve the efficiency of processing related to stereoscopic image display.

[0015] The configurations of the server device 10 and the terminal device 12 will now be described in detail.

[0016] The server device 10 includes a communication unit 101, a storage unit 102, and a control unit 103. When the server device 10 is configured with two or more server computers, these components are appropriately arranged in the two or more computers.

[0017] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used in the operation of the server device 10, and transmits information obtained by the operation of the server device 10. The server device 10 is connected to a network 11 by the communication unit 101, and communicates information with a terminal device 12 via the network 11.

[0018] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, functioning as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable ROM (EEPROM). The storage unit 102 stores information used in the operation of the server device 10 and information obtained by the operation of the server device 10.

[0019] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc. The control unit 103 executes information processing related to the operation of the server device 10 while controlling each unit of the server device 10.

[0020] The functions of the server device 10 are realized by a processor included in the control unit 103 executing a control program. The control program is a program for causing a computer to function as the server device 10. Also, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Also, the control program may be stored in a non-transitory recording / storage medium readable by the server device 10, and read by the server device 10 from the medium.

[0021] The terminal device 12 includes a communication unit 111 , a storage unit 112 , a control unit 113 , an input unit 115 , a display / output unit 116 , and an imaging unit 117 .

[0022] The communication unit 111 has a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as LTE, 4G, 5G, etc. The terminal device 12 is connected to the network 11 by the communication unit 111 via a nearby router device or a mobile communication base station, and performs information communication with the server device 10, etc. via the network 11.

[0023] The memory unit 112 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memories are, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The memory unit 112 functions as, for example, a main memory device, an auxiliary memory device, or a cache memory. The memory unit 112 stores information used in the operation of the control unit 113 and information obtained by the operation of the control unit 113.

[0024] The control unit 113 has, for example, one or more general-purpose processors such as a CPU, an MPU (Micro Processing Unit), or one or more dedicated processors such as a GPU specialized for a specific process. Alternatively, the control unit 113 may have one or more dedicated circuits such as an FPGA, an ASIC, or the like. The control unit 113 generally controls the operation of the terminal device 12 by operating according to a control / processing program, or operating according to an operating procedure implemented as a circuit. The control unit 113 transmits and receives various information to and from the server device 10, etc. via the communication unit 111, and executes the operation according to this embodiment.

[0025] The input unit 115 includes one or more input interfaces. The input interfaces include, for example, physical keys, capacitive keys, a pointing device, and a touch screen that is integral with a display. The input interfaces also include a microphone that accepts voice input. Furthermore, the input interfaces may include a scanner or camera that scans image codes, and an IC card reader. The input unit 115 accepts an operation to input information used in the operation of the control unit 113, and sends the input information to the control unit 113.

[0026] The display / output unit 116 includes one or more output interfaces for outputting information generated by the operation of the control unit 113. The output interfaces include, for example, a display and a speaker. The display is a light field display corresponding to the "display unit" of this embodiment.

[0027] The imaging unit 117 includes a camera that captures an image of a subject using visible light, and a distance measurement sensor that measures the distance to the subject to obtain a distance image. The camera captures images of the subject at, for example, 15 to 30 frames per second to generate a video consisting of continuous captured images. The distance measurement sensor includes a ToF (Time Of Flight) camera, LiDAR (Light Detection And Ranging), and a stereo camera, and generates a distance image of the subject including distance information. The imaging unit 117 sends the captured image and the distance image to the control unit 113.

[0028] The functions of the control unit 113 are realized by a processor included in the control unit 113 executing a control program. The control program is a program for causing the processor to function as the control unit 113. Also, some or all of the functions of the control unit 113 may be realized by a dedicated circuit included in the control unit 113. Also, the control program may be stored in a non-transitory recording / storage medium readable by the terminal device 12, and read by the terminal device 12 from the medium.

[0029] In this embodiment, the control unit 113 acquires a captured image and a distance image of the user of the terminal device 12 by the imaging unit 117, and collects the user's speech by the microphone of the input unit 115. The control unit 113 generates encoded information by encoding the captured image and distance image of the user for generating a 3D model representing the user, and audio information for reproducing the user's voice. When encoding, the control unit 113 may perform any processing (e.g., changing the resolution and trimming) on ​​the captured image, etc. The control unit 113 sends the encoded information to another terminal device 12 via the server device 10 by the communication unit 111.

[0030] Furthermore, the control unit 113 receives encoded information sent from the other terminal device 12 via the server device 10, by the communication unit 111. When the control unit 113 decodes the encoded information received from the other terminal device 12, the control unit 113 uses the decoded information to generate a 3D model representing a user who uses the other terminal device 12, and places the 3D model in the virtual space. The control unit 113 generates a virtual space image including the 3D model from a predetermined viewpoint in the virtual space by rendering. The virtual space image includes element images for each of a plurality of viewpoint positions with respect to the light field display. That is, the control unit 113 renders the plurality of element images. The control unit 113 outputs the virtual space image to the display / output unit 116. The display / output unit 116 displays the virtual space image by outputting light rays constituting the plurality of element images to each viewpoint position, and outputs speech based on the voice information of each user.

[0031] By such operations of the control unit 113 etc., the user of the terminal device 12 can participate in a virtual event in real time and talk with other users.

[0032] 2A and 2B are flow charts for explaining the operation procedure of the terminal device 12 for implementing a virtual event. The server device 10 mediates communication between a plurality of terminal devices 12 on the network 11, thereby enabling communication between the terminal devices 12, thereby implementing the virtual event.

[0033] The procedure in FIG. 2A relates to an operation procedure of the control unit 113 when each terminal device 12 transmits information for generating a 3D model of a user who uses that terminal device 12.

[0034] In step S201, the control unit 113 causes the imaging unit 117 to capture a visible light image and a distance image of the user at an arbitrarily set frame rate, and collects the speech of the user by the input unit 115. The control unit 113 acquires the visible light captured image and the distance image from the imaging unit 117, and acquires audio information from the input unit 115.

[0035] In step S202, the control unit 113 derives the position of the user. The position of the user is specified by the spatial coordinates of an arbitrary part of the user's face that can be image-recognized. The arbitrary part is, for example, either eye or the midpoint between both eyes. The control unit 113 recognizes the user and the parts of his face by image processing such as pattern matching on the captured image. The control unit 113 also derives, for example, the spatial coordinates of the user's eyes relative to the position of the camera of the imaging unit 117 based on the distance image. The control unit 113 derives the position of the user based on the determined spatial coordinates.

[0036] In step S203, the control unit 113 encodes the captured image, the distance image, and the audio information to generate encoded information.

[0037] In step S 204 , the control unit 113 packetizes the encoded information using the communication unit 111 , and transmits the packetized information to the server device 10 and other terminal devices 12 .

[0038] When the control unit 113 acquires information inputted corresponding to an operation for interrupting image capture and sound collection or an operation for exiting the virtual event (Yes in S205), this processing procedure ends. While the control unit 113 does not acquire information corresponding to an operation for interrupting or exiting (No in S205), the control unit 113 executes steps S201 to S204 to derive the user's gaze point and transmit information for generating a 3D model representing the user and information for outputting sound.

[0039] The procedure in Fig. 2B relates to an operation procedure of the control unit 113 when the terminal device 12 outputs a virtual event image and the voice of another user. When the control unit 113 receives, via the server device 10, a packet transmitted by another terminal device 12 executing the procedure in Fig. 2A, the control unit 113 executes steps S211 to S213.

[0040] In step S211, the control unit 113 decodes the encoded information contained in the packet received from the other terminal device 12 to obtain a captured image, a distance image, and audio information.

[0041] In step S212, the control unit 113 generates a 3D model representing the other user based on the captured image and the distance image. When generating the 3D model, the control unit 113 generates a polygon model using the distance image of the other user, and applies texture mapping using the captured image of the other user to the polygon model to generate the 3D model of the other user. However, the generation of the 3D model is not limited to the example shown here, and any method can be adopted.

[0042] When receiving information from the terminal devices 12 of a plurality of other users, the control unit 113 executes steps S211 to S212 for each of the other terminal devices 12, and generates a 3D model for each user.

[0043] In step S213, the control unit 113 places a 3D model representing each user in the virtual space where the virtual event is held. The control unit 113 places the generated 3D models of other users at coordinates in the virtual space. The storage unit 112 stores coordinate information of the virtual space and information on the coordinates at which the 3D models of each user should be placed. For example, the 3D models of each user are assigned a position according to the order in which each user logged in to the virtual event. Alternatively, the 3D models can be moved in the virtual space by the operation of each user, and in that case, a position according to the user's operation is assigned to the 3D models.

[0044] In step S214, the control unit 113 determines the number of element images. The control unit 113 determines the number of element images according to the position of the user.

[0045] In step S215, the control unit 113 generates a virtual space image by rendering one or more 3D models arranged in the virtual space, which is obtained by capturing an image from a virtual viewpoint. The virtual space image includes a plurality of element images. The control unit 113 generates the number of element images derived in step S214.

[0046] In step S216, control unit 113 displays a virtual space image and outputs sound by display / output unit 116. That is, control unit 113 outputs information for displaying a virtual space image in which a 3D model is arranged in a virtual space to display / output unit 116 so as to cause the display unit to output light rays constituting the number of element images derived in step S214. Display / output unit 116 displays the virtual space image and outputs sound.

[0047] FIG. 3A is a schematic plan view showing an example of element images constituting a virtual space image. Here, an example is shown in which five element images P31 to P35 are output toward five viewpoint positions 31 to 35 with respect to the stereoscopic image 36 when the light field display 30 displays the stereoscopic image 36 near the center of the display surface 37. The stereoscopic image 36 is, for example, an image of a head including a person's face. Microlens groups L31 to L37, each of which corresponds to a plurality of pixels, are arranged on the display surface 37. The microlens groups L31 to L37 output light rays for constituting the element images P31 to P35 output by the corresponding pixels as light rays group 38 at different angles with respect to the center of the display surface 37. The light rays group 38 are output at angles that form the element images P31 to P35 at the viewpoint positions 31 to 35, respectively. The user visually recognizes the element images P31 to P35 at the viewpoint positions 31 to 35, respectively, and perceives the stereoscopic image 36. For example, elemental images P31 to P35 correspond to images of the stereoscopic image 36 from the right side, the right diagonal front, the front, the left diagonal front, and the right side. Note that the number of elemental images and the number of microlens groups shown here are simplified examples, and needless to say, are not limited to the examples shown here. Also, the microlens group may be, for example, a lenticular lens group that extends in the vertical direction of the display surface 37 and corresponds to a pixel group arranged in the extension direction.

[0048] 3B shows examples of user positions 301-304 relative to the display surface 37 of the light field display 30. The control unit 113 derives the distances of the user positions 301 and 302 from the display surface 37 based on the user positions. Here, an example of distances D1 and D2 (D2>D1) is shown. The control unit 113 also derives the distances of the user positions 304 and 303 from the center C30 of the display surface 37. Here, an example of distances d1 and d2 (d2>d1) relative to the center C30 is shown.

[0049] The control unit 113 determines the number of element images so that the number of element images when the distance of the user's position from the display surface 37 is a distance D2 larger than the distance D1 is smaller than the number of element images when the distance of the user's position from the center C30 of the display surface 37 is a distance D1. Furthermore, the control unit 113 determines the number of element images so that the number of element images when the distance of the user's position from the center C30 of the display surface 37 is a distance d2 larger than the distance d1 is not larger than the number of element images when the distance of the user's position from the center C30 of the display surface 37 is a distance d1. Any means can be used to determine such element images according to the user's position. For example, the range of values ​​of the distance of the user's position from the display surface 37, the range of values ​​of the distance from the center C30 of the display surface 37, and the number of element images are stored in advance in correspondence with each other in the storage unit 112, and the control unit 113 can read out the number of element images corresponding to the user's position from the storage unit 112 to determine the number of element images. Alternatively, the control unit 113 may determine the number of element images by multiplying the number of element images, which is arbitrarily determined in advance, by a coefficient that decreases according to the distance of the user's position from the display surface 37 and a coefficient that increases according to the distance from the center C30 of the display surface 37. In this case, it is possible to weight the distance from the display surface 37 more heavily than the distance from the center C30 of the display surface 37. The control unit 113 generates elemental images equal to or less than the number of viewpoint positions to which the light beam group 38 is directed. The control unit 113 may reduce the number of elemental images, for example, by outputting light beams for forming the same elemental image toward a plurality of adjacent viewpoint positions.

[0050] The farther the user's position is from the display surface 37, the fewer the number of elemental images that the user must view in order to perceive the stereoscopic image 36 when the user moves parallel to the display surface 37. Therefore, by controlling to generate fewer elemental images when the user's position is farther from the display surface 37, the processing load of the control unit 113 is reduced compared to the case where a certain number or more of elemental images are constantly generated.

[0051] In addition, the greater the angle of the light beams output from the microlenses arranged on the display surface 37 relative to the vertical direction of the display surface 37, the greater the possibility that the light intensity will decrease due to interference with the light beams from adjacent microlenses. In this case, when the stereoscopic image 36 is displayed near the center C30 of the display surface 37, the further the user's position is from the center C30, that is, the greater the angle of the light beams toward the user's position relative to the vertical direction of the display surface 37, the greater the possibility that the elemental images formed by the light beams will become unclear. Therefore, by increasing the number of elemental images as the user's position is farther from the center C30, it is possible to maintain the clarity of the elemental images viewed by the user.

[0052] By repeatedly executing steps S211 to S216 by the control unit 113, the user can hear the voices of the other users while watching videos of virtual space images including the 3D models of the other users.

[0053] 2A and 2B are executed in parallel, for example, in a time-sharing manner, so that a user can use the terminal device 12 to talk to other users while viewing each other's 3D models in a virtual event. At that time, the control unit 113 can reduce unnecessary processing load by increasing or decreasing the number of element images generated according to the user's position. Furthermore, it is possible to maintain the clarity of the element images. Therefore, it is possible to further improve the efficiency of processing related to stereoscopic image display.

[0054] In the above, the embodiment has been described based on the drawings and examples, but it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present disclosure. For example, the functions included in each means, each step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined into one or divided into one. [Explanation of symbols]

[0055] 1. Virtual Event Delivery System 10. Server equipment 11 Network 12 Terminal Equipment 101, 111 Communications Department 102, 112 Storage section 103, 113 Control section 115 Input section 116 Display and output section 117 Imaging unit

Claims

1. An imaging unit that images a user; a display unit having a display surface for outputting light rays constituting elemental images for each of a plurality of viewpoint positions with respect to the stereoscopic image for displaying the stereoscopic image, toward each of the viewpoint positions; a control unit that outputs, to the display unit, light rays constituting a first number of the element images when a distance of the user from the display surface obtained from a captured image of the user is a first distance, and light rays constituting a second number of the element images smaller than the first number when the distance is a second distance larger than the first distance, Terminal device.

2. In claim 1, Further comprising a communication unit for communicating with other terminal devices, The control unit transmits information for generating a 3D model representing the user based on the captured image to another terminal device, and an image including the 3D model is displayed on the other terminal device. Terminal device.

Citation Information

Patent Citations

  • Multiple viewpoint solid image display method, multiple viewpoint solid image display unit and multiple viewpoint solid image display program

    JP2006262191A

  • Image processing device, method, and program, and stereoscopic image display device

    JP2014135590A

  • Display apparatus and controlling method thereof

    JP2015149718A

  • Head-up display device

    JP2021189279A