Electronic apparatus and method for controlling electronic apparatus

Abstract

To accurately draw a valuable area in contents even when users appreciate a virtual space from free viewing locations, and reduce a processing load.SOLUTION: An electronic apparatus has: acquisition means that acquires users' eyes statistical information representing the frequency at which users turn their eyes to every area of a virtual space; and control means that controls, for every area, a method for processing of drawing an image of the virtual space on the basis of the users' eyes statistical information. The acquisition means generates the users' eyes statistical information on the basis of the directions of the users' eyes looking at the image and the virtual positions of the users in the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] In recent years, virtual reality (VR) technology that allows users to experience a virtual world has become well known. For example, VR technology has been used to recreate tourist spots and museums in virtual spaces, providing content called virtual tours that can be viewed from any viewpoint. Content such as virtual tours can be viewed using a head-mounted display (HMD) or similar. The HMD displays virtual images to the user by rendering display data in accordance with the position and posture of the user wearing the HMD.

[0003] When high-definition rendering is performed when generating a virtual image, the processing load increases. In order to allow the user to view the image comfortably, it is preferable to reduce the processing load. Patent Document 1 discloses a technique for rendering such that the rendering process in an area including the focal position on which the user is paying attention has a larger processing load than the rendering process in an area not including the focal position. Patent Document 2 discloses a technique for generating a stereoscopic image based on the viewpoint position of a virtual camera, and delivering a two-dimensional image that takes into account the viewer's reaction and viewpoint even after the fact based on the generated stereoscopic image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-73104 A [Patent Document 2] JP 2018-74294 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the position where the user directs his / her gaze may not coincide with a valuable position (a position that requires attention) in the content. When the user is viewing a virtual space from a free viewpoint, if the area including the focus that the user is paying attention to is rendered in high resolution, the rendering accuracy in the valuable area in the content may decrease, or the rendering may be delayed due to a change in the field of view.

[0006] An object of the present invention is to render valuable areas of content in high definition and reduce the processing load even when a user is viewing a virtual space from any viewpoint. [Means for solving the problem]

[0007] The electronic device of the present invention is characterized by having an acquisition means for acquiring gaze statistical information indicating to what extent the gaze is directed to each area in a virtual space, and a control means for controlling a method of drawing processing of an image in the virtual space for each area based on the gaze statistical information. Effect of the Invention

[0008] According to the present invention, even when a user is viewing a virtual space from any viewpoint, valuable areas of content can be rendered in high definition and the processing load can be reduced. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image processing system according to a first embodiment. [Diagram 2] FIG. 11 is a diagram illustrating gaze statistical information. [Diagram 3] 4 is a flowchart showing processing of the image processing system according to the first embodiment. [Figure 4] FIG. 11 is a diagram illustrating an example of the configuration of an image processing system according to a second embodiment. [Diagram 5] 10 is a flowchart showing the processing of an image processing system according to a second embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of the configuration of an image processing system according to a third embodiment. [Figure 7] 11 is a flowchart showing the processing of an image processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Furthermore, unless otherwise specified, the contents and relative arrangement of components in each device are not intended to limit the scope of the present invention.

[0011] In the first embodiment, the image processing device draws a virtual image based on the position and orientation of a user wearing a display device such as an HMD, and transmits the drawn image (content image) to the display device. The display device displays the image received from the image processing device on a display.

[0012] 1 is a diagram showing an example of the configuration of an image processing system 100 according to a first embodiment. The image processing system 100 includes a display device 110 and an image processing device 120. The image processing device 120 according to the first embodiment is an example of an electronic device according to the present invention. The display device 110 and the image processing device 120 are connected to each other so as to enable data communication. The connection between the display device 110 and the image processing device 120 may be wired or wireless.

[0013] The display device 110 is a small terminal such as an HMD, a handheld display (HHD), or a smartphone. A user can view images of content while wearing (holding) the display device 110 on a part of the body. The display device 110 includes a control unit 111, a ROM 112, a RAM 113, a communication unit 114, a display unit 115, a position and orientation acquisition unit 116, and a gaze direction acquisition unit 117.

[0014] The control unit 111 is, for example, a CPU. The control unit 111 reads out an operation program for each block included in the display device 110 from the ROM 112, loads it into the RAM 113, and executes it to control the operation of each block included in the display device 110.

[0015] The ROM 112 is a read-only non-volatile memory. The ROM 112 stores operation programs for each block of the display device 110, as well as parameters and data used in the operation of each block. The RAM 113 is a rewritable volatile memory. The RAM 113 is used, for example, as a temporary storage area for data output by the operation of each block of the display device 110.

[0016] The communication unit 114 is, for example, a network card. The communication unit 114 is an interface for communication between the display device 110 and the image processing device 120. The display unit 115 is a display device such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The display unit 115 sequentially displays images to the user.

[0017] The position and orientation acquisition unit 116 is, for example, a gyro sensor. The position and orientation acquisition unit 116 acquires the positions and orientations of the user and the display device 110. The gaze direction acquisition unit 117 is, for example, a camera for gaze detection. The gaze direction acquisition unit 117 acquires the direction in which the user is gazing.

[0018] The image processing device 120 is, for example, a personal computer (PC), etc. The image processing device 120 includes a control unit 121, a ROM 122, a RAM 123, a communication unit 124, a drawing unit 125, and a statistical information acquisition unit 126. The control unit 121, the ROM 122, the RAM 123, and the communication unit 124 are similar to the control unit 111, the ROM 112, the RAM 113, and the communication unit 114 of the display device 110, and therefore description thereof will be omitted.

[0019] The drawing unit 125 renders an image to be displayed on the display unit 115 from the data recorded in the ROM 122. The rendering process is a process of generating an image to be displayed on the display unit 115 for the user to view from content data such as a virtual tour.

[0020] The statistical information acquisition unit 126 acquires gaze statistical information using information acquired from the position and orientation acquisition unit 116 and the gaze direction acquisition unit 117. That is, the statistical information acquisition unit 126 generates gaze statistical information based on the virtual position of the user in the virtual space and the gaze direction of the user viewing the image. In this specification, the virtual space includes not only a space that does not exist in the real space, but also a space that reproduces a tourist spot or a museum that exists in the real space. The gaze statistical information is information that indicates the extent to which the user's gaze is directed to each area in the virtual space. The drawing unit 125 can perform rendering processing by controlling the image quality for each area according to the degree to which the user's gaze is directed based on the gaze statistical information.

[0021] The user here may include a user currently viewing the content, or may include a user who has viewed the content in the past. The statistical information acquisition unit 126 stores the acquired gaze statistical information in a storage device (storage medium, memory) not shown. The storage medium includes, for example, an external hard disk and a memory card.

[0022] The gaze statistics information will be described with reference to Figures 2(A) and 2(B). Figure 2(A) illustrates a virtual space 200 in content such as a virtual tour. The virtual space 200 is a space in which a virtual object 201 and a virtual object 202 are arranged.

[0023] 2(A) shows a state in which a first user 211 and a second user 212 experiencing a virtual space 200 are arranged at their respective virtual positions in the virtual space 200. When a popular object in the virtual space 200 is a virtual object 201, it is assumed that a line of sight 221 of the first user 211 and a line of sight 222 of the second user 212 are directed toward the virtual object 201. The statistical information acquisition unit 126 can collect information on the lines of sight directed toward the virtual object 201 and the virtual object 202, and acquire line of sight statistical information such as a three-dimensional heat map 230 illustrated in FIG. 2(B).

[0024] The gaze statistical information may also be a gaze frequency map indicating the frequency at which the user's gaze is directed to each area in the virtual space. The gaze frequency map is generated, for example, by integrating the intersections between the areas and the user's gazes over time to map the gaze frequency. The gaze statistical information may also be a gaze density map indicating a weight according to the number of users whose gazes are directed to each area in the virtual space. The gaze density map is generated, for example, by weighting each area by the number of users whose gazes are directed to each area in the virtual space to map the gaze density.

[0025] The gaze statistical information may be information on the frequency of gaze directed to an object arranged in a virtual space. For example, when an ID is set to a rendered object, the statistical information acquisition unit 126 can acquire gaze statistical information by tallying up the frequency of the ID of an object to which the user's gaze is directed.

[0026] The gaze statistical information may be information on the frequency with which the gaze is directed to a divided area obtained by applying semantic area division to an image of a virtual space displayed to a user. For example, the statistical information acquiring unit 126 can acquire the gaze statistical information by tallying up the frequency of semantic labels corresponding to semantic areas to which the gaze of the user is directed.

[0027] The gaze statistical information may be recorded in association with the elapsed time from the start of content such as a virtual tour, or with an event such as a tour guide explaining a specific object. It is expected that the object or area to which the user directs his or her gaze will change depending on the elapsed time from the start of the content or the timing of the event. Therefore, the statistical information acquisition unit 126 acquires time-series gaze statistical information according to the time or the event, and the drawing unit 125 can appropriately determine the area to perform high-definition rendering based on the time-series gaze statistical information.

[0028] The statistical information acquisition unit 126 acquires time-series gaze statistical information based on the gaze direction of the user viewing the image in the virtual space, the virtual position of the user in the virtual space, and the time at which the gaze direction was detected. The time-series gaze statistical information may be acquired, for example, at predetermined time intervals. By using the time-series gaze statistical information acquired based on time, the drawing unit 125 can appropriately determine an area for high-definition rendering according to the elapsed time of the content.

[0029] In addition, the statistical information acquisition unit 126 acquires gaze statistical information for each event based on the gaze direction of the user viewing the image in the virtual space, the virtual position of the user in the virtual space, and information on an event occurring when the gaze direction is detected. By using the gaze statistical information for each event acquired based on the event, the drawing unit 125 can appropriately determine an area for high-definition rendering in accordance with an event occurring during viewing.

[0030] Note that the gaze statistical information is not limited to gaze information of the user wearing display device 110, and may be generated using gaze information of other users who have viewed the same content in the past.

[0031] 1 are merely examples and are not limited thereto. For example, the display device 110 may have a configuration similar to that of the statistical information acquisition unit 126. In this case, the image processing device 120 can receive gaze statistical information from the display device 110 and render an image to be displayed on the display unit 115 using the received gaze statistical information.

[0032] The process of the image processing system 100 that renders an image of content for experiencing a virtual space such as a virtual tour will be described with reference to Figures 3(A) and 3(B). Figure 3(A) is a flowchart illustrating the process of the display device 110 worn by a user who watches the content. Figure 3(B) is a flowchart illustrating the process of the image processing device 120 that transmits an image of the rendered content to each of the display devices 110 worn by a plurality of users who watch the content.

[0033] 3A, the processing of the display device 110 will be described. In step S311, the position and orientation acquisition unit 116 acquires information on the position and orientation of the user or the display device 110. The position and orientation acquisition unit 116 can acquire the information on the position and orientation of the display device 110 by using a gyro sensor built into the display device 110, for example.

[0034] The position and orientation acquisition unit 116 may acquire information on the position and orientation of the user or the display device 110 obtained by an external device. The information on the position and orientation of the user or the display device 110 can be obtained by, for example, analyzing an image captured by an external camera. The information on the position and orientation is , can also be determined by attaching motion capture markers to the user or to the display device 110 and tracking the position of the markers.

[0035] In step S312, gaze direction acquisition unit 117 acquires information on the gaze direction of the user. Gaze direction acquisition unit 117 can estimate the gaze direction of the user by using a gaze detection camera built into display device 110. Note that gaze direction acquisition unit 117 may estimate the direction in which the user's face is facing from the position and orientation acquired in step S311, and acquire the estimated direction as information on the gaze direction.

[0036] In step S313, the control unit 111 transmits to the image processing device 120 via the communication unit 114 the position and orientation information acquired in step S311 and the line of sight direction information acquired in step S312.

[0037] In step S314, the control unit 111 determines whether or not the image processing device 120 has received a rendered image that has been rendered in step S322 (described later). If the rendered image has been received, the process proceeds to step S315. If the rendered image has not been received, the process of step S314 is repeated until the rendered image is received. In step S315, the control unit 111 displays the rendered image received in step S314 on the display unit 115.

[0038] The processing of the image processing device 120 will be described with reference to Fig. 3(B). In step S321, the control unit 121 determines whether or not the information transmitted from the display device 110 in step S313 has been received via the communication unit 124. If the information has been received from the display device 110, the processing proceeds to step S322. If the information has not been received from the display device 110, the processing of step S321 is repeated until the information is received.

[0039] In step S322, drawing unit 125 performs rendering processing (drawing processing) of an image to be displayed to a user wearing display device 110. In order to reduce the load of the rendering processing, drawing unit 125 controls the method of rendering processing of an image in the virtual space for each area in the virtual space, based on the gaze statistical information acquired by statistical information acquisition unit 126.

[0040] Specifically, the drawing unit 125 performs rendering processing by controlling image quality (detail level) for each region based on gaze statistical information. For example, the drawing unit 125 can control to render with higher resolution an attention region to which gazes are directed more frequently than other objects. An attention region is an area worth looking at by a user, such as an area that attracts attention from users and an area that is popular with users in a virtual space, and is an area to which gazes are directed more frequently than other areas. An attention region may be an area of ​​a specific object.

[0041] 2(A), when it is known from the gaze statistics information that the virtual object 201 is a popular object in the virtual space 200, the drawing unit 125 may draw the virtual object 201 so that the image quality of the virtual object 201 is higher than the image quality of the virtual object 202. When it is known from the gaze statistics information that the virtual object 202 is an unpopular object, the drawing unit 125 may draw the virtual object 202 so that the image quality of the virtual object 202 is lower than the image quality of the virtual object 201.

[0042] The drawing unit 125 can determine how to control the image quality for each region and perform rendering, depending on the method of collecting gaze statistical information. In the example of FIG. 2(A), gaze information is collected for each virtual object, so the drawing unit 125 controls the image quality for each object and performs rendering processing. Also, if the gaze statistical information is information collected for each of a predetermined number of regions obtained by dividing a virtual space into equal sizes, for example, the drawing unit 125 controls the image quality for each of the predetermined number of regions of equal size and performs rendering processing. can be done.

[0043] Furthermore, in order to prevent a delay in the rendering process even if there is a sudden change in the field of view, the drawing unit 125 can pre-render (draw) objects or areas outside the field of view (outside the display range) of the user viewing the image of the virtual space on the display device 110. Pre-rendering objects or areas outside the display range is hereinafter referred to as pre-rendering. The drawing unit 125 can timely respond to changes in the field of view and suppress an increase in the processing load by controlling whether or not to perform pre-rendering according to the line-of-sight statistics information.

[0044] 2(A), when it is known from the gaze statistics information that the virtual object 201 is a popular object in the virtual space 200, the drawing unit 125 may prepare for a sudden change in the field of view by performing preliminary rendering even if the virtual object 201 is outside the display range. In addition, when it is known from the gaze statistics information that the virtual object 202 is an unpopular object, the drawing unit 125 may not perform preliminary rendering on the virtual object 202 in order to reduce the processing load.

[0045] The gaze statistical information may be generated based on gaze information of a user who has viewed (experienced) the same content in the past, not limited to gaze information of a user who is currently viewing (experiencing) the content. Also, the gaze statistical information may be generated based on the gaze information of the same user in the past, not using the gaze information of all past users.

[0046] Furthermore, the statistical information acquisition unit 126 may estimate in advance users whose attributes correspond (are similar) to those of the user currently viewing the content, and generate gaze statistical information based on past gaze information of the users whose attributes correspond. The user attributes include, for example, the user's gender, age, hobbies, etc., and are stored together with the gaze statistical information when the user views the content. For example, the statistical information acquisition unit 126 can determine that the attributes of the users correspond when more than a predetermined number of attributes out of a plurality of attributes match.

[0047] In step S323, the control unit 121 transmits the rendering image drawn in step S322 to the display device 110 via the communication unit .

[0048] In step S324, the statistical information acquisition unit 126 acquires gaze statistical information using the position and orientation information and gaze direction information received from the display device 110 in step S313. In step S325, the statistical information acquisition unit 126 records the gaze statistical information acquired in step S324 in a storage device or the like. Through the processes of steps S324 and S325, the statistical information acquisition unit 126 can generate gaze statistical information or update existing gaze statistical information based on the gaze information of the user currently viewing the content.

[0049] The gaze statistical information acquisition process in step S324 and the gaze statistical information recording process in step S325 may be performed offline after the user finishes viewing the content. The image processing device 120 can reduce the processing load by performing the gaze statistical information acquisition process and recording process offline. The reduced processing load allows the user to have a comfortable virtual experience. When the gaze statistical information acquisition process and recording process are performed offline, in step S322, the drawing unit 125 performs rendering process using gaze statistical information based on past gaze information.

[0050] In the above-described first embodiment, the image processing device 120 acquires gaze statistical information collected for each area in the virtual space. The image processing device 120 controls the method of rendering the image of the virtual space for each area based on the gaze statistical information, thereby rendering the attention area with high definition. In addition, by lowering the image quality of the non-attention area, the image processing device 120 can reduce the processing load while rendering the valuable area (attention area) with high resolution.

[0051] Moreover, the image processing device 120 controls whether or not to perform preliminary rendering in accordance with the gaze statistics information, thereby enabling the image processing device 120 to flexibly respond to changes in the user's field of view and improve the processing speed.

[0052] <Embodiment 2> In the first embodiment, the image processing device 120 transmits a rendered image to the display device 110 worn by each user, and the display device 110 displays the received image. In contrast, in the second embodiment, the image processing device 420 transmits data such as a three-dimensional model for generating an image of a virtual space to the display device 410, which is a user terminal, and the display device 410 renders and displays the received data. That is, in the first embodiment, the rendering process is executed by the image processing device 120, whereas in the second embodiment, the rendering process is executed by the display device 410. The display device 410 according to the second embodiment is an example of an electronic device.

[0053] Detailed description of the same processes as those in the first embodiment will be omitted. Also, the same components as those in the first embodiment will be denoted by the same reference numerals and detailed description thereof will be omitted. The main features of the second embodiment will be described below.

[0054] 4 is a diagram showing an example of the configuration of an image processing system 400 according to the second embodiment. The image processing system 400 includes a display device 410 and an image processing device 420. The display device 410 and the image processing device 420 are connected to each other so as to enable data communication. The display device 410 and the image processing device 420 may be connected to each other via a wired or wireless connection.

[0055] The display device 410 is a small terminal such as an HMD, an HHD, or a smartphone. The display device 410 includes a control unit 111, a ROM 112, a RAM 113, a communication unit 114, a display unit 115, a position and orientation acquisition unit 116, a gaze direction acquisition unit 117, and a drawing unit 418. The configuration of the display device 410 is the same as that of the first embodiment except for the drawing unit 418. The drawing unit 418 generates an image to be displayed on the display unit 115 by performing rendering processing on data received from the image processing device 420. The gaze statistics information used by the drawing unit 418 for the rendering processing may be received from the image processing device 420 together with the content data, or may be stored in advance in a storage device of the display device 410.

[0056] The image processing device 420 is, for example, a PC. The image processing device 420 includes a control unit 121, a ROM 122, a RAM 123, a communication unit 124, a data processing unit 425, and a statistical information acquisition unit 126. The configuration of the image processing device 420 is the same as that of the first embodiment except for the data processing unit 425. The data processing unit 425 processes data to be transmitted to the display device 410 based on the gaze statistical information.

[0057] 5(A) and 5(B), the process of the image processing system 400 that renders images of content for experiencing a virtual space such as a virtual tour will be described. The image processing device 420 transmits data such as a three-dimensional model for generating an image of the virtual space to the display device 410. The display device 410 performs rendering processing on the received data within the display device 410.

[0058] 5A is a flowchart illustrating the process of the display device 410 worn by a user who is viewing the content. 11 is a flowchart illustrating the process of an image processing device 420 that transmits data for generating an image of a virtual space to each of the display devices 410.

[0059] The processing of the display device 410 will be described with reference to Fig. 5(A). In step S511, the control unit 111 determines whether or not the image processing device 420 has received processed data processed in step S521 described below. If the processed data has been received, the processing proceeds to step S512. If the processed data has not been received, the processing of step S511 is repeated until the processed data is received.

[0060] The processes in steps S512 and S513 are the same as those in steps S311 and S312 of the display device 110 described in Fig. 3A. In steps S512 and S513, the display device 410 acquires information on the position and orientation of the user or the display device 110, and information on the line of sight of the user.

[0061] The process of step S514 is the same as the process of step S322 of the image processing device 120 described in FIG. 3B. That is, the rendering process (drawing process) performed by the image processing device 120 in the first embodiment is performed by the display device 410 in the second embodiment. The drawing unit 418 of the display device 410 performs the rendering process of the image to be displayed to the user wearing the display device 410, similar to step S322 in FIG. 3B. The drawing unit 418 controls the method of rendering the image of the virtual space for each area in the virtual space based on the gaze statistics information, thereby drawing a valuable area in the content with high resolution and reducing the processing load. The drawing unit 418 may use the gaze statistics information stored in advance in a storage device provided in the display device 410, or may use the gaze statistics information received from the image processing device 420.

[0062] The process of step S515 is the same as the process of step S315 of the display device 110 described with reference to Fig. 3A. The control unit 111 displays on the display unit 115 the rendering image that has been subjected to the rendering process in step S514.

[0063] The process of step S516 is the same as the process of step S313 of the display device 110 described in Fig. 3(A). The control unit 111 transmits the position and orientation information acquired in step S512 and the line of sight direction information acquired in step S513 to the image processing device 420 via the communication unit 114.

[0064] The processing of the image processing device 420 will be described with reference to Fig. 5(B). In step S521, the data processing unit 425 performs processing to reduce the amount of data such as a three-dimensional model in the virtual space based on gaze statistical information previously generated using gaze information of users who have viewed the same content. Since the processed data is transmitted to the display device 410, by reducing the amount of processed data to be transmitted, the user can experience the content in the virtual space more comfortably.

[0065] The data processing unit 425 may reduce the amount of data by, for example, processing the number of polygons of the three-dimensional model data based on the gaze statistics information. The data processing unit 425 can reduce the amount of communication data by reducing the number of polygons of objects that are not often gazed upon while maintaining the number of polygons of popular objects.

[0066] In the example of FIG. 2A, when it is known from the line-of-sight statistics information that the virtual object 201 is a popular object in the virtual space 200, the drawing unit 125 may make the number of polygons of the virtual object 201 greater than those of the other objects. Also, when it is known from the line-of-sight statistics information that the virtual object 202 is an unpopular object, the drawing unit 125 may make the number of polygons of the virtual object 202 greater than those of the other objects. The polygon count of the body 202 may be reduced.

[0067] The data processing unit 425 may also change the type of model of the three-dimensional model data based on, for example, gaze statistical information. The data processing unit 425 may change an object that is not often gazed at to a model in which a texture is simply pasted on a flat surface, or to a simplified model in which the texture is expressed by a known method such as bump mapping.

[0068] In the example of Fig. 2(A), when it is found from the gaze statistics information that the virtual object 202 is an unpopular object in the virtual space 200, the virtual object 202 may be changed to a simplified model. By changing the unpopular object to a simplified model in the data to be transmitted to the display device 410, the data processing unit 425 can reduce not only the amount of communication data but also the load of the rendering process on the display device 410. The data processing unit 425 can appropriately control the load of the rendering process on the display device 410 by changing the type of model of the three-dimensional model data.

[0069] In step S522, the control unit 121 transmits the processed data processed in step S521 to the display device 410 via the communication unit 124. For example, when transmitting the processed data to the display device 410 of the user who views the content, the control unit 121 may control the order of transmission to the display device 410 for each area based on the gaze statistics information.

[0070] 2(A), when it is known from the gaze statistical information that the virtual object 201 is a popular object in the virtual space 200, the control unit 121 may preferentially transmit data of the virtual object 201 to the display device 410. Even if a delay occurs in communication, the control unit 121 can reduce stress on the user experiencing the virtual space by preferentially transmitting the popular object to the display device 410.

[0071] The process of step S523 is the same as the process of step S321 of the image processing device 120 described in Fig. 3(B). In step S523, the control unit 121 determines whether or not the information transmitted from the display device 410 in step S516 has been received via the communication unit 124. If the information has been received from the display device 410, the process proceeds to step S524. If the information has not been received from the display device 410, the process of step S523 is repeated until the information is received.

[0072] The processes of steps S524 and S525 are the same as the processes of steps S324 and S325 of the image processing device 120 described in Fig. 3(B) respectively. In steps S524 and S525, the statistical information acquisition unit 126 acquires gaze statistical information and records the acquired gaze statistical information in a storage device or the like.

[0073] In the above-mentioned second embodiment, the image processing device 420 acquires gaze statistical information collected for each area in the virtual space. The image processing device 420 transmits data such as a three-dimensional model for generating an image of the virtual space to the display device 410 worn by the user. The display device 410 controls the method of rendering processing of the data received from the image processing device 420 for each area based on the gaze statistical information. That is, the display device 410 renders the attention area with high resolution and renders the non-attention area with lower image quality. As a result, even when the rendering processing is performed within the display device 410, the display device 410 can suppress the processing load and improve the processing speed while drawing a valuable area (attention area) with high resolution.

[0074] The image processing device 420 may process data such as a three-dimensional model for generating an image of a virtual space based on gaze statistics information. By processing the data, such as reducing the number of polygons in the image or changing the three-dimensional model to a simpler model, the image processing device 420 can reduce the amount of data to be transmitted to the display device 410.

[0075] Furthermore, the image processing device 420 may control the order of data to be transmitted to the display device 410 based on the gaze statistics information. The image processing device 420 transmits data of the attention area to the display device 410 with priority, so that the user can view the valuable area (attention area) without delay.

[0076] <Embodiment 3> The third embodiment is an embodiment in which an image captured by an image processing device 620, which is a 360-degree camera (omnidirectional camera), is transmitted to a display device 410, which is a user terminal, and an image is displayed on the display device 410 according to the orientation of the user.

[0077] Detailed description of the same processes as those in the first or second embodiment will be omitted. Also, the same configurations as those in the first or second embodiment will be denoted by the same reference numerals and detailed description thereof will be omitted. The main features of the third embodiment will be described below.

[0078] 6 is a diagram showing an example of the configuration of an image processing system 600 according to the third embodiment. The image processing system 600 includes a display device 410 and an image processing device 620. The display device 410 and the image processing device 620 are connected so as to enable data communication. The display device 410 and the image processing device 620 may be connected by wire or wirelessly. The display device 410 is the same as that in the second embodiment, and therefore a description thereof will be omitted.

[0079] The image processing device 620 is a 360-degree camera or the like, and includes a control unit 121, a ROM 122, a RAM 123, a communication unit 124, a shooting data processing unit 625, a statistical information acquisition unit 126, an optical system 627, and an imaging unit 628. The configuration of the image processing device 620 is the same as that of the first embodiment, except for the shooting data processing unit 625, the optical system 627, and the imaging unit 628.

[0080] The photographing data processing unit 625 processes the image captured by the imaging unit 628. The optical system 627 is, for example, a fisheye lens, and forms an image of the surroundings on the imaging unit 628. The imaging unit 628 is, for example, an imaging element such as a CCD or CMOS sensor, and photoelectrically converts the optical image formed on the imaging element by the optical system 627, and outputs the obtained signal to the RAM 123 for storage.

[0081] Using Figures 7(A) and 7(B), we will explain the processing of the image processing system 600, which transmits an image captured by the image processing device 620 to the display device 410 and displays an image on the display device 410 according to the user's orientation.

[0082] Fig. 7(A) is a flowchart illustrating the processing of the display device 410 worn by a user who is viewing the content. Fig. 7(B) is a flowchart illustrating the processing of the image processing device 620 that processes and transmits data captured by a 360-degree camera to each of the display devices 410 worn by multiple users who are viewing the content.

[0083] The processing of the display device 410 will be described with reference to FIG. 7(A). In step S711, the control unit 111 determines whether or not the image processing device 620 has received processed data processed in step S722 described below. If processed data has been received, the processing proceeds to step S712. If processed data has not been received, the processing of step S711 is repeated until processed data is received. The processing of steps S712 to S716 is the same as the processing of steps S512 to S516 of the display device 410 described with reference to FIG. 5(A), respectively.

[0084] 7B, the process of the image processing device 620 will be described. In step S721, the optical system 627 and the imaging unit 628 capture images in all directions of 360 degrees by combining a plurality of lenses and a plurality of imaging sensors.

[0085] In step S722, the image data processing unit 625 processes the image data captured in step S721. For example, the image data processing unit 625 generates a 360-degree image by synthesizing a plurality of images captured in different shooting directions using a combination of a plurality of lenses and a plurality of image sensors. The image data processing unit 625 performs image processing such as alignment and brightness correction to eliminate the sense of incongruity of the joints of the synthesis process.

[0086] The photographed data processing unit 625 may control the load of image processing for synthesizing a plurality of photographed images for each region based on, for example, gaze statistics information. Specifically, the photographed data processing unit 625 may control the priority and processing performance of image processing for each region. The photographed data processing unit 625 can reduce the processing load by performing high-load image processing on the region where the user's gaze is concentrated and omitting at least a part of the image processing on the other regions.

[0087] 2(A), when it is known from the gaze statistics information that the area in the virtual space 200 where the virtual object 201 exists is popular, the photographed data processing unit 625 may perform high-load image processing on the area where the virtual object 201 exists. When it is known from the gaze statistics information that the area where the virtual object 202 exists is not popular, the photographed data processing unit 625 may not perform image processing on the area where the virtual object 202 exists, or may omit some of the processing.

[0088] Furthermore, the photographed data processing unit 625 may control the compression rate for compressing the 360-degree image for each region, for example, based on gaze statistical information. In the example of Fig. 2(A), if it is known from gaze statistical information that the region in the virtual space 200 where the virtual object 202 is located is not popular, the compression rate of the photographed data for the region where the virtual object 202 is located may be made higher than the compression rate for other regions. By setting a higher compression rate for the region where the user does not often direct his or her gaze, the photographed data processing unit 625 can reduce the amount of data transmitted from the image processing device 620 to the display device 410 while maintaining the image quality in the region of interest.

[0089] In step S723, the control unit 121 transmits the processed data processed in step S722 to the display device 410 via the communication unit .

[0090] The process of step S724 is the same as the process of step S523 of the image processing device 420 described in Fig. 5(B). In step S724, the control unit 121 determines whether or not the information transmitted from the display device 410 to the image processing device 620 in step S716 has been received via the communication unit 124. If the information has been received from the display device 410, the process proceeds to step S725. If the information has not been received from the display device 410, the process of step S724 is repeated until the information is received.

[0091] The processes of steps S725 and S726 are the same as the processes of steps S524 and S525 of the image processing device 420 described in Fig. 5B. In steps S725 and S726, the statistical information acquisition unit 126 acquires gaze statistical information and records the acquired gaze statistical information in a storage device or the like.

[0092] In the above-mentioned third embodiment, the image processing device 620 acquires gaze statistical information collected for each area in the virtual space. The image processing device 620 processes the captured data captured by the 360-degree camera and transmits it to the display device 410 worn by the user. The display device 410 controls the method of rendering processing of the data received from the image processing device 420 for each area based on the gaze statistical information. As a result, the display device 410 performs rendering within the display device 410. Even when processing is performed, it is possible to render a valuable area (area of ​​interest) in high definition while suppressing the processing load and improving the processing speed.

[0093] The image processing device 620 may also process the captured data based on the gaze statistics information. The image processing device 620 can reduce the processing load by performing high-load image processing in the region of interest and omitting part of the image processing in regions other than the region of interest. The image processing device 620 can reduce the amount of data to be transmitted while maintaining the image quality in the region of interest by setting the compression rate of the captured data in the region of interest higher than that in other regions.

[0094] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made to the configurations of the above-mentioned embodiments within the scope of the gist of the present invention. In addition, the configurations of the above-mentioned embodiments can be appropriately combined.

[0095] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring gaze statistical information indicating the extent to which the gaze is directed to each area in the virtual space; a control means for controlling a method of rendering a virtual space image for each of the areas based on the line-of-sight statistics information; 1. An electronic device comprising: (Configuration 2) The acquisition means generates the gaze statistical information based on a gaze direction of a user viewing the image and a virtual position of the user in the virtual space. 2. The electronic device according to configuration 1. (Configuration 3) The control means controls the image quality of the image for each of the regions based on the line-of-sight statistics information. 3. The electronic device according to configuration 1 or 2. (Configuration 4) Before performing the drawing process, a processing process for reducing the amount of data of the image can be performed, The control means controls the processing for each of the regions based on the line-of-sight statistics information. 4. The electronic device according to configuration 3. (Configuration 5) The control means controls a compression rate at which the image is compressed for each of the regions based on the line-of-sight statistics information when the image is an image captured by a camera. 4. The electronic device according to configuration 3. (Configuration 6) The control means controls whether or not to render in advance an area outside the range of a visual field of a user viewing the image, based on the line-of-sight statistics information. 6. The electronic device according to any one of configurations 1 to 5. (Configuration 7) The control means, when transmitting the data of the image to the display device, determines whether the image is displayed based on the line-of-sight statistics information. and controlling the order of transmission to the display device for each of the areas. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) When the image is generated by synthesizing a plurality of captured images captured in different shooting directions, the control means controls a load of image processing for synthesizing the plurality of captured images for each of the regions based on the line-of-sight statistics information. 8. The electronic device according to any one of configurations 1 to 7. (Configuration 9) The gaze statistical information is a gaze frequency map indicating a frequency at which the gaze is directed to each of the regions. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 10) The gaze statistical information is a gaze density map indicating a weight according to the number of users to whom the gaze is directed for each of the regions. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 11) The gaze statistical information is information on the frequency with which the gaze is directed to an object placed in the virtual space. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 12) The gaze statistics information is information on the frequency with which the gaze is directed to a segmented region obtained by applying semantic region segmentation to the image. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 13) the acquiring means acquires the gaze statistical information in time series based on a gaze direction of a user viewing the image, a virtual position of the user in the virtual space, and a time at which the gaze direction was detected; The control means controls a drawing process of the image for each of the regions based on the time-series line-of-sight statistical information. 13. The electronic device according to any one of configurations 1 to 12. (Configuration 14) the acquiring means acquires the gaze statistical information for each event based on a gaze direction of a user viewing the image, a virtual position of the user in the virtual space, and information on an event occurring when the gaze direction is detected; The control means controls the drawing process of the image for each of the areas based on the line-of-sight statistical information for each of the events. 14. The electronic device according to any one of configurations 1 to 13. (Configuration 15) The gaze statistical information is generated based on past gaze information of other users whose attributes correspond to those of the user viewing the image. 15. The electronic device according to any one of configurations 1 to 14. (Configuration 16) A communication means for communicating with a display device including a detection means for detecting a line of sight of a user viewing the image and a display means for displaying an image of the virtual space, A method of drawing an image displayed on the display means is controlled for each of the areas via the communication means. 16. The electronic device according to any one of configurations 1 to 15. (Configuration 17) a display means for displaying an image of the virtual space, The control means controls the display means to control a method of the drawing process for each of the areas. Control 16. The electronic device according to any one of configurations 1 to 15. (method) An acquisition step of acquiring gaze statistical information representing the extent to which the gaze is directed to each area in the virtual space; a control step of controlling a method of rendering processing of the image in the virtual space for each of the regions based on the gaze statistics information; 13. A method for controlling an electronic device comprising: (program) A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 17. [Explanation of symbols]

[0097] 110: display device, 111: control unit, 120: image processing device, 121: control unit

Claims

1. An acquisition means for acquiring gaze statistical information indicating the extent to which the gaze is directed to each area in the virtual space; a control means for controlling a method of rendering a virtual space image for each of the areas based on the line-of-sight statistics information; 1. An electronic device comprising:

2. The acquisition means generates the gaze statistical information based on a gaze direction of a user viewing the image and a virtual position of the user in the virtual space.

2. The electronic device according to claim 1 .

3. The control means controls the image quality of the image for each of the regions based on the line-of-sight statistics information.

2. The electronic device according to claim 1 .

4. Before performing the drawing process, a processing process for reducing the amount of data of the image can be performed, The control means controls the processing for each of the regions based on the line-of-sight statistics information.

4. The electronic device according to claim 3.

5. The control means controls a compression rate at which the image is compressed for each of the regions based on the line-of-sight statistics information when the image is an image captured by a camera.

4. The electronic device according to claim 3.

6. The control means controls whether or not to render in advance an area outside the range of a visual field of a user viewing the image, based on the line-of-sight statistics information.

2. The electronic device according to claim 1 .

7. The control means controls an order of transmission of the image data to the display device for each of the regions based on the line-of-sight statistics information when transmitting the image data to the display device.

2. The electronic device according to claim 1 .

8. When the image is generated by synthesizing a plurality of captured images captured in different shooting directions, the control means controls a load of image processing for synthesizing the plurality of captured images for each of the regions based on the line-of-sight statistics information.

2. The electronic device according to claim 1 .

9. The gaze statistical information is a gaze frequency map indicating a frequency at which the gaze is directed to each of the regions.

2. The electronic device according to claim 1 .

10. The gaze statistical information is a gaze density map indicating a weight according to the number of users to whom the gaze is directed for each of the regions.

2. The electronic device according to claim 1 .

11. The gaze statistical information is information on the frequency with which the gaze is directed to an object placed in the virtual space.

2. The electronic device according to claim 1 .

12. The gaze statistics information is information on the frequency with which the gaze is directed to a segmented region obtained by applying semantic region segmentation to the image.

2. The electronic device according to claim 1 .

13. the acquiring means acquires the gaze statistical information in time series based on a gaze direction of a user viewing the image, a virtual position of the user in the virtual space, and a time at which the gaze direction was detected; The control means controls a drawing process of the image for each of the regions based on the time-series line-of-sight statistical information.

2. The electronic device according to claim 1 .

14. the acquiring means acquires the gaze statistical information for each event based on a gaze direction of a user viewing the image, a virtual position of the user in the virtual space, and information on an event occurring when the gaze direction is detected; The control means controls the drawing process of the image for each of the areas based on the line-of-sight statistical information for each of the events.

2. The electronic device according to claim 1 .

15. The gaze statistical information is generated based on past gaze information of other users whose attributes correspond to those of the user viewing the image.

2. The electronic device according to claim 1 .

16. A communication means for communicating with a display device including a detection means for detecting a line of sight of a user viewing the image and a display means for displaying an image of the virtual space, A method of drawing an image displayed on the display means is controlled for each of the areas via the communication means.

2. The electronic device according to claim 1 .

17. a display means for displaying an image of the virtual space, The control means controls the display means to control the method of the drawing process for each of the areas.

2. The electronic device according to claim 1 .

18. An acquisition step of acquiring gaze statistical information representing the extent to which the gaze is directed to each area in the virtual space; a control step of controlling a method of rendering processing of the image in the virtual space for each of the regions based on the gaze statistics information; 13. A method for controlling an electronic device comprising:

19. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 17.

Images