Information processing device, control method for information processing device, and program

Abstract

An information processing device (10) generates a screen including an image to be operated by a user, and acquires line-of-sight position information representing a position on the generated image of the line of sight of the user who performs an operation while looking at the screen including the image. Then, on the basis of the acquired line-of-sight position information of the user, the information processing apparatus 10 generates output information including the generated image and an additional image superimposed on the generated image, the output information being used in a predetermined process.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, in virtual reality (VR) technology, there have been examples where gaze information of a user viewing a VR video is acquired and used to generate the VR video. Summary of the Invention [Problem to be solved by the invention]

[0003] However, the current situation is that such user gaze information acquired for generating an image is used solely for the purpose of generating the image, and is not effectively utilized for other purposes.

[0004] The present invention has been made in consideration of the above-mentioned situation, and one of its objectives is to provide an information processing device, a control method for an information processing device, and a program that can effectively utilize user gaze information acquired when generating images such as VR. [Means for solving the problem]

[0005] An information processing device according to one aspect of the present invention includes a screen generation means for generating a screen including an image to be operated by a user, an acquisition means for acquiring gaze position information representing a position on the generated image, the position of the gaze of a user who is looking at a screen including the image and performing an operation, and an output information generation means for generating output information including additional information representing the acquired gaze position information of the user and the image included on the screen, and the output information is subjected to a predetermined process. [Effects of the Invention]

[0006] According to the present invention, it is possible to effectively utilize the user's line of sight information. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram illustrating an example of the configuration of an information processing device according to an embodiment of the present invention. [Figure 2] 1 is a functional block diagram illustrating an example of an information processing device according to an embodiment of the present invention. [Figure 3] 1 is an explanatory diagram illustrating an example of a composite image generated by an information processing device according to an embodiment of the present invention. [Figure 4] FIG. 10 is an explanatory diagram illustrating another example of a composite image generated by the information processing device according to the embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram illustrating yet another example of a composite image generated by the information processing device according to the embodiment of the present invention. [Figure 6] FIG. 2 is a flowchart illustrating an example of the operation of the information processing device according to the embodiment of the present invention. [Figure 7] 1 is a functional block diagram illustrating an example of a system including an information processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment of the present invention will be described with reference to the drawings. An information processing device 10 according to the embodiment of the present invention is, for example, a personal computer or a home game console, and includes a control unit 11, a storage unit 12, an operation acquisition unit 13, a display control unit 14, and a communication unit 15, as shown in Fig. 1, and is connected to a controller device 20, a display device 30, and a gaze detection device 40.

[0009] The control unit 11 is a program-controlled device such as a CPU, and operates according to a program stored in the storage unit 12. In this embodiment, the control unit 11 generates and displays a screen including an image to be operated by the user, and also acquires from the gaze detection device 40 information indicating the gaze position of the user who is operating the screen including the image, which is located on the generated image.

[0010] Furthermore, based on user instructions and settings, the control unit 11 generates output information including additional information representing the acquired information on the user's gaze position and an image included in the screen operated by the user. In this example, the additional information is an additional image that is superimposed on the generated image based on the acquired information on the user's gaze position, and in this example, the control unit 11 generates output information including the additional image and the image included in the screen operated by the user. The control unit 11 then provides the generated output information for predetermined processing such as distribution. The detailed content of the processing by the control unit 11 will be described later.

[0011] The storage unit 12 is configured to include a storage device such as a disk device or a memory device. A program executed by the control unit 11 is stored in this storage unit 12. This program may be provided by being stored on a computer-readable, non-transitory recording medium and stored in this storage unit 12. The storage unit 12 also operates as a work memory for the control unit 11.

[0012] The operation acquisition unit 13 receives the content of the user's operation from the controller device 20 and outputs it to the control unit 11. The display control unit 14 is connected to the display device 30, and outputs the instructed image to the display device 30 in accordance with the instruction input from the control unit 11.

[0013] The communication unit 15 is a network interface or the like, and sends data such as output information to other information processing devices 10, server devices (not shown), etc. connected via the network in accordance with instructions input from the control unit 11. The communication unit 15 also outputs data received from other information processing devices 10, server devices, etc. via the network to the control unit 11.

[0014] The controller device 20 is a game controller, a mouse, a keyboard, etc., and outputs information representing operations performed by the user to the information processing device 10. The display device 30 is a display, a head mounted display (HMD), etc., and presents the specified image to the user in accordance with instructions input from the information processing device 10.

[0015] The gaze detection device 40 detects the gaze of a user who performs an operation related to an image displayed by the display device 30 while viewing the image, and outputs information (gaze position information) that identifies the part of the image that is in the user's gaze to the information processing device 10. The gaze position information may take any form as long as it is information that can indicate the position on the target image that is in the user's gaze.

[0016] In one example of the present embodiment, the gaze detection device 40 may be disposed inside the HMD and output gaze position information that indicates the gaze position of the user on the image presented by the HMD, or may be attached to the display, equipped with a camera that captures an image of the user, and output gaze position information that indicates the gaze position of the user on the image presented by the display based on the relative position of the user's pupils and the display. Since widely known gaze detection devices can be used as these gaze detection devices 40, detailed description thereof will be omitted here.

[0017] Next, a description will be given of the operation of the control unit 11 of the information processing device 10. In this embodiment, the control unit 11 executes a program stored in the storage unit 12, thereby realizing a configuration that functionally includes a screen generation unit 21, a screen display control unit 22, a line-of-sight information acquisition unit 23, a position determination unit 24, and an output information generation unit 25, as exemplified in FIG.

[0018] Screen generation unit 21 generates a screen including an image that is the target of user operation, using information indicating the content of the user operation input from operation acquisition unit 13. Here, if control unit 11 executes a game application program, for example, the image that is the target of user operation is a sequentially changing game image, and the user operates a character or the like displayed in the image. If control unit 11 executes an application program that plays videos or the like, the image that is the target of user operation is the video, and screen generation unit 21 performs processing such as playing or stopping the video based on the operation to play or stop the video input by the user.

[0019] The screen display control unit 22 sequentially outputs the target images sequentially generated by the screen generation unit 21 to the display device 30 via the display control unit 14, and causes the display device 30 to display the images.

[0020] The gaze information acquisition unit 23 acquires gaze position information from the gaze detection device 40, which represents the position on the image (hereinafter referred to as the target image) generated by the screen generation unit 21 and displayed on the display device 30, and indicates the position of the gaze of the user who is looking at the screen including the target image and performing an operation.

[0021] When the user instructs to generate an image to which gaze position information is added based on the user's instructions and settings, the position determination unit 24 determines, based on the acquired user's gaze position information, the superimposition position of the additional image to be superimposed on the target image generated by the screen generation unit 21. As an example, the position determination unit 24 determines the range in which the additional image is to be placed, with the center of the circumscribing rectangle of the additional image (intersection of the diagonals) as the point on the target image represented by the gaze position information.

[0022] The output information generating unit 25 synthesizes the additional image in the range determined by the position determining unit 24 on the target image generated by the screen generating unit 21, and generates output information including the target image and the additional image.

[0023] As illustrated in FIG. 3, the output information generation unit 25 according to one example of the present embodiment superimposes an additional image (Q) representing a graphic of a predetermined shape (e.g., a cross, a rectangle, a circle, a star, etc.) on the target image (T) generated by the screen generation unit 21 in an area determined by the position determination unit 24, thereby generating output information as a composite image (M). In this example, the size of the composite image, which is the output information, is the same size as the target image generated by the screen generation unit 21 (the number of pixels in both the vertical and horizontal directions is the same). Note that the type of additional image to be composited may be selectable by the user, for example, or may be selected based on the content of the target image generated by the screen generation unit 21 without relying on human manipulation, as will be described later.

[0024] In this case, the output information generation unit 25 may apply a blurring process (such as Gaussian blur) to the image portion of the target image generated by the screen generation unit 21 that includes the range determined by the position determination unit 24, thereby improving the visibility of the additional image.

[0025] Furthermore, if part of the range determined by the position determination unit 24 extends outside the composite image, which is the output information, that is, if the point indicated by the gaze position information is on the periphery of the target image and part of the additional image would extend outside the target image if the additional image were to be composited as is, the output information generation unit 25 may control the additional image so that it does not extend outside the composite image, which is the output information, by reducing the size of the additional image and aligning its center with the center of the range determined by the position determination unit 24 before composite.

[0026] In another example, the output information generating unit 25 may change the content of the additional image based on the content of the target image. In one example of this embodiment, the output information generating unit 25 sets the range of the target image determined by the position determining unit 24 (or a range of a size that can include a circle or polygon circumscribing the additional image, centered on a point represented by the gaze position information) as the target portion, and changes the content of the additional image based on the content of a part of the target image (corresponding to a partial image in the present invention) included in this target portion.

[0027] Here, the content of the additional image is, for example, the color, size, shape, etc. of the additional image. As an example, the output information generation unit 25 calculates the average pixel value (or the most frequent pixel value) of the pixels of the target image included in the target portion, and by referring to the hue of this average pixel value, etc., sets the color of the additional image to the color of the hue farthest from the referenced hue (for example, a color that is complementary to the referenced hue) based on the value of the referenced hue.

[0028] According to this example, it is possible to prevent the additional image from becoming difficult to see due to being blended into the color of the target image.

[0029] [Gaze position information between frames] As already explained, the target images generated by the screen generation unit 21 do not need to be still images, but may be sequentially changing game images, moving images, etc. In this case, the gaze information acquisition unit 23 repeatedly acquires gaze position information, which is a position on each target image (frame image) sequentially generated by the screen generation unit 21 and represents the position of the gaze of the user who is performing an operation while looking at the screen including the target image, from the gaze detection device 40 at predetermined timings.

[0030] When the gaze information acquisition unit 23 acquires gaze position information, the output information generation unit 25 superimposes and synthesizes the additional image within the range determined by the position determination unit 24 based on the acquired gaze position information of the target image (frame image) being output by the screen display control unit 22 at that time, and generates output information as a composite image.

[0031] In this example, the output information generating unit 25 may also blur a predetermined range centered on the position where the additional image is to be combined with the target image, or may perform peripheral processing on the additional image. Furthermore, the output information generating unit 25 may also perform processing to change the content of the additional image based on the content of the target image.

[0032] [Changes in gaze position information] Here, if the timing at which the gaze information acquisition unit 23 acquires gaze position information from the gaze detection device 40 coincides with the timing at which each frame of the target image is displayed, gaze position information is acquired for each frame, and for each frame, the additional image is superimposed and synthesized in an area centered on the user's gaze position in the corresponding frame image.

[0033] However, the user's gaze position changes successively, and if the synthesis position of the additional image is determined by following the minute changes for each frame, the additional image will appear to fluctuate and will be difficult to view. Therefore, in one example of this embodiment, the position determination unit 24 may determine the synthesis range of the additional image based on the gaze position information acquired by the gaze information acquisition unit 23 for each predetermined N frames (N is an integer of 2 or more).

[0034] In this example, the position determination unit 24 determines the range in which to synthesize an additional image based on the gaze position information acquired by the gaze information acquisition unit 23 for the first frame, and then outputs the determined range as is for N frames including the frame. As a result, for the N frames including the frame, the output information generation unit 25 generates a composite image by synthesizing the additional image in the synthesis range determined for the first frame, and generates and outputs output information including a series of composite images corresponding to each frame.

[0035] Furthermore, at the timing when the target image is displayed from the first frame to the N+1th frame, the position determination unit 24 again determines the synthesis range of the additional image based on the gaze position information acquired by the gaze information acquisition unit 23. Then, information on the determined synthesis range is output over the next N frames including the frame in question. In this example of the present embodiment, the synthesis range of the additional image changes every N frames.

[0036] To explain this using a specific example, when N=3, for example, the position determination unit 24 determines the synthesis range of the additional image for the target image in the first, second, and third frames based on information about the gaze position of the user viewing the target image in the first frame. Then, the output information generation unit 25 synthesizes the additional image for the target image in the first, second, and third frames within the synthesis range determined for the first frame, thereby generating synthesized images corresponding to the first, second, and third frames. Next, the position determination unit 24 determines the synthesis range of the additional image for the target image in the fourth, fifth, and sixth frames based on information about the gaze position of the user viewing the target image in the fourth frame. Then, the output information generation unit 25 synthesizes the additional image for the target image in the fourth, fifth, and sixth frames within the synthesis range determined for the fourth frame, thereby generating synthesized images corresponding to the fourth, fifth, and sixth frames... This process is repeated in this manner. Since the position of the additional image is changed only once per frame, and the additional image does not change position frequently, the additional image does not appear to fluctuate, thereby improving its visibility.

[0037] Also, although the description has been given here assuming that the synthesis range of the additional image is determined based on gaze position information corresponding to the first frame among the N frames, the present embodiment is not limited to this example. For example, the position determination unit 24 may determine the synthesis range of the additional image based on user gaze position information acquired during display of target images in N (N is an integer of 2 or more) frames (for example, consecutive N frames including the most recent currently displayed frame) that have been displayed so far.

[0038] As a specific example, the position determination unit 24 may determine the synthesis range of the additional image for each N frames based on the average of the user's gaze position information during the N frames (while the N frame images are displayed).

[0039] In another example, the output information generation unit 25 may use information on the synthesis range determined by the position determination unit 24 for each of the N (N is an integer greater than or equal to 2) frames (for example, the N consecutive frames including the most recent currently displayed frame) that have been displayed so far, and synthesize multiple additional images with the currently displayed target image to obtain a composite image.

[0040] In this example, the output information generation unit 25 stores information on the past N synthesis ranges determined by the position determination unit 24. Then, for the target image of the (N+1)th frame (for example, when N=3, the third frame), information on the synthesis ranges corresponding to the most recent N (here, 3) frames is used to synthesize additional images into the synthesis ranges corresponding to the second frame, the third frame, and the fourth frame itself in the target image of the fourth frame. In this case, the output information generation unit 25 may synthesize additional images with higher brightness (i.e., lighter colors) for images to be synthesized at positions corresponding to gaze position information of past frames (FIG. 4: Q). The synthesis method may be semi-transparent synthesis, or the images may be synthesized in order starting from the past (if images overlap, the past images are obscured).

[0041] In yet another example, the position determination unit 24 may use user gaze position information acquired during the display of the target image of N (N is an integer greater than or equal to 2) frames (for example, N consecutive frames including the most recent, currently displayed frame) that have been displayed so far to determine, as the synthesis range, an area within a circle or polygon (called a circumscribing figure) that circumscribes the points represented by the N pieces of gaze position information corresponding to the N frames.

[0042] In this example, the output information generating unit 25 generates a composite image by combining a figure of a predetermined shape that includes the above-mentioned circumscribing figure as an additional image with the target image.

[0043] Here, the circumscribing figure may be a convex hull of points represented by N pieces of gaze position information, or may have a predetermined shape such as a circumscribing rectangle. According to this example, when the gaze movement is relatively large, an additional image with a relatively large area is synthesized accordingly, and when the gaze movement is relatively small, an additional image with a relatively small area is synthesized accordingly. In addition, the shape of the additional image also changes depending on the direction of gaze movement. For example, when the gaze moves significantly horizontally, a relatively horizontally long additional image is synthesized.

[0044] Furthermore, in the examples described so far, the position determination unit 24 is assumed to update the synthesis range, for example, every predetermined number of frames. However, instead, for example, when the difference between the previous gaze position information and the current gaze position information (the distance between the position indicated by the previous gaze position information and the position indicated by the current gaze position information) exceeds a predetermined threshold, the position determination unit 24 may update and output information on the range for synthesizing the additional image.

[0045] In this example, the synthesis range of the additional image changes only when the user's line of sight movement becomes relatively large.

[0046] [Adjusting additional images based on the content of the target image] Furthermore, the determination of the synthesis range of the additional image by the position determination unit 24 and the output information generation unit 25, or the mode of synthesis of the additional image, may be controlled based on the content of the target image generated by the screen generation unit 21.

[0047] In this example, screen generation unit 21 outputs information on the display range of an object included in the target image, such as the display range of a game character, and provides the information for use by position determination unit 24 and output information generation unit 25. Specifically, when control unit 11 is executing a game application, screen generation unit 21 outputs information (specific part designation information) to output information generation unit 25 that designates the display range of a game character to be drawn in the target image as a specific part.

[0048] In this case, the position determination unit 24 corrects the determined synthesis range when the synthesis range of the additional image determined by the above-described process overlaps with a specific portion specified by the information output by the screen generation unit 21. Examples of this correction include reducing the synthesis range or setting the peripheral portion of the display range as the synthesis range. When the position determination unit 24 corrects the synthesis range, it may output information indicating that the correction has been made, along with information specifying the synthesis range.

[0049] When the information specifying the synthesis range of the additional image includes information indicating that the synthesis range has been corrected, the output information generation unit 25 determines the mode of the additional image in accordance with the information on the corrected synthesis range. For example, when the position determination unit 24 outputs information indicating that the synthesis range has been corrected along with information specifying the synthesis range of the additional image, the output information generation unit 25 may set the corrected synthesis range as a figure filled with a predetermined color as the additional image. In this case, when the synthesis range is reduced and corrected based on the information on the specified specific range (P), a relatively small additional image (Q) is synthesized, and the range obscuring the game character becomes smaller ( FIG. 5( a) ). Furthermore, when the synthesis range is set to the periphery of the display range of the game character, the additional image is synthesized to surround the game character (Q′), so that the game character itself is not obscured ( FIG. 5( b) ).

[0050] In this example of the present embodiment, the design of the object to be displayed, such as a game character, contained in the target image is no longer obscured by a figure representing the position of the user's gaze, and the visibility of the object is no longer reduced.

[0051] [Gaze position information corresponding to the left and right eyes] In the explanation so far, it has been assumed that the image generated by the screen generation unit 21 is one per frame and the gaze position information acquired by the gaze information acquisition unit 23 is one, but this embodiment is not limited to this.

[0052] For example, when displaying frame images on a display device 30 such as a VR HMD, the screen generation unit 21 may generate a pair of frame images for each frame, one for the user's left eye and one for the user's right eye.

[0053] In this case, the gaze position information acquired by the gaze information acquisition unit 23 may be for either the left eye or the right eye, or it may acquire gaze position information for both the left eye viewing the frame image for the left eye and the right eye viewing the frame image for the right eye.

[0054] In the latter case where the gaze information acquisition unit 23 acquires both the gaze position information of the left eye and the gaze position information of the right eye, the position determination unit 24 determines the synthesis range of the additional image based on the gaze position information of the left eye and the synthesis range of the additional image based on the gaze position information of the right eye for each of the frame image for the left eye and the frame image for the right eye.

[0055] The output information generation unit 25 then synthesizes the additional image into the synthesis range of the additional image based on the gaze position information of the left eye in the frame image for the left eye to generate output information that is a synthesis image for the left eye, and synthesizes the additional image into the synthesis range of the additional image based on the gaze position information of the right eye in the frame image for the right eye to generate output information that is a synthesis image for the right eye.

[0056] In this example, if the screen generation unit 21 further generates frame images for both eyes (general frame images) to be displayed on a display device 30 such as a VR HMD, in addition to a frame image for the left eye and a frame image for the right eye, the output information generation unit 25 may use these frame images for both eyes as target images and composite additional images into synthesis ranges corresponding to the synthesis range of additional images based on gaze position information for the left eye and the synthesis range of additional images based on gaze position information for the right eye, respectively. In this case, the additional images may be composited so that the gaze positions of the left eye and the right eye can be distinguished by varying the aspects of the additional images, such as the color (hue, saturation, brightness), shape, and drawing method (solid line, dotted line, dashed line, etc.).

[0057] [Use of output information] Furthermore, the output information generated as in the above example may be output as moving image data and provided for distribution, for example.

[0058] [Example of operation] The information processing device 10 according to the present embodiment basically has the above-described configuration and operates as follows: The information processing device 10 according to the present embodiment is, for example, a home game console that runs a game application, and records the screen displayed in the running game application as moving image data in accordance with a user's instructions.

[0059] One of the features of the information processing device 10 according to the present embodiment is that the following process is performed during this recording: The information processing device 10 sequentially generates images (target images) that will become the screen of the game, and sequentially outputs and displays them on a display device 30 such as an HMD.

[0060] The user performs game operations while viewing the screen displayed on the display device 30. The information processing device 10 accepts the user's operations, uses them for game processing, and updates the game screen.

[0061] Meanwhile, the gaze detection device 40 connected to the information processing device 10 detects the gaze position (a point on the game screen) of the user looking at the game screen. In parallel with the game processing, the information processing device 10 executes the processing exemplified in Fig. 6 and acquires gaze position information indicating the gaze position of the user from the gaze detection device 40 (S11).

[0062] The information processing device 10 arranges a rectangle circumscribing a predetermined additional image (for example, an image of a cross graphic additional image) so that its center is a point on the target image represented by the user's gaze position information acquired in step S11, and generates information on a composite range representing the range of the rectangle (specified by the coordinates of its upper left vertex and lower right vertex) (S12).

[0063] The information processing device 10 composites the predetermined additional image onto the target image in the composite range determined in step S12 (S13), and generates and records a composite image including the target image and the additional image (S14).

[0064] The information processing device 10 repeats the processes from step S11 to step S14 until the user issues an instruction to stop recording the game screen or until the game application is terminated.

[0065] When the user issues an instruction to stop recording the game screen, or when the information processing device 10 completes the repetition of the processes from step S11 to step S14, the information processing device 10 generates moving image data as output information, making the composite images recorded up to that point playable in the order in which they were recorded (S15). A widely known encoding method can be used to generate such moving image data, so a detailed description will be omitted here. The information processing device 10 outputs the generated moving image data and completes the process.

[0066] The user uses the moving image data generated by this process by uploading it to, for example, a moving image data sharing site.

[0067] In another example, instead of recording the output information sequentially generated in step S14, the information processing device 10 may sequentially send the output information to a video distribution server (not shown) and distribute it as a video.

[0068] [Example of combining additional images during playback] Furthermore, in the explanation up to this point, the information processing device 10 that generates the target image generates a composite image by combining the target image with an additional image as output information, but the present embodiment is not limited to this.

[0069] In one example of this embodiment, instead of generating a composite image, the information processing device 10 may transmit or record additional information indicating the gaze position, such as gaze position information itself or information on a synthesis range into which an additional image is synthesized, as output information together with the target image, as information necessary for generating the composite image. In this example, when the user gives an instruction to output additional information or when a setting to that effect is made, the additional information indicating the gaze position, such as gaze position information itself or information on a synthesis range into which an additional image is synthesized, may be transmitted or recorded as output information together with the target image. Here, if the information on the synthesis range is used as the additional information, the information processing device 10 may determine the synthesis range by the processing of the position determination unit 24 already described.

[0070] An example of an information processing system including an information processing device 10 according to this example is shown in Fig. 7. As illustrated in Fig. 7, the control unit 11 of the information processing device 10 on the side that generates output information (hereinafter referred to as the generation side) executes a program stored in the storage unit 12, thereby functionally realizing a configuration that includes a screen generation unit 21, a screen display control unit 22, a line-of-sight information acquisition unit 23, and a synchronization output unit 31, and the control unit 11 of the information processing device 10 on the side that generates and displays a composite image based on the output information (hereinafter referred to as the reproduction side) executes a program stored in the storage unit 12, thereby functionally realizing a configuration that includes a synchronization acceptance unit 41, a position determination unit 24, a composite image generation unit 25', and a reproduction processing unit 42.

[0071] Here, the generating information processing device 10 may be a server device or the like that emulates the information processing device 10. In this case, the generating information processing device 10 (which may be a virtual one) receives and accepts operations accepted by a user of another information processing device 10, such as a personal computer, connected via a network. In this example, the generating information processing device 10 sends the generated target image (frame image) or the like to the other information processing device 10 for display (i.e., the frame image is displayed on a display device 30 connected to the other information processing device 10). In this example, the other information processing device 10 may function as the playing information processing device 10.

[0072] In the following description, components that perform the same operations as those already described will be assigned the same reference numerals to avoid redundant explanation and the description will be simplified.

[0073] In this example, the screen generation unit 21 of the generating information processing device 10 sequentially generates frame images that are to be target images. Furthermore, the screen display control unit 22 sequentially outputs and displays the frame images sequentially generated by the screen generation unit 21 on the display device 30 via the display control unit 14. In this example as well, the user of the generating information processing device 10 performs operations while viewing the frame images displayed on the display device 30.

[0074] The gaze information acquisition unit 23 acquires gaze position information from the gaze detection device 40 for each frame image, which is the target image sequentially generated by the screen generation unit 21, which is a position on each frame image and represents the position of the gaze of the user who is looking at the screen including the frame image and performing an operation.

[0075] The synchronization output unit 31 outputs additional information based on the gaze position information obtained for each frame image in synchronization with the corresponding frame image. As already mentioned, the additional information here is, for example, information necessary for generating a composite image, and may be the gaze position information itself, information on the synthesis range for synthesizing the additional image, or the like. In this embodiment, to achieve this synchronization, the synchronization output unit 31 may encode the additional information into a computer-readable image and embed it in the corresponding frame image (using image embedding technology). Alternatively, the synchronization output unit 31 may achieve synchronization by encoding the additional information into a computer-readable image and recording it in a relatively inconspicuous location, such as an edge portion of the corresponding frame image.

[0076] In another example, the synchronous output unit 31 may transmit or record frame images by associating each frame image with unique identification information, and may also transmit or record additional information corresponding to the frame image identified by the identification information, in association with the identification information. Using identification information to identify frame images in this way makes it possible to transmit or record additional information, such as gaze position information corresponding to each frame image included in the video data, as information for synthesis processing, separately from video data containing a series of frame images. Such information for synthesis processing may be distributed via a file server or the like on a network, separately from or together with the video data.

[0077] Furthermore, if there is other information to be reproduced in synchronization with the frame images (hereinafter referred to as synchronized reproduction information), the synchronized output unit 31 may include additional information corresponding to the frame images in the other information, and transmit or record the synchronized reproduction information including the additional information such as gaze position information together with the series of frame images as output information. Such synchronized reproduction information includes audio information to be reproduced in synchronization with the frame images.

[0078] In this example, the synchronization output unit 31 encodes and includes additional information by, for example, modulating and combining a baseband signal in a frequency range inaudible to humans with audio information, which is the synchronized playback information. If identification information for identifying a frame image is further included, the additional information is combined with the left channel of the stereo audio information, and the identification information for the corresponding frame image is combined with the right channel at the time the corresponding additional information is played back. In this case, too, a first signal is generated by modulating a baseband signal in a frequency range inaudible to humans with gaze position information, and a second signal is generated by modulating a baseband signal in a frequency range inaudible to humans with identification information, and these signals are combined with the audio information of the left and right channels.

[0079] However, this is just one example, and when obtaining gaze position information for the left and right eyes, the synchronization output unit 31 may obtain a signal (called an L signal) obtained by modulating a baseband signal in a frequency band inaudible to humans with additional information based on the gaze position information for the left eye, and synthesize this L signal into the left channel of the stereo audio information, and obtain a signal (called an R signal) obtained by modulating a baseband signal in a frequency band inaudible to humans with additional information based on the gaze position information for the right eye, and synthesize this R signal into the right channel of the stereo audio information.

[0080] If identification information for identifying the frame image is further included, the synchronization output unit 31 may generate a third signal by modulating a baseband signal that has a different frequency from the baseband signals used to generate the L signal and R signal and is in an inaudible frequency band for humans with the identification information, and further synthesize this third signal into at least one of the left channel or right channel of the stereo audio information into which the L signal or R signal has been synthesized.

[0081] In yet another example, the synchronization output unit 31 may record an audio signal in either the left channel or the right channel of the stereo audio information (i.e., as a monaural audio signal), and record the L signal or R signal in an empty channel of the left channel or the right channel of the stereo audio information where no audio signal is recorded. If the frequencies of the baseband signals used to generate the L signal and the R signal are different, the synchronization output unit 31 may combine and record both of them. The synchronization output unit 31 may further superimpose a third signal on the L signal or the R signal and record it in this empty channel.

[0082] When information is recorded in this manner on an audio information channel, the information processing device 10 on the playback side, which will be described later, demodulates and obtains the information from the audio information, for example, by using the baseband signal used to modulate the information.

[0083] Here, the synchronized playback information is not limited to audio information. For example, when information representing an operation performed by a user is recorded in synchronization with a frame image, additional information or identification information based on gaze position information may be recorded superimposed on the information representing the operation (separable by a computer from the information representing the operation).

[0084] In addition, if the output destination specified by the user is not a predetermined output destination such as a distribution server, for example, if a memory card connected via USB is specified as the output destination, the synchronous output unit 31 may output the frame image, additional information, synchronous playback information, etc. as is, without including the additional information or synchronous playback information in the output.

[0085] Next, the synchronization receiving unit 41 of the control unit 11 of the playback information processing device 10, which corresponds to the generation information processing device 10 in this example, receives the output information generated by the generation information processing device 10, or acquires and accepts the output information from a server, storage device, or the like that recorded the output information. The synchronization receiving unit 41 extracts a series of frame images from the accepted output information. The synchronization receiving unit 41 also extracts additional information such as gaze position information from the accepted output information.

[0086] When additional information is encoded into a computer-readable image and recorded within the corresponding frame image, the synchronization receiving unit 41 reads the encoded image and decodes the additional information to extract the additional information based on the gaze position information for each frame image.

[0087] Furthermore, if additional information is distributed or recorded in association with the identification information of a frame image, the synchronization receiving unit 41 extracts the additional information along with the identification information. Furthermore, if additional information is included in synchronized playback information such as audio information, the synchronization receiving unit 41 decodes and extracts the additional information from the synchronized playback information.

[0088] The position determination unit 24 determines the superimposition position of the additional image to be superimposed on the corresponding frame image extracted by the synchronization reception unit 41, based on the user's additional information extracted for each frame image by the synchronization reception unit 41. As an example, if the additional information is gaze position information itself, the position determination unit 24 sets the center of the circumscribing rectangle of the additional image (intersection of the diagonals) as a point on the corresponding frame image indicated by the gaze position information, which is the additional information, and determines the range in which to place the additional image.

[0089] If the additional information is gaze position information itself, this position determination unit 24 performs the same operation as the position determination unit 24 already described, and performs processing related to the example of the position determination unit 24 already described, such as performing processing to determine the range of synthesis of the additional image with the corresponding frame image based on the extracted additional information for each of a predetermined N (N is an integer of 2 or more) frames.

[0090] Furthermore, if the additional information is a synthesis range of an additional image, the position determination unit 24 may perform the same operation as the position determination unit 24 already described, using the center of the synthesis range as gaze position information, or may output the synthesis range of the additional image represented by the additional information to the synthesis image generation unit 25' as a synthesis range for synthesizing the additional image.

[0091] The composite image generation unit 25' performs the same operation as the output information generation unit 25 described above, and generates a composite image by compositing an additional image onto a frame image as a target image extracted by the synchronization acceptance unit 41 in the synthesis range determined by the position determination unit 24 corresponding to the frame image.

[0092] The playback processing unit 42 sequentially plays back the composite images generated by the composite image generation unit 25′ and outputs them to the display device 30. The user of the information processing device 10 on the playback side views the series of composite images displayed on the display device 30.

[0093] When the synchronization receiving section 41 receives synchronized playback information such as audio information to be played back together with the frame image, the playback processing section 42 plays back the synchronized playback information in synchronization with the frame image.

[0094] The control unit 11 of the information processing device 10 on the playback side may determine, based on instructions or settings input by the user, whether or not to perform a synthesis process for an additional image using additional information extracted from the accepted output information, and if it is decided to perform the synthesis process, the synthetic image generation unit 25' may generate a synthetic image by synthesizing the additional image, but if it is decided not to perform the synthesis process, the frame images may be reproduced sequentially without synthesizing the additional image. In this way, synthesis of the additional image can be turned on or off according to instructions or settings input by the user.

[0095] Furthermore, whether or not to perform a synthesis process for an additional image using additional information may be set not only by a user's instruction but also according to, for example, the model of the playback-side information processing device 10. For example, it may be determined that the synthesis process is to be performed when the information processing device 10 is a personal computer or a home game console, and that the synthesis process is not to be performed when the information processing device 10 is a smartphone or the like.

[0096] [Another example of operation of the playback information processing device] In addition, the composite image generation unit 25′ in the information processing device 10 on the playback side may vary the type and manner of the additional image to be composited with the frame image depending on the type of information processing device 10 on the playback side (personal computer, home game console, smartphone, tablet terminal, etc.), the type and screen size of the display device 30 connected thereto, the usage scene (outdoors, indoors, etc.), etc.

[0097] [Other examples of using gaze position information] Furthermore, the information processing device 10 that acquires the gaze position information may not only combine this gaze position information with the target image or output it for combination, but may also use it for other purposes.

[0098] For example, when a user performs a predetermined operation on a target image displayed on the display device 30, information required by the user may be displayed near the user's gaze position acquired at that time. Here, the vicinity of the gaze position is not the exact position indicated by the gaze position information, but a position a predetermined distance away from the position indicated by the gaze position information, which is a position that the user can see without moving their gaze significantly. In this way, it is possible to display information without obscuring the area that the user or viewer is trying to view.

[0099] Furthermore, when playing a shooting game, for example, the information required by the user may include the number of remaining bullets, geographical information (map), position information of the enemy character, information on button operation of the game controller (which button was pressed), and information on the enemy character. Here, the information on the enemy character may be displayed in a display area that is closest to the area set as the area near the line of sight, within an area that does not overlap with the line of sight or the display range of the enemy character.

[0100] Furthermore, the display may be avoided so as not to overlap with the display range of the character being operated by the user. In this way, the display will not overlap with the operation target, etc., and will not interfere with the user's operation.

[0101] The present disclosure also includes the following aspects. [Item 1] having one or more processors, The one or more processors: generating a screen including an image to be manipulated by a user; acquiring gaze position information representing a position on the generated image, the gaze position being a position of a user's gaze while viewing a screen including the image and performing an operation; Generate output information including additional information representing the acquired gaze position information of the user and the image included on the screen, An information processing device that subjects the output information to predetermined processing. [Item 2] Item 1. The information processing device according to item 1, the generated image that is the target of user manipulation is a moving image including a plurality of frame images, The one or more processors: sequentially acquiring gaze position information indicating the position of a user's gaze while watching the moving image and performing an operation while the moving image is being displayed; an information processing device that generates output information by associating the additional information representing the acquired user's gaze position information with a corresponding frame image of the moving image; [Item 3] Item 2. The information processing device according to item 2, The one or more processors: An information processing device generates output information by associating additional information representing the user's gaze position information acquired while displaying N (N is an integer greater than or equal to 2) consecutive frame images of the moving image, including at least one of the frame images before and after the corresponding frame image of the moving image, with the corresponding frame image of the moving image. [Item 4] Item 1. The information processing device according to item 1, the additional information is an additional image representing the position of the user's gaze represented by the gaze position information, The one or more processors: An information processing device that generates output information including an additional image superimposed on the image and an image included on the screen based on the acquired information on the user's gaze position. [Item 5] Item 2. The information processing device according to item 2, the additional information is an additional image representing the position of the user's gaze represented by the gaze position information, The one or more processors: An information processing device that generates output information by superimposing an additional image on a corresponding frame image of the moving image based on the acquired information on the user's gaze position. [Item 6] Item 5. The information processing device according to item 5, The one or more processors: An information processing device that, when superimposing an additional image on a corresponding frame image of a moving image, determines the content of the additional image based on a partial image of the frame image at the position where the additional image is to be superimposed, and generates output information. [Item 7] Item 6. The information processing device according to item 6, The one or more processors: An information processing device that, when an additional image is superimposed on a corresponding frame image of a moving image, determines the appearance of the additional image based on information relating to the color of a partial image of the frame image at the position where the additional image is superimposed, and generates output information. [Item 8] Item 4. The information processing device according to item 4, The one or more processors: An information processing device that acquires information representing a specific portion included in the image that is the target of the operation from a screen generation means, and determines a position at which to arrange the additional image using the acquired information. [Item 9] Item 4. The information processing device according to item 4, The one or more processors: An information processing device that acquires information representing a specific portion included in the image that is the target of the operation from a screen generation means, and determines the mode of the additional image using the acquired information. [Item 10] A method for controlling an information processing device having one or more processors, comprising: the one or more processors: generating a screen including an image to be manipulated by a user; acquiring gaze position information representing a position on the generated image, the gaze position being a position of a user's gaze while viewing a screen including the image and performing an operation; generating output information including additional information representing the acquired gaze position information of the user and an image included on the screen; A control method for an information processing device, which subjects the output information to predetermined processing. [Item 11] A computer-readable non-transitory recording medium, generating a screen including an image to be manipulated by a user; acquiring gaze position information representing a position on the generated image, the gaze position being a position of a user's gaze while viewing a screen including the image and performing an operation; Generate output information including additional information indicating the acquired gaze position information of the user and the image included on the screen. A recording medium storing a program for causing a computer to execute a process of subjecting the output information to a predetermined process. [Explanation of symbols]

[0102] 10 Information processing device, 11 Control unit, 12 Memory unit, 13 Operation acquisition unit, 14 Display control unit, 15 Communication unit, 20 Controller device, 21 Screen generation unit, 22 Screen display control unit, 23 Gaze information acquisition unit, 24 Position determination unit, 25 Output information generation unit, 25' Composite image generation unit, 30 Display device, 31 Synchronous output unit, 40 Gaze detection device, 41 Synchronous reception unit, 42 Playback processing unit.

Claims

1. a screen generating means for generating a screen including an image to be operated by a user; an acquisition means for acquiring gaze position information representing a position on the generated image, the gaze position being a position of a user's gaze while performing an operation while looking at a screen including the image; an output information generating means for generating output information including additional information representing the acquired gaze position information of the user and an image included on the screen; Including, the generated image that is the target of user manipulation is a moving image including a plurality of frame images, the additional information is an additional image representing the position of the user's gaze represented by the gaze position information, When the additional image is arranged so as to be superimposed on the corresponding frame image of the moving image, the output information generating means determines the shape of the additional image based on the shape of a partial image of the frame image at the position where the additional image is to be superimposed, and generates output information. Information processing device.

2. 2. The information processing device according to claim 1, the acquiring means successively acquires gaze position information indicating a gaze position of a user who is watching the moving image and performing an operation while the moving image is being displayed; The output information generating means generates output information by associating the additional information representing the acquired gaze position information of the user with a corresponding frame image of the moving image.

3. 3. The information processing device according to claim 2, The output information generation means is an information processing device that generates output information by associating additional information representing the user's gaze position information acquired during the display of N (N is an integer greater than or equal to 2) consecutive frame images of the moving image, including at least one of the frame images before and after the corresponding frame image.

4. An information processing device according to claim 1, Determining the shape of the additional image and generating output information includes matching the shape of the partial image with the shape of the additional image to generate output information. Information processing device.

5. An information processing device according to claim 1, The output information generation means is an information processing device that, when an additional image is superimposed on a corresponding frame image of a moving image, determines the appearance of the additional image based on information relating to the color of a partial image of the frame image at the position where the additional image is superimposed, and generates output information.

6. An information processing device according to any one of claims 1 to 5, The output information generating means is an information processing device that acquires information representing a specific part included in the image that is the target of the operation from a screen generating means, and determines the position where the additional image is to be placed using the acquired information.

7. An information processing device according to any one of claims 1 to 6, The output information generating means acquires information representing a specific portion included in the image to be operated from a screen generating means, and determines the appearance of the additional image using the acquired information.

8. A control method for an information processing device, comprising: a screen generating means for generating a screen including an image to be operated by a user; an acquisition means acquires gaze position information representing a position on the generated image, the gaze position being a position of a user who is performing an operation while looking at a screen including the image; a control method for an information processing device, wherein output information generating means generates output information including additional information representing the acquired gaze position information of the user and an image included on the screen, the generated image that is the target of user manipulation is a moving image including a plurality of frame images, the additional information is an additional image representing the position of the user's gaze represented by the gaze position information, When the additional image is arranged so as to be superimposed on the corresponding frame image of the moving image, the output information generating means determines the shape of the additional image based on the shape of a partial image of the frame image at the position where the additional image is to be superimposed, and generates output information. A method for controlling an information processing device.

9. Computer, a screen generating means for generating a screen including an image to be operated by a user; an acquisition means for acquiring gaze position information representing a position on the generated image, the gaze position being a position of a user's gaze while performing an operation while looking at a screen including the image; an output information generating means for generating output information including additional information representing the acquired gaze position information of the user and an image included on the screen; A program that functions as the generated image that is the target of user manipulation is a moving image including a plurality of frame images, the additional information is an additional image representing the position of the user's gaze represented by the gaze position information, When the additional image is arranged so as to be superimposed on the corresponding frame image of the moving image, the output information generating means determines the shape of the additional image based on the shape of a partial image of the frame image at the position where the additional image is to be superimposed, and generates output information. program.

Images