Image generation device, image reproduction device, image generation method, and program

Abstract

This image generation device: on the basis of scene information, generates, as display image data, image data representing a portion of a scene represented by the scene information; determines, as a recording target, a section which is at least a portion of the scene and at which the display image data overlaps said at least a portion of the scene; and generates, as recording image data, image data for the section determined to be the recording target, the section being at least the portion of the scene. The image generation device is configured to that the generated recording image data is provided to a prescribed reproduction process.

Description

[Technical Field]

[0001] The present invention relates to an image generating device that generates an image to be displayed on a display device, an image reproducing device that reproduces an image, an image generating method, and a program. [Background technology]

[0002] For example, in video games, an image generating device generates images showing a virtual space or the like, and repeatedly executes a process of displaying the generated images on the screen of a display device to present the images to a user. At this time, the image generating device may record the content of the images generated for display so that they can be later made public to a third party. Summary of the Invention [Problem to be solved by the invention]

[0003] In the above-described conventional technology, the content of the image generated for display changes depending on the user's instructions. Furthermore, particularly when displaying an image on a head-mounted display device, the image generating device may change the image displayed on the display device in conjunction with changes in the user's facial orientation. When video that changes in real time in response to the user's instructions or facial orientation is recorded and the recorded video is later viewed by another viewer, the recorded video may change in an unexpected way for the viewer or may cause the viewer to feel nauseous. Therefore, even if the video generated for display is recorded as is, it may not necessarily be suitable for later viewing.

[0004] The present invention has been made in consideration of the above-mentioned circumstances, and one of its objectives is to provide an image generation device, an image playback device, an image generation method, and a program that enable content generated in real time and displayed as video to be played back in a form that is easy to view later. [Means for solving the problem]

[0005] An image generating device according to one aspect of the present invention includes a display image generating unit that generates, based on scene information, image data representing a portion of a scene represented by the scene information as display image data; a determination unit that determines, as a recording target, at least a portion of the scene where the display image data and at least a portion of the display image data overlap; and a recording image generating unit that generates, as recording image data, image data of at least a portion of the scene determined to be the recording target, and the generated recording image data is subjected to a predetermined playback process.

[0006] An image playback device according to one aspect of the present invention includes a recording image data receiving unit that receives recording image data that is generated based on scene information together with display image data representing a portion of the scene represented by the scene information, and that includes at least a portion of the scene that corresponds to the display image data, and a playback unit that plays back the recording image data received by the recording image data receiving unit, wherein the playback unit acquires field of view information regarding the field of view of a viewer watching video played back from the recording image data, and extracts and displays at least a portion of the recording image data based on the field of view information.

[0007] An image generation method according to one aspect of the present invention includes the steps of: generating, based on scene information, image data representing a portion of a scene represented by the scene information as display image data; determining, as a recording target, at least a portion of the scene where the display image data and at least a portion of the display image data overlap; and generating, as recording image data, image data of at least a portion of the scene determined to be the recording target, wherein the generated recording image data is subjected to a predetermined playback process.

[0008] A program according to one aspect of the present invention is a program for causing a computer to execute the steps of: generating, based on scene information, image data representing a portion of a scene represented by the scene information as display image data; determining, as a recording target, at least a portion of the scene where the display image data and at least a portion of the scene overlap; and generating, as recording image data, image data for at least the portion of the scene determined to be the recording target, wherein the generated recording image data is subjected to a predetermined playback process. This program may be provided by being stored on a computer-readable, non-transitory information storage medium. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an image generating device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a functional block diagram showing functions of the image generating apparatus according to the embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of a recording target portion. [Figure 4] FIG. 10 is a diagram illustrating another example of a recording target portion. [Figure 5] 1 is a block diagram showing a configuration of an image playback device according to an embodiment of the present invention; [Figure 6] 1 is a functional block diagram showing functions of an image playback device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a block diagram showing the configuration of an image generating device 10 according to one embodiment of the present invention. As will be described later, the image generating device 10 is used to generate an image to be presented to the user and to record video data corresponding to the presented image. Hereinafter, a user using the image generating device 10 will be referred to as a recording user. For example, the image generating device 10 may be a home game console, a portable game console, a personal computer, or the like, and as shown in the figure, includes a control unit 11, a storage unit 12, and an interface unit 13. The image generating device 10 is also connected to a display device 14 and an operation device 15.

[0012] The control unit 11 includes at least one processor such as a CPU, and performs various information processing by executing programs stored in the storage unit 12. Specific examples of the processing performed by the control unit 11 in this embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores the programs executed by the control unit 11 and data processed by the programs.

[0013] The interface unit 13 is an interface for data communication between the display device 14 and the operation device 15. The image generation device 10 is connected to the display device 14 and the operation device 15 via the interface unit 13 either wired or wirelessly. Specifically, the interface unit 13 includes a multimedia interface for transmitting a video signal supplied by the image generation device 10 to the display device 14. It also includes a data communication interface for receiving a signal indicating the operation content performed by the recording user on the operation device 15. Furthermore, the interface unit 13 may include a communication interface for transmitting and receiving data to and from other communication devices via a communication network such as the Internet.

[0014] The display device 14 displays on a screen an image corresponding to the video signal supplied from the image generating device 10. In particular, in this embodiment, the display device 14 is a head-mounted display device capable of presenting a stereoscopic image by presenting separate images to each of the user's left and right eyes. In this embodiment, the display device 14 is also equipped with a posture sensor for detecting changes in its own posture. This posture sensor may be an acceleration sensor, a gyro sensor, a geomagnetic sensor, or the like.

[0015] The operation device 15 is, for example, a controller for a home game console, and receives operation input from the recording user. The operation device 15 is connected to the image generating apparatus 10 by wire or wirelessly, and transmits an operation signal indicating the content of the operation input received from the recording user to the image generating apparatus 10. The operation device 15 may be of various shapes, such as a device that the recording user holds in their hand or a device that they wear on their hand.

[0016] The functions realized by the image generating device 10 will be described below with reference to the functional block diagram of FIG. 2. As shown in FIG. 2, the image generating device 10 is functionally configured to include a scene information generating unit 21, a viewing direction information receiving unit 22, a display image generating unit 23, a recording target portion determining unit 24, and a recording image generating unit 25. These functions are realized by the control unit 11 operating in accordance with one or more programs stored in the storage unit 12. These programs may be provided to the image generating device 10 via a communication network such as the Internet, or may be provided by being stored on a computer-readable information storage medium such as an optical disc.

[0017] The scene information generating unit 21 determines the content of the scene to be presented to the recording user and generates scene information for specifying the content. The scene information generating unit 21 may be realized by an application program such as a game program, for example.

[0018] In this embodiment, the scene to be presented to the recording user is assumed to be the interior of a virtual three-dimensional space (virtual space). In this case, the scene information generating unit 21 determines, as scene information, information for specifying the shape and appearance of objects to be placed in the virtual space, their positions within the virtual space, etc.

[0019] Furthermore, the scene information generation unit 21 also determines information that identifies the portion of the scene defined by the scene information that is actually to be displayed. Specifically, the scene information generation unit 21 determines the position and viewing direction of the viewpoint to be set in the virtual space based on the viewing direction information received by the viewing direction information receiving unit 22, which will be described later.

[0020] The viewing direction information receiving unit 22 receives viewing direction information related to the orientation of the recorded user's face. In this embodiment, the viewing direction information receiving unit 22 receives information indicating the detection results of the posture sensor from the display device 14, and based on the detection results, identifies information indicating how the orientation of the display device 14 worn on the recorded user's head has changed as viewing direction information. The scene information generating unit 21 changes the viewing direction set in the virtual space in conjunction with this viewing direction information. As a result, for example, when the recorded user turns their face to the right, the image displayed on the display device 14 also changes to a view to the right of the virtual space compared to the previous front view, allowing the recorded user to view the image displayed on the display device 14 as if they were directly looking around the virtual space.

[0021] The display image generation unit 23 generates display image data ID based on the scene information generated by the scene information generation unit 21. The display image generation unit 23 may be realized by a program such as a rendering engine separate from the application program that realizes the scene information generation unit 21. The display image generation unit 23 generates display image data ID representing a state of the virtual three-dimensional space as viewed from the viewpoint position and viewing direction specified by the scene information generation unit 21, based on the scene information. This display image data ID becomes image data representing a portion of the scene (the state of the virtual three-dimensional space) represented by the scene information provided by the scene information generation unit 21. Note that the display device 14 presents images to the right and left eyes of the recording user, and therefore generates an image for the right eye and an image for the left eye, respectively. For convenience of explanation, the following description will focus on only one of the images, but the image generation device 10 may perform similar processing on both the image for the right eye and the image for the left eye.

[0022] The display image generation unit 23 transmits a video signal including the generated display image data ID to the display device 14. As a result, the display image is displayed on the screen of the display device 14 and presented to the recorded user. The image generation device 10 according to this embodiment repeatedly executes the process of generating scene information by the scene information generation unit 21, the process of generating display image data ID by the display image generation unit 23, and the process of transmitting the display image data ID to the display device 14 at a given frame rate, thereby causing the display device 14 to display an image showing the state of the virtual space that changes over time.

[0023] When the display image generation unit 23 generates display image data ID based on scene information provided by the scene information generation unit 21, the recording target portion determination unit 24 determines a portion to be recorded (hereinafter referred to as the recording target portion) when recording image data based on that scene information. This recording target portion is a portion at least partially overlapping with the display image data ID. In other words, the image data to be recorded (hereinafter referred to as the recording image data IR) and the display image data ID are both image data showing the content of the same scene defined by the same scene information, and are image data drawn with at least a portion overlapping each other.

[0024] As a typical example, the recording target portion determination unit 24 determines, as the recording target portion, at least a portion of a scene defined by the scene information generated by the scene information generation unit 21, that includes the entire display image data ID and is wider than the display image data ID. FIG. 3 illustrates an example of a method for determining such a recording target portion, showing the relationship between the display image data ID and the recording image data IR. In this example, the recording target portion is determined so that the recording image data IR includes the entire display image data ID and its surrounding areas with equal widths on all four sides. The recording target portion determination unit 24 can include such a recording target portion in the recording image data IR by setting the viewpoint position when generating the recording image data IR to a position a predetermined distance behind (opposite the viewing direction) the viewpoint position used to generate the display image data ID. Hereinafter, the portion of the recording image data IR corresponding to the display image data ID will be referred to as the display target portion, and the portion excluding the display target portion (the outer peripheral portion surrounding the display target portion) will be referred to as the extended portion. Other specific examples of the recording target portion determination method will be described later.

[0025] The recording image generation unit 25 generates image data corresponding to the recording target portion determined by the recording target portion determination unit 24 as recording image data IR. The process of generating the recording image data IR may be the same as the process of generating display image data ID by the display image generation unit 23, except for the target portion. The recording image generation unit 25 may be realized by the same program as the display image generation unit 23. Furthermore, while the display image generation unit 23 is executing the process of generating display image data ID, the recording image generation unit 25 may execute the process of generating recording image data IR in parallel based on the same scene information. The recording image data IR generated by the recording image generation unit 25 is later subjected to a predetermined playback process.

[0026] Here, the recording image generation unit 25 generates the recording image data IR so that the display target portion in the recording image data IR has the same resolution as the display image data ID. Since the recording image data IR includes a wider range than the display image data ID, the data size of the recording image data IR is larger than that of the display image data ID.

[0027] When the recording user instructs recording of the recording image data IR, the recording image generation unit 25 generates the recording image data IR for the instructed period. Hereinafter, the period during which the recording image data IR is generated is referred to as the target recording period. The target recording period may be the period from when the recording user instructs the start of recording to when the recording user instructs the end of recording, or it may be the period from when the recording user instructs the start of recording to when a predetermined time has passed. Each time the display image generation unit 23 generates one piece of display image data ID within the target recording period, the recording image generation unit 25 generates one piece of recording image data IR based on the same scene information.

[0028] The recording image generation unit 25 saves, as recording video data, data consisting of multiple pieces of recording image data IR generated in chronological order within the target recording period. The recording video data may contain multiple pieces of recording image data IR in their original format, or may be data encoded using a given video encoding method. The image generation device 10 can upload the saved recording video data to a predetermined server device in response to a recording user's instruction, or distribute it in response to a request from another user, making it available to others. The image playback device 30, described below, plays back the contents of this recording video data, allowing other users to view video of the same length that depicts the same scene as the video displayed on the display device 14 during the target recording period. Details of the process by which the image playback device 30 plays back the contents of the recording video data will be described later.

[0029] Hereinafter, several specific examples of the method by which the recording portion determination unit 24 determines the recording portion will be described.

[0030] As a first example, the recording target portion determination unit 24 may determine the recording target portion based on the content of the scene information. For example, the recording target portion determination unit 24 determines the recording target portion based on the position and movement of a target object T that is expected to attract the recording user's attention, among the objects placed in the virtual space specified by the scene information.

[0031] The object of interest T in this example may be an object designated by the scene information generation unit 21. For example, in a game, an object such as a character that is to be operated by the recording user may be included in the display objects. In this case, the object that is to be operated by the recording user may be designated as the object of interest T. Furthermore, the scene information generation unit 21 may designate an object that is particularly important to the recording user (for example, an enemy object that the recording user is attacking) among the objects to be displayed as the object of interest T.

[0032] More specifically, for example, the recording target portion determination unit 24 determines the recording target portion so that the object of attention T is located near the center of the recording image data IR while including the display target portion. As an example, if the object of attention T is located to the right of the display image data ID, the recording target portion determination unit 24 determines the recording target portion so that the object of attention T is located to the right of the display target portion and has a width greater than the default width. In this case, the closer the position of the object of attention T is to the edge of the display target portion, the wider the width of the extension portion in that direction may be. Furthermore, when the width of the extension portion in a predetermined direction is increased, the width of the extension portion in the opposite direction may be correspondingly narrower than the default width, thereby controlling the size of the recording image data IR to always be constant. Alternatively, the width of the extension portion in a predetermined direction may be increased as needed while the default widths in other directions may be maintained, resulting in the size of the recording image data IR changing depending on the position of the object of attention T.

[0033] 4 shows an example of a recording target portion determined according to the position of a target object T. In this example, the target object T is located at a position closer to the upper left than the center of the display target portion, and therefore the width of the left and upper sides of the extended portion is wider than in FIG.

[0034] The recording target portion determination unit 24 may determine the recording target portion not only based on the position of the attention object T but also based on the direction of movement of the attention object T in the virtual space. Specifically, for example, when the attention object T is moving upward as viewed from the viewpoint position, the recording target portion determination unit 24 determines the recording target portion so as to widen the width of the extension portion above the display target portion. Furthermore, in this case, the width of the extension portion in the movement direction may be widened as the recent movement speed of the attention object T is faster. Note that the recording target portion determination unit 24 may determine the recording target portion taking into consideration both the position and movement direction of the attention object T.

[0035] Furthermore, the recording target portion determination unit 24 may change the overall size of the recording target portion in accordance with the moving speed of the attention object T, regardless of the moving direction. In this case, the greater the moving speed of the nearest attention object T, the wider the width of the expansion portion will be in all directions.

[0036] As a second example, the recording portion determination unit 24 may determine the recording portion based on the operation of the recording user on the operation device 15. For example, when the recording user performs an operation on the operation device 15 to instruct a leftward direction, the recording portion is determined so that the expanded portion on the left side is wider than the default width. As in the first example, the width of the expanded portion in the direction opposite to the direction instructed by the recording user may be narrower than the default width.

[0037] Furthermore, if the amount of operation performed by the recording user can be identified, recording portion determination unit 24 may determine how wide to widen the expanded portion depending on the amount of operation. For example, if operation device 15 is provided with a tilt operation member that can indicate a direction by tilting it, recording portion determination unit 24 may determine the recording portion so that the wider the expanded portion is, the greater the amount by which the recording user tilts the tilt operation member.

[0038] Furthermore, the recording portion determination unit 24 may change the size of the recording portion simply in accordance with the frequency of operation or the magnitude of the operation amount, regardless of the operation direction of the recording user. For example, the recording portion determination unit 24 increases the overall size of the recording portion as the number of times or the magnitude of the operation amount of a predetermined operation performed within a predetermined period of time in the most recent past increases.

[0039] As a third example, the recording target portion determination unit 24 may determine the recording target portion based on the viewing direction information of the recording user received by the viewing direction information receiving unit 22. For example, if the recording user's facial direction changes over time, the recording target portion determination unit 24 widens the width of the expansion portion corresponding to the direction of change. The width of the expansion portion may also be changed according to the speed of change in the facial direction. As a specific example, if the recording user is turning their face to the left, the recording target portion is determined so that the greater the movement, the wider the left expansion portion becomes. Furthermore, as in the first and second examples described above, the size of the recording target portion may be changed according to the speed of change, regardless of the direction of change in the facial direction.

[0040] As a fourth example, the recording target portion determination unit 24 may determine the recording target portion based on the gaze point of the recording user. In this example, the viewing direction information receiving unit 22 receives not only viewing direction information regarding the direction of the recording user's face, but also information (gaze point information) indicating at which position the recording user is directing their gaze in the image displayed on the display device 14. The gaze point of the recording user can be identified by, for example, capturing an image of the recording user's pupils using an internal camera provided in the display device 14 and analyzing the pupil movement contained in the captured image.

[0041] Using this gaze point information, the recording target portion determination unit 24 determines the recording target portion so that the position at which the recording user is directing their gaze (i.e., the position at which the recording user is gazing) is closer to the center. For example, if it is determined that the recording user is directing their gaze at a position corresponding to the lower left of the display target portion, the recording target portion determination unit 24 determines the recording target portion so that the width of the left and bottom sides of the extended portion is wider. This brings the position at which the recording user is directing their gaze closer to the center of the image data IR for recording.

[0042] Furthermore, similar to the first example targeting the attention object T and the third example targeting the viewing direction, the recording target portion determination unit 24 may determine the position and size of the recording target portion according to the moving direction and moving speed of the point of interest. Specifically, for example, the recording target portion determination unit 24 may widen the width of the extension portion according to the most recent moving direction of the point of interest, or may increase the size of the recording target portion as the moving speed of the point of interest increases.

[0043] The various methods for determining the portion to be recorded as described above may be used in combination. For example, the position of the portion to be recorded may be determined taking into consideration both the position of the object of attention T and the viewing direction of the recording user.

[0044] Furthermore, when generating and recording the recording image data IR, the recording image generation unit 25 may also record at least some of the information indicating the position of the target object T, information indicating the operation content of the recording user, the recording user's viewing direction information, and gaze point information. This information is used when playing back the contents of the recorded video data. Note that the recording image generation unit 25 may also record information that is not used to determine the recording target portion. In particular, by recording the viewing direction information sequentially received by the viewing direction information reception unit 22 over the recording period together with the recording image data IR, it is possible to easily realize a playback process that takes changes in the viewing direction into account during the playback process described below.

[0045] Next, a playback process using the recording image data IR generated by the image generation device 10 will be described. Hereinafter, the information processing device that executes this playback process will be referred to as the image playback device 30, and a user who views the video played back using the image playback device 30 will be referred to as the viewing user. As shown in FIG. 5 , the image playback device 30 includes a control unit 31, a storage unit 32, and an interface unit 33, and is connected to a display device 34 and an operation device 35 via the interface unit 33. These hardware configurations may be similar to those of the image generation device 10. However, the display device 34 does not necessarily have to be a device worn by the user on the head; for example, it may be a stationary display device such as a home television set or a portable display device. The image generation device 10 that generated the recording video data may itself function as the image playback device 30.

[0046] The following describes the functions realized by the image playback device 30. As shown in Fig. 6, the image playback device 30 is functionally configured to include a recording image receiving unit 41 and a playback unit 42. These functions are realized by the control unit 31 of the image playback device 30 executing a program stored in the storage unit 32.

[0047] The recording image receiving unit 41 receives the recording image data IR that has been generated and recorded together with the facial expression image data ID by the image generating device 10. Specifically, the image reproducing device 30 may acquire the recording image data IR by reading out recorded video data that has been downloaded in advance and stored in the image reproducing device 30, or may receive the recording image data IR by receiving the recording image data IR that is distributed in real time from a server device via a communication network such as the Internet.

[0048] The playback unit 42 plays back video using the recording image data IR accepted by the recording image accepting unit 41. However, the playback unit 42 does not display the entire recording image data IR as is on the screen of the display device 34, but rather transmits an image of a partial area extracted from the recording image data IR to the display device 34 as a playback image and displays it on the screen. The extracted playback image may be an image of the same size as the display image data ID. As explained above, the recording image data IR covers a wider range than the display image data ID, so the playback unit 42 can extract, as the playback image, an image that has the same size and resolution as the display image data ID and at least partially overlaps with the display image data ID.

[0049] Specifically, the playback unit 42 detects movement in the viewing direction when the recording image data IR was generated by referring to a predetermined number of recording image data IR before and after the recording image data IR to be extracted from a series of recording image data IR recorded in chronological order. Such a process of detecting movement in the viewing direction may be realized by using a known method used in video coding, etc.

[0050] The playback unit 42 then specifies an area to be extracted as an image for playback from the recording image data IR to be extracted, so as to cancel out small changes in the detected movement in the viewing direction that may cause motion sickness. For example, the playback unit 42 determines, as the extraction target, an area corresponding to a direction obtained by averaging the viewing direction for a predetermined period before and after the timing of interest. Alternatively, the playback unit 42 may determine, as the extraction target, an area corresponding to a midpoint between the viewing directions before and after the timing of interest.

[0051] By determining the position from which to extract the playback image from each of the multiple recording image data IR contained in the recorded video data so as to reduce small changes in the field of view as described above, the image playback device 30 can play back the recorded video in a manner that is less likely to cause motion sickness for the viewing user.

[0052] Furthermore, the playback unit 42 may determine the extraction position of the playback image not only by reducing movement in the viewing direction when the recording image data IR was recorded, but also by taking into account the facial direction of the viewing user. In this case, the display device 34 is assumed to be a head-worn display device used by the viewing user, similar to the display device 14, and the image playback device 30 receives viewing direction information related to the facial direction of the viewing user, similar to the viewing direction information receiving unit 22 of the image generation device 10. The playback unit 42 then determines the extraction position of the playback image in response to changes in the viewing direction information of the viewing user. As described above, the recording image data IR covers a wider range than the display image data ID. Therefore, this technique allows the viewing user to view a scene in a direction different from the direction the recording user was looking within the range covered by the recording image data IR. Regardless of the type of display device 34, the playback unit 42 may change the extraction position of the playback image in real time in response to, for example, the content of an instruction operation performed by the viewing user on the operation device 35.

[0053] The playback unit 42 may determine the position from which to extract an image for playback using various methods, not limited to the examples described above. Below, several specific examples of methods for the playback unit 42 to determine the position from which to extract an image for playback will be described.

[0054] As a first example, the playback unit 42 may determine the extraction position of the playback image by referencing viewing direction information recorded together with the recording image data IR. In this example, the recording image generation unit 25 records, together with the recording image data IR, the viewing direction information used when generating the display image data ID. By referencing this viewing direction information, the playback unit 42 can identify movement in the viewing direction without performing the viewing direction movement detection process described above. Therefore, compared to when the playback unit 42 itself performs such movement detection process, it is possible to accurately reduce small changes in the viewing direction with a smaller amount of calculation.

[0055] As a second example, the playback unit 42 may determine the extraction position of the image for playback by referring to the position of the object of attention T. In this example, it is assumed that information indicating the position of the object of attention T is recorded together with the image data for recording IR. Specifically, for example, the playback unit 42 determines the extraction position of the image for playback by taking into account the past position of the object of attention T so that the display position of the object of attention T does not change too much. Alternatively, the extraction position of the image for playback may be determined so that the position of the object of attention T is closer to the center of the image for playback. In this way, by determining the extraction position based on the position of the object of attention T included in the image data for recording IR, it is possible to prevent the position of the object of attention T from changing drastically in the played-back video or to display the object of attention T in a position closer to the center of the screen. This makes it easier for the viewing user to follow the movement of the object of attention T. Note that, as described above, if the recording target portion is determined by taking into account the position, direction of movement, and speed of the object of attention T, it becomes easier to extract an image for playback that focuses on the object of attention T.

[0056] As a third example, the playback unit 42 may determine the extraction position of the playback image by referring to the operation information of the recording user. In this example, operation information indicating the operation performed by the recording user on the operation device 15 is recorded along with the recording image data IR. In this example, the playback unit 42 determines the extraction position based on the operation performed by the recording user, such as the type of operation indicating a direction. This allows the extraction position of the playback image to be determined so as to follow the direction indicated by the recording user, similar to when the extraction position is determined based on the movement of the target object T. For example, if the display target portion includes an object that moves in response to the recording user's instruction, such processing can be expected to prevent the display position of the object from changing significantly or to continue to display it in a specific position, even if the playback unit 42 is unable to identify the object. Note that, as described above, when the recording target portion is determined based on the user's operation information, it becomes easier to extract a playback image that focuses on such operation information.

[0057] As a fourth example, the playback unit 42 may determine the extraction position by referring to information about the point of gaze of the recording user. In this case, as with the attention object T in the second example, information indicating the position of the point of gaze is also recorded together with the image data for recording IR. The playback unit 42 determines the extraction position of the image for playback by referring to this point of gaze information so that the position of the point of gaze does not change significantly or so that the point of gaze is located closer to the center of the image for playback. This makes it easier for the viewing user to confirm the point that the recording user was gazing at. Note that, if the portion to be recorded is determined taking into consideration the position, movement direction, and movement speed of the point of gaze, as described above, it becomes easier to extract an image for playback that focuses on such a point of gaze.

[0058] As a fifth example, the playback unit 42 may extract the display target portion included in the recording image data IR as the playback image in response to an instruction from the viewing user. As explained above, the display target portion of the recording image data IR has content corresponding to the display image data ID actually viewed by the recording user. Therefore, by extracting the display target portion as is and displaying it as the playback image, the image playback device 30 can play back video with the same content as the video viewed by the recording user.

[0059] The image playback device 30 may switch between a process of playing back video corresponding to the display image data ID as shown in this fifth example, and a process of playing back video with suppressed movement in the viewing direction as described above, in response to instructions from the viewing user.

[0060] Note that, if the position of the display target portion included in the recording image data IR is always fixed, as illustrated in FIG. 3, the playback unit 42 can obtain the same image as the display image data ID by extracting an image at that position. On the other hand, if the recording target portion determination unit 24 determines the recording target portion based on information about the attention object T, operation information of the recording user, viewing direction information, gaze point information, etc., the position of the display target portion included in the recording image data IR is not constant. In this case, when generating and recording the recording image data IR, the recording image generation unit 25 may also record information indicating the position of the display target portion within the recording image data IR. The playback unit 42 can determine the display target portion as the extraction target by referring to such information indicating the position of the display target portion. Alternatively, the recording image generation unit 25 may record, together with the recording image data IR, information about the attention object T, operation information of the recording user, viewing direction information, gaze point information, etc., which the recording target portion determination unit 24 referenced when determining the recording target portion. In this case, the playback unit 42 can identify the display target portion contained in the recording image data IR by referring to the recorded information and performing the same processing as when the recording target portion determination unit 24 determined the recording target portion.

[0061] Furthermore, the playback unit 42 may superimpose a given direction guide image that guides the viewing direction on the playback image extracted from the recording image data IR, thereby including this direction guide image among the objects to be displayed on the display device 34. This guide image may be, for example, an image of an arrow that indicates a direction.

[0062] Specifically, for example, the playback unit 42 refers to the viewing direction information recorded together with the recording image data IR and includes a direction guide image indicating the direction of the change in the viewing direction in the display target. As an example, when displaying a playback image extracted from the recording image data IR recorded while the recording user was changing the viewing direction from forward to left, a direction guide image indicating the left direction is also displayed. This allows the viewing user to understand the direction in which the video they are currently viewing is changing. In particular, when displaying video linked to the viewing user's facial direction by determining the position to be extracted from the recording image data IR in accordance with the change in the viewing direction of the viewing user, as described above, the viewing user can view video that changes in a manner similar to the video the recording user was viewing by changing the viewing direction in the direction indicated by the direction guide image. Note that, while the direction guided by the direction guide image is determined based on the viewing direction information recorded together with the recording image data IR, this is not limiting and the direction guided by the direction guide image may also be determined based on the detection results when detecting changes in the viewing direction by analyzing changes in the recording image data IR over time.

[0063] Furthermore, when determining the extraction position from the recording image data IR in response to changes in the viewing user's facial direction, the playback unit 42 may display a direction guide image that indicates the viewing direction of the recording user. For example, when determining the extraction position of the playback image using recording image data IR recorded while the recording user was looking straight ahead, even if the viewing user's facial direction deviates from the forward direction, the extraction position can be determined according to the viewing user's facial direction as long as it is within the range included in the recording image data IR, and the extracted playback image can be displayed. However, if the viewing user's facial direction deviates significantly from the forward direction, it may exceed the range included in the recording image data IR, making it impossible to change the viewing direction in response to the viewing user's facial direction. Therefore, the playback unit 42 displays a direction guide image that guides the viewing direction of the recording user when the viewing direction of the viewing user deviates by more than a predetermined amount from the viewing direction of the recording user identified by the viewing direction information. This allows the viewing user to recognize that they need to return their facial direction to the direction indicated by the direction guide image.

[0064] As described above, the image generating device 10 according to this embodiment generates display image data ID and also generates recording image data IR that overlaps at least a portion of the display image data ID based on the same scene information, thereby making it possible to play back video of the same scene as video later viewed by the recording user in a manner that is easier for the viewing user to view, such as in a manner that is less likely to cause motion sickness or in a manner that makes it easier to identify the object of interest.

[0065] Note that the embodiments of the present invention are not limited to those described above. For example, in the above description, the display device 14 is assumed to be equipped with an attitude sensor, and the viewing direction information receiving unit 22 receives viewing direction information indicating the facial orientation of the recorded user, which is determined based on the detection results of the attitude sensor. However, the method for determining the facial orientation of the recorded user is not limited to this, and the viewing direction information receiving unit 22 may receive viewing direction information determined by various methods. For example, the viewing direction information receiving unit 22 may receive viewing direction information determined by analyzing an image captured from a remote location of the display device 14 to detect the movement of the display device 14, or by analyzing an image captured by a camera located on the display device 14 itself to detect the movement of the display device 14.

[0066] Furthermore, in the above description, the display device 14 is described as a display device that the user can wear on their head to view 3D images, but embodiments of the present invention are not limited to this. Even if the display device 14 is a stationary display device, if the viewing direction set in the virtual space changes minutely depending on the user's operation of the operating device 15, the user may experience nausea if they subsequently view the video as is, as they cannot predict the movements. In such cases, as described above, by generating and recording an image of the recording portion including the display portion as recording image data IR, it is possible to later play back video of the same scene in a manner that reduces changes in the viewing direction. Furthermore, by generating recording image data IR including the recording portion determined taking into account the position of the target object T, it is possible to later play back video focusing on the target object T.

[0067] In the above explanation, the image generating device 10 always determines a range that includes the entire display portion and is wider than the display portion as the recording portion, and the image reproducing device 30 extracts a portion of the recording image data IR as the image for reproduction. However, this is not limiting, and the image generating device 10 may specify a portion to be used as the image for reproduction when generating the recording image data IR, and generate recording image data IR that includes only that portion.

[0068] Specifically, in this embodiment, the recording target portion determination unit 24 determines the recording target portion by a process similar to the process for determining the extraction position executed by the playback unit 42 in the above description. More specifically, for example, the recording target portion determination unit 24 determines the recording target portion based on the most recent viewing direction information obtained so as to reduce small changes in the viewing direction. Alternatively, the recording target portion determination unit 24 may determine the recording target portion so that the position of the target object T is closer to the center. In this case, the recording target portion determination unit 24 determines the recording target portion so that it at least partially overlaps with the display target portion corresponding to the display image data ID and is the same size as the display target portion. The recording image generation unit 25 generates image data of the recording target portion determined in this manner as recording image data IR. Because the recording image data IR generated in this manner does not necessarily include the entire display image data ID, it becomes impossible to later play back video equivalent to the video viewed by the recording user, as in the fifth example of the playback process described above. However, in this example, the playback unit 42 of the image playback device 30 does not need to determine the extraction position of the image to be played back, and the contents of the recorded video data can be played back by processing in the same way as normal video data.

[0069] In the above description, the image generation device 10 is an information processing device located relatively close to the recording user and directly connected to the display device 14 and the operation device 15. However, this is not limited to this. For example, in services such as cloud gaming services, images to be displayed on the screen of the display device 14 may be generated not by a client device directly connected to the display device 14 and the operation device 15 used by the recording user, but by a server device connected to the client device via a communication network. In such cases, the server device connected to the client device used by the recording user via a communication network may function as the image generation device 10 of the present invention. In this case, operation information for the operation device 15 and information about the user's viewing direction acquired by the client device are transmitted to the image generation device 10 via the communication network. The image generation device 10 generates display image data ID based on the received information and transmits it to the client device. The client device then displays the received display image data ID on the screen of the display device 14. Meanwhile, as described above, the image generation device 10 generates recording image data IR along with the display image data ID. The generated image data for recording IR may be stored in the image generating device 10 itself, or may be transmitted to a client device and stored in the client device.

[0070] Similarly, the image playback device 30 may be a server device connected via a communication network to a display device 34 used by a viewing user and a client device to which the operation device 35 is directly connected. In this case, the playback unit 42 of the image playback device 30 transmits playback images extracted from the recordable image data IR to the client device used by the viewing user via the communication network. The playback images received by the client device are displayed on the screen of the display device 34, allowing the viewing user to view images based on the recorded video data.

[0071] The present disclosure may include the following aspects. [Item 1] 1. An image generation device, comprising: one or more processors; The one or more processors: generating, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; determining, as a recording target, at least a portion of the scene, where the display image data and at least a portion of the scene overlap with each other; generating image data of at least a part of the scene that has been determined to be the part to be recorded as image data for recording; an image generating device, The generated image data for recording is subjected to a predetermined reproduction process. Image generating device. [Item 2] Item 1, the image generating device, the one or more processors determine at least a portion of the scene to be recorded, the portion including the display image data and larger than the display image data; Image generating device. [Item 3] Item 2: The image generating device according to item 2, the one or more processors determine, based on object information included in the scene information, at least a portion of the scene that includes a portion corresponding to the display image data, as a recording target; Image generating device. [Item 4] Item 2: The image generating device according to item 2, The one or more processors: Accepting an operation from a user who refers to the display image data; determining, based on the content of the accepted operation, a portion of the scene that includes a portion corresponding to the display image data, as a recording target; Image generating device. [Item 5] Item 2: The image generating device according to item 2, The one or more processors: receiving viewing direction information relating to the face direction of a user who refers to the display image data; determining, using the viewing direction information, at least a portion of the scene that includes a portion corresponding to the display image data; Image generating device. [Item 6] 6. The image generating device according to any one of items 1 to 5, The one or more processors: sequentially acquiring and recording viewing direction information relating to the face direction of a user who refers to the display image data; The sequentially received viewing direction information is used for reproduction processing of the image data for recording. Image generating device. [Item 7] 1. An image reproduction device, comprising: one or more processors; The one or more processors: receiving recording image data that is generated based on scene information together with display image data representing a portion of a scene represented by the scene information, the recording image data including at least a portion of the scene that corresponds to the display image data; reproducing the received image data for recording; an image reproduction device, acquiring field of view information relating to the field of view of a viewer who watches a video in which the recording image data is reproduced, and extracting and displaying at least a part of the recording image data based on the field of view information; Image reproduction device. [Item 8] generating, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; determining, as a recording target, at least a portion of the scene, where the display image data and at least a portion of the scene overlap with each other; generating image data of at least a part of the scene that has been determined to be the part to be recorded as image data for recording; an image generation method, The generated image data for recording is subjected to a predetermined reproduction process. Image generation method. [Item 9] A computer-readable, non-transitory information storage medium, comprising: generating, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; determining, as a recording target, at least a portion of the scene, where the display image data and at least a portion of the scene overlap with each other; generating image data of at least a part of the scene that has been determined to be the part to be recorded as image data for recording; Stores a program for causing a computer to execute a process, The generated image data for recording is subjected to a predetermined reproduction process. Information storage medium. [Explanation of symbols]

[0072] 10 Image generation device, 11 Control unit, 12 Memory unit, 13 Interface unit, 14 Display device, 15 Operation device, 21 Scene information generation unit, 22 View direction information reception unit, 23 Display image generation unit, 24 Recording target portion determination unit, 25 Recording image generation unit, 30 Image playback device, 31 Control unit, 32 Memory unit, 33 Interface unit, 34 Display device, 35 Operation device, 41 Recording image reception unit, 42 Playback unit.

Claims

1. a display image generating unit that generates, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; a determination unit that determines, as a recording target, at least a portion of the scene, the portion including the display image data and being larger than the display image data; a recording image generating unit that generates image data of at least a portion of the scene that has been determined to be recorded as recording image data; Including, an image generating device in which the generated image data for recording is subjected to a predetermined reproduction process, the determining unit determines a position of the portion to be included in the recording target relative to the display image data based on information regarding a position and / or movement of a predetermined object included in the scene information. Image generating device.

2. a display image generating unit that generates, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; a determination unit that determines, as a recording target, at least a portion of the scene, the portion including the display image data and being larger than the display image data; a recording image generating unit that generates image data of at least a portion of the scene that has been determined to be recorded as recording image data; Including, an image generating device in which the generated image data for recording is subjected to a predetermined reproduction process, further comprising an operation receiving unit that receives an operation from a user who refers to the display image data; the determining unit determines a position of the portion to be included in the recording target relative to the display image data based on at least one of a direction of the operation, an amount of operation, and an operation frequency accepted by the operation accepting unit; Image generating device.

3. a display image generating unit that generates, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; a determination unit that determines, as a recording target, at least a portion of the scene, the portion including the display image data and being larger than the display image data; a recording image generating unit that generates image data of at least a portion of the scene that has been determined to be recorded as recording image data; Including, an image generating device in which the generated image data for recording is subjected to a predetermined reproduction process, a viewing direction receiving unit that receives viewing direction information relating to the orientation of a face of a user who refers to the display image data; the determining unit determines a position of the portion to be included in the recording target relative to the display image data based on a change in face direction specified by the viewing direction information; Image generating device.

4. 4. The image generating device according to claim 1, a viewing direction receiving unit that sequentially acquires and records viewing direction information relating to the face direction of a user who refers to the display image data; The sequentially received information on the direction of view is used for the reproduction process of the image data for recording. Image generating device.

5. a recording image data receiving unit that receives recording image data that is generated based on scene information together with display image data that represents a portion of a scene represented by the scene information, the recording image data being at least a portion of the scene, the recording image data including the display image data, and including a portion that is wider than the display image data; a playback unit that plays back the recording image data received by the recording image data receiving unit; Including, the playback unit acquires visual field information relating to a visual field of a viewer who watches a video reproduced from the image data for recording, and extracts and displays at least a part of the image data for recording based on the visual field information; The recording image data is data including a portion whose position relative to the display image data is determined based on information regarding the position and / or movement of a predetermined object included in the scene information. Or, The recording image data is data including a portion whose position relative to the display image data is determined based on at least one of a direction of an operation, an amount of an operation, and an operation frequency received from a user who references the display image data. Or, The recording image data is data including a portion whose position relative to the display image data is determined based on a change in the orientation of the face of a user who is viewing the display image data. Image reproduction device.

6. generating, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; a determining step of determining, as a recording target, at least a portion of the scene, the portion including the display image data and being larger than the display image data; generating image data of at least a portion of the scene determined to be recorded as image data for recording; Including, An image generating method in which the generated image data for recording is subjected to a predetermined reproduction process, In the determining step, a position of the portion to be included in the recording target relative to the display image data is determined based on information regarding a position and / or movement of a predetermined object included in the scene information. Or, further comprising an operation receiving step of receiving an operation from a user who refers to the display image data; In the determining step, a position of the part to be included in the recording target relative to the display image data is determined based on at least one of a direction of the operation, an amount of operation, and an operation frequency accepted in the operation accepting step. Or, further comprising a viewing direction receiving step of receiving viewing direction information relating to the orientation of the face of a user who refers to the display image data; In the determining step, a position of the portion to be recorded relative to the display image data is determined based on a change in face direction specified by the viewing direction information. Image generation method.

7. generating, based on the scene information, image data representing a part of the scene represented by the scene information as display image data; a determining step of determining, as a recording target, at least a portion of the scene, the portion including the display image data and being larger than the display image data; generating image data of at least a portion of the scene determined to be recorded as image data for recording; A program for causing a computer to execute the above, a program for subjecting the generated image data for recording to a predetermined reproduction process, In the determining step, a position of the portion to be included in the recording target relative to the display image data is determined based on information regarding a position and / or movement of a predetermined object included in the scene information. Or, further causing the computer to execute an operation receiving step of receiving an operation from a user who refers to the image data for display; In the determining step, a position of the part to be included in the recording target relative to the display image data is determined based on at least one of a direction of the operation, an amount of operation, and an operation frequency accepted in the operation accepting step. Or, further causing the computer to execute a viewing direction receiving step of receiving viewing direction information relating to the orientation of the face of a user who refers to the display image data; In the determining step, a position of the portion to be recorded relative to the display image data is determined based on a change in face direction specified by the viewing direction information. program.

Images