Information processing apparatus, control method, and storage medium

Abstract

To enable a user to easily recognize the speech contents of a speaker in a virtual space.SOLUTION: A first acquisition unit configured to acquire three dimensional shape data and position information of the three dimensional shape data, a second acquisition unit configured to acquire sound data corresponding to the three dimensional shape data and text data generated from the sound data, a third acquisition unit configured to acquire virtual viewpoint information, and a generation unit configured to generate a virtual viewpoint image based on the sound data, the text data, the three dimensional shape data, the position information, and the virtual viewpoint information, even in a case where the three dimensional shape data is not displayed in the virtual viewpoint image, the generation unit generates the virtual viewpoint image so that the text data generated from the sound data corresponding to the three dimensional shape data is displayed at a position based on the position information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for displaying audio in a virtual space. [Background technology]

[0002] There is a technology that captures images using multiple imaging devices placed in multiple different positions and generates a virtual viewpoint video using the resulting images. The virtual viewpoint video is generated by an information processing device such as a server performing processing such as rendering based on a virtual viewpoint on the images captured by the multiple imaging devices. By displaying the virtual viewpoint video on a display device such as a smartphone or television, a user can view the subject captured by the imaging devices from various angles.

[0003] Furthermore, virtual space and virtual reality experience content includes content such as talks and live music performances, in which the subject's speech is important. In such content, the speech may be made visually recognizable using subtitles or other means. Patent Document 1 discloses a technology for displaying audio in accordance with the distance between the speaker and the receiver in a virtual space and the receiver's viewing angle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-228794 Summary of the Invention [Problem to be solved by the invention]

[0005] Here, for example, if there are multiple subjects, the increased number of targets to be aware of may result in missing or failing to hear what the subjects are saying. However, Patent Document 1 does not disclose a method for recognizing a speaking user from a receiving user.

[0006] Therefore, an object of the present invention is to make it easier for a user to recognize the content of speech made by a speaker in a virtual space. [Means for solving the problem]

[0007] an information processing device comprising: a first acquisition means for acquiring three-dimensional shape data and positional information of the three-dimensional shape data; a second acquisition means for acquiring audio data corresponding to the three-dimensional shape data and text data generated from the audio data; a third acquisition means for acquiring virtual viewpoint information; and a generation means for generating a virtual viewpoint image based on the audio data, the text data, the three-dimensional shape data, the positional information, and the virtual viewpoint information, wherein the generation means generates the virtual viewpoint image so that the text data generated from the audio data corresponding to the three-dimensional shape data is displayed at a position based on the positional information even when the three-dimensional shape data is not displayed in the virtual viewpoint image. [Effects of the Invention]

[0008] According to the present invention, it becomes easier for a user to recognize the content of an utterance made by a speaker in a virtual space. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an example of a block diagram of an image generating device according to a first embodiment. [Figure 2] 4A to 4C are diagrams illustrating an example of a method for determining subtitles by a subtitle data generating unit of the first embodiment and an example of a display form. [Figure 3] 10 is an example of a GUI for setting subtitle display conditions of the subtitle data generating unit of the first embodiment. [Figure 4] 1 is a block diagram illustrating an example of a hardware configuration of an image generation device according to a first embodiment. [Figure 5] 4 is a flowchart illustrating an example of processing performed by the image generation device of the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a display form of a virtual viewpoint video according to the second embodiment. [Figure 7] 10 is a flowchart showing an example of a method for determining a display form of subtitles by a subtitle data generating unit of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on preferred embodiments thereof with reference to the accompanying drawings. The same components will be denoted by the same reference numerals. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0011] First Embodiment In this embodiment, an image generating device, which is an information processing device, performs so-called transcription using speech recognition processing using audio data acquired during imaging, converts the audio data into text data, and describes a method for displaying the text data thus acquired as subtitles in the three-dimensional space of a virtual viewpoint video.

[0012] [composition] 1 is a block diagram showing the configuration of a system according to this embodiment. In this system, an image generation device 1 is connected to an external sound collection device 110, a storage device 111, a virtual viewpoint information input device 112, an imaging device 113, a three-dimensional shape data generation device 114, and an output device 115. The image generation device 1 is also made up of an audio data acquisition unit 101, an audio data determination unit 102, an audio data reproduction unit 103, a model position estimation unit 104, a subtitle data generation unit 105, and a virtual viewpoint image generation unit 106. Here, the image generation device 1 is an example of an information processing device.

[0013] The audio data acquisition unit 101 receives audio data from one or more people from the audio collection device 110 and transmits the audio data to the audio data determination unit 102 and the audio data playback unit 103. The audio collection device 110 acquires the audio of each subject so that the audio data determination unit 102 can analyze the audio of the subject. For example, the acquisition method may involve having each subject carry a pin microphone and acquiring audio data transmitted from each pin microphone. Other methods may include arranging multiple microphones in an array around the subject or installing a shotgun microphone to acquire audio for each subject, or separating the audio for each subject from the acquired audio data. The audio data may include time information for synchronization with the video, and an identifier for identifying each subject.

[0014] The audio data determination unit 102 performs processing to generate text data from the audio data received from the audio data acquisition unit 101 using the speech recognition function of the storage device 111. The text data received from the storage device 111 is transmitted to the subtitle data generation unit 105. The speech recognition function of the storage device 111 is realized, for example, by speech-to-text using an acoustic model and a language model. Note that the storage device 111 may also be a speech recognition system provided by an online cloud service. Note that in this embodiment, the video generation device 1 uses the storage device 111, which is an external device; however, if the audio data determination unit 102 has functions equivalent to those of the storage device 111, the processing of the storage device 111 may be executed by the audio data determination unit 102.

[0015] The audio data reproducing unit 103 transmits the audio data received from the audio data acquiring unit 101 to the virtual viewpoint video generating unit 106 .

[0016] The model position estimation unit 104 detects the position of the subject within the imaging area and transmits the position information to the subtitle data generation unit 105. The imaging area refers to the area imaged by the imaging device 113. The model position estimation unit 104 assigns the position of the subject in the virtual viewpoint video as tracking information based on the position of the subject within the imaging area from the position information of the three-dimensional shape data generated by the three-dimensional shape data generation device 114. The model position estimation unit 104 also assigns an identifier to each piece of three-dimensional shape data generated by the three-dimensional shape data generation device 114. The three-dimensional shape data generation device 114 generates three-dimensional shape data from multiple viewpoint images received from the imaging device 113. The generated three-dimensional shape data is transmitted to the virtual viewpoint video generation unit 106. The three-dimensional shape data is generated based on a foreground image, for example, by the following method. Note that the three-dimensional shape data generation function and the like possessed by the three-dimensional shape data generation device 114 may be possessed by the video generation device 1.

[0017] First, the imaging device 113 captures images from different directions to acquire multiple viewpoint images. Next, from the multiple viewpoint images, a foreground image is acquired in which a foreground region corresponding to a predetermined object, such as a person or a ball, is extracted, and a background image is acquired in which a background region other than the foreground region is extracted. A foreground image is an image in which an object region (foreground region) is extracted from an image captured by the imaging device 113. An object extracted as a foreground region refers to a dynamic object (moving body) that moves (its absolute position and shape may change) when images are captured from the same direction in time series. Examples of objects include people such as players and referees on the field in a sport, a ball in a ball game, or singers, musicians, performers, and presenters in a concert or entertainment event. Next, three-dimensional shape data of the subject is generated using the foreground image. In this embodiment, the three-dimensional shape data is generated using a volume intersection method.

[0018] The background refers to an object that remains stationary or nearly stationary when images are captured from the same direction over time. Examples of such an object include a stage for a concert, a stadium for an event such as a sports match, a structure such as a goal used in a ball game, or a field. However, the background is at least an area that is different from the object in the foreground, and the object may include other objects in addition to the object and background.

[0019] The identifier is data that includes unique ID information for identifying each subject and attribute information for identifying the subject's attributes, such as the role of each subject. The tracking information includes position information of the three-dimensional shape data for each time, and is recorded in association with the identifier. The position of the subject is, for example, the center of gravity of the bounding box that encloses the three-dimensional shape data of the subject. Note that the position information of the subject may be obtained by a wireless tag (for example, a GPS) attached to the subject.

[0020] The subtitle data generation unit 105 accumulates the text data transmitted from the audio data determination unit 102 and the tracking information transmitted by the model position estimation unit 104, and records them in association with an identifier. Subtitle data is generated based on the recorded data and input information received from the virtual viewpoint information input device 112.

[0021] The virtual viewpoint information input device 112 is an operation device including a general device for a user to perform an input operation, such as a controller, keyboard, or mouse, or a tracking controller such as a head-mounted display (HMD). The input information is data related to virtual viewpoint information, such as the position and orientation of the virtual viewpoint. The virtual viewpoint information is a parameter set including a parameter representing the position of the virtual viewpoint in three-dimensional space and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the input information is not limited to the above. For example, the parameter set serving as input information may include a parameter representing the size of the field of view (angle of view) of the virtual viewpoint. The input information may also be a parameter representing the relative amount of operation for changing the current virtual viewpoint.

[0022] The virtual viewpoint is a position in a virtual space that is used when generating a virtual viewpoint video. That is, the virtual viewpoint is a virtual position that differs from the multiple imaging devices 113 that are actually installed around the imaging area.

[0023] Here, the subtitle data generation unit 105 generates / selects subtitle data from multiple subtitle data in accordance with subtitle selection conditions, which will be described later. Furthermore, for the subtitle data to be displayed, it is desirable for the subtitles to be displayed on a plane perpendicular to the viewing user's line of sight for visibility reasons. Therefore, subtitle layout information is transmitted to the virtual viewpoint video generation unit 106 along with the subtitle data. Subtitle data may consist of only text, but a frame may be added around the text to improve visibility. Multiple types of frame lines may be generated for the subtitle data. For example, there are frame lines based on square, round, jagged, or polygonal shapes, as well as subtitle frame lines with corners (so-called horns), known as speech bubble shapes. In this case, the corners of the frame lines may move within the subtitle frame depending on the position of the subject when the subtitles are displayed.

[0024] 2 illustrates an example of a method for determining subtitle selection conditions in the subtitle data generation unit 105 and a method for displaying subtitles based on the determined selection conditions. Although this explanation is based on a three-dimensional space, for the sake of explanation, a two-dimensional diagram will be used.

[0025] FIG. 2(a) shows a means for determining subtitle selection conditions. A virtual viewpoint 201 indicates input information of a viewing user input from the virtual viewpoint information input device 112. A viewing angle region 202 indicates a viewing angle obtained from the input information of the virtual viewpoint 201. The viewing angle region 202 indicates a viewing area of ​​the viewing user in three-dimensional space. The viewing area approximately coincides with the area displayed in the virtual viewpoint video. A gaze direction 203 indicates the viewing direction of the viewing user obtained from the input information of the virtual viewpoint 201. The gaze direction indicates the central field of vision based on the viewing angle of the virtual viewpoint video. Note that the gaze direction may be determined based on a gaze direction detection means using eye tracking of the viewing user by the virtual viewpoint information input device 112.

[0026] Subject 204, subject 205, and subject 206 represent the positions in three-dimensional space of the subjects whose sounds were acquired. In this embodiment, a distance d from line of sight 203 to the subjects and an outside field of view area setting angle D are set in advance as conditions. Perpendicular distances 207 and 208 to the line of sight direction represent the distance between line of sight 203 and subjects 204 and 205 according to distance d. Outside field of view area 209 represents a boundary outside field of view area 202 according to a preset outside field of view area setting angle D.

[0027] The subtitle data generation unit 105 performs the following judgments as a method of determining subtitle selection conditions, and switches the subtitle display method according to the conditions resulting from each judgment. The subtitle data generation unit 105 first judges whether or not the objects 204, 205, and 206 are within the viewing angle region 202. Next, the subtitle data generation unit 105 judges the distance between the position of the object and the line of sight 203 based on the distance d. In FIG. 2(a), the distance between the line of sight 203 and the object 204 is shorter than the distance d, and the distance between the line of sight 203 and the object 205 is longer than the distance d. In this case, the subtitles are selected as shown in FIGS. 2(b) and 2(c).

[0028] FIG. 2(b) illustrates an example of a subtitle display method for the subtitle selection conditions. Virtual viewpoint space 211 represents the three-dimensional space viewed through virtual viewpoint 201. Three-dimensional space 212 represents the three-dimensional space viewed from the same direction as the line of sight viewed through virtual viewpoint 201. That is, in three-dimensional space 212, an area not included in three-dimensional space 211 is an area not displayed in the virtual viewpoint video. Subtitles 213, 214, and 215 are examples of subtitles set under the conditions of FIG. 2(a). Also, for the sake of explanation, FIG. 2(b) depicts subject 206 as if part of it were included in three-dimensional space 211, but the following description will be given assuming that subject 206 is entirely in an area not included in three-dimensional space 211 (outside three-dimensional space 211).

[0029] Subtitles 213 are for subject 204 in Figure 2(a), and the subtitle display condition is that the subject is within viewing angle region 202 and the distance between the line of sight and the subject is shorter than distance d. Subtitles 214 are for subject 205 in Figure 2(a), and the subtitle display condition is that the subject is within viewing angle region 202 and the distance between the line of sight and the subject is longer than distance d. Subtitles 215 are for subject 206 in Figure 2(a), and the subtitle display condition is that the subject is outside viewing angle region 202 and within outside viewing angle region 209.

[0030] Fig. 2(c) shows the three-dimensional space viewed by the viewer from virtual viewpoint 201 using the display method of Fig. 2(b). That is, Fig. 2(c) is the video (image) displayed in the virtual viewpoint video. Because subjects 204 and 205 exist within virtual viewpoint space 211, both subjects are displayed in virtual viewpoint space 211, and subtitles according to the subtitle selection conditions are arranged around subjects 204 and 205. Subject 206 exists within the three-dimensional space, but does not exist in virtual viewpoint space 211, so only the subtitles display the spoken content while displaying corners in the direction of subject 206.

[0031] By displaying subtitles in this manner, the speech content of each subject, both near the viewing user's gaze center and within the viewing angle but not near the gaze center, can be displayed in three-dimensional space. In addition, for subject 206 outside the viewing angle range, only a speech bubble is displayed to show the speech content of the subject outside the viewing angle, making it easier for the viewing user to recognize the content of the speech of the subject outside the viewing angle. This makes it easier for the viewing user to understand the overall content of the speech of the subject outside the viewing angle, even if they are concentrating on the subject they want to view, rather than concentrating on listening to the speech of the subject outside the viewing angle. Furthermore, the viewing user can recognize the position of the subject outside the viewing angle within the viewing angle.

[0032] Here, the method of displaying subtitles that are not displayed in the virtual viewpoint image can be considered as follows. For example, as shown in FIG. 2(a), for a subject located at a location that is a predetermined distance or more from the line of sight of the virtual viewpoint 201, subtitles are displayed from the center to the top of the virtual viewpoint image as shown in FIG. 2(c). In this case, the left and right positions of the subtitles are based on the position with respect to the line of sight from the virtual viewpoint. The virtual viewpoint image is generated so that the display position of the speech bubble moves from the center to the bottom as the subject's position approaches the line of sight of the virtual viewpoint. In this embodiment, when a subject is located at a position that passes through the virtual viewpoint and almost overlaps with the direction perpendicular to the line of sight, the subtitles corresponding to the subject are displayed at the bottom of the virtual viewpoint image. Furthermore, when the subject is located on the opposite side of the line of sight of the virtual viewpoint, i.e., behind the virtual viewpoint, the subtitles corresponding to the subject are displayed at the bottom of the virtual viewpoint image so as to reflect the subject's position relative to the virtual viewpoint. For example, when the subject is directly behind the virtual viewpoint, the subtitles are displayed at the bottom center of the virtual viewpoint image.

[0033] Furthermore, depending on these display conditions, different subtitle shapes (speech bubbles) may be selected for the text data depending on the position of the subject. If each subject has the same subtitle display conditions, different subtitle data may be applied separately. Furthermore, the display position of the subtitles may be moved from within the viewing angle area depending on the movement of the subject's position.

[0034] Fig. 3 shows an example of a UI for setting the subtitle display method of the subtitle data generation unit 105 according to the subtitle selection conditions. In Fig. 3, the range of the display conditions is indicated by the hatched area in the condition display unit 302, with the condition being a range of distance d from the center of the viewing angle surrounded by a thick frame in the condition setting unit 301. If a subject is present in the viewing angle region 202, (I) is selected if it is within the range, and (II) is selected if it is outside the range. Either (I) or (II) is selected for the boundary of the viewing angle region 202.

[0035] The condition setting section 301 is an area for setting conditions for determining subtitle selection conditions, and the setting parameters are specified by the user. When the same parameters are referenced in each subtitle selection condition, the parameters are shared within the area. When the viewer user changes the settings, the selected condition may be highlighted, for example, by doubling the frame.

[0036] The condition display unit 302 is an area that illustrates the range of the subtitle selection condition currently selected by the user. The virtual viewpoint 201, line of sight direction 203, and outside viewing angle area 209 are the same as those in FIG. 2, so their explanation will be omitted. The condition display unit 302 enables the viewing user to visually recognize the range of each subtitle selection condition. Furthermore, the parameters set by the condition setting unit 301 may be changed by the viewing user operating the condition display unit 302.

[0037] The subtitle display unit 303 is an area that displays a list of subtitle shapes (speech bubble shapes) generated or displayed by the subtitle data generation unit 105. As for the subtitle data, a plurality of subtitle data are recorded on the recording medium of the video generation device 1. Furthermore, the video generation device 1 may have a function that allows the viewing user to add and edit subtitle data.

[0038] The display subtitle setting unit 304 is an area for selecting subtitle data for each subtitle selection condition. The subtitle data selected by the display subtitle setting unit 304 may be highlighted by, for example, doubling the frame in the subtitle display unit 303. Furthermore, the subtitle setting may have a function for selecting fixed subtitles for each subject, regardless of the subtitle selection condition. In this embodiment, as an example, the process for determining the subtitles is described, but the subtitles may also be determined based on the distance from the center of the viewing angle. Furthermore, the subtitle data generation unit 105 may have a management function for managing placement information.

[0039] Returning to the explanation of FIG. 1 , the virtual viewpoint video generation unit 106 synchronizes the information generated by the audio data reproduction unit 103, the 3D shape data generation device 114, and the subtitle data generation unit 105 to generate a virtual viewpoint video. The generated virtual viewpoint video is transmitted to the output device 115. The virtual viewpoint video generation unit 106 generates the virtual viewpoint video by performing rendering using, for example, the 3D shape data received from the 3D shape data generation device, the position and orientation of the virtual viewpoint indicated by the virtual viewpoint information, the position of the subtitle data, and background data. However, the method for generating the virtual viewpoint video is not limited to this, and various methods can be used, such as a method for generating a virtual viewpoint video by projective transformation of a captured image without using 3D shape data. Furthermore, the subjects to be captured do not need to be captured at the same capture timing, and it is also possible to output captured subjects in different time series to the same three-dimensional space.

[0040] The output device 115 is a device having a display function, such as a display, a tablet, a smartphone, a PC, or a head-mounted display. The user can view the virtual viewpoint video whose display is controlled by the output device 115.

[0041] The hardware configuration of the image generation device 1 will be described with reference to Fig. 4. The image generation device 1 has a CPU 411, a ROM 412, a RAM 413, an auxiliary storage device 414, a display unit 415, an operation unit 416, a communication I / F 417, and a bus 418.

[0042] The CPU 411 controls the entire image generation device 1 using computer programs and data recorded in the ROM 412 and RAM 413, thereby realizing each function of the image generation device 1 shown in FIG. 4 . Note that the image generation device 1 may have one or more dedicated hardware components different from the CPU 411, and at least a portion of the processing by the CPU 411 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM 412 stores programs that do not require modification. The RAM 413 temporarily stores programs and data supplied from the auxiliary storage device 414 and data supplied from the outside via the communication I / F 417. The auxiliary storage device 414 is configured, for example, by a hard disk drive or the like, and stores various data such as image data and audio data.

[0043] The display unit 415 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the image generation device 1. The operation unit 416 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives operations by the user to input various instructions to the CPU 411. The CPU 411 operates as a display control unit that controls the display unit 415 and an operation control unit that controls the operation unit 416.

[0044] The communication I / F 417 is used for communication with devices external to the image generation device 1. For example, if the image generation device 1 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 417. If the image generation device 1 has a function for wireless communication with an external device, the communication I / F 417 is equipped with an antenna. The bus 418 connects each part of the image generation device 1 to transmit information.

[0045] In this embodiment, the display unit 415 and the operation unit 416 are assumed to exist as separate devices outside the image generation device 1, but at least one of the display unit 415 and the operation unit 416 may also exist inside the image generation device 1.

[0046] [Operation flow] The operational flow of how the video generation device 1 displays audio subtitles on the virtual viewpoint video will be described with reference to the flow diagram shown in FIG.

[0047] In step S501, the sound collection device 110 transmits the generated sound data of the subject to the sound data acquisition unit 101. The generated sound data is then transmitted to the sound data determination unit 102 and the sound data reproduction unit 103.

[0048] In step S502, the audio data determination unit 102 performs analysis such as voice recognition using the storage device 111 on the audio data sent by the audio data acquisition unit 101 in step S501, and generates text data from the spoken content. The text data is sent to the subtitle data generation unit 105.

[0049] In step S503, the three-dimensional shape data generation device 114 generates three-dimensional shape data of the subject from the captured image captured by the imaging device 113, and transmits the virtual viewpoint video to the model position estimation unit 104 and the virtual viewpoint video generation unit 106. The model position estimation unit 104 assigns tracking information to the three-dimensional shape data based on the three-dimensional shape data generated by the three-dimensional shape data generation device 114. The assigned tracking information is transmitted to the subtitle data generation unit 105. The subtitle data generation unit 105 records the received tracking information in association with the text data transmitted in step S502. The subtitle data generation unit 105 determines a display method according to the subtitle selection conditions for each text data in three-dimensional space, based on the recorded information and the input information from the virtual viewpoint information input device 112.

[0050] In step S504, the subtitle data generation unit 105 determines whether to switch subtitles due to a change in the subtitle selection conditions. Here, a grace period for switching the subtitle display by the viewing user is set in advance. The subtitle data generation unit 105 determines whether the subtitle selection conditions will be switched again in a time period equal to or greater than the grace period, with respect to whether or not subtitle switching is applicable. If the subtitle selection conditions are changed and the post-switching subtitle selection conditions are satisfied for the grace period or longer, the process of step S505 is performed. On the other hand, if the conditions remain the same as before the change in the conditions within the grace period, or if the subtitle selection conditions are changed to different conditions, the process of step S506 is performed.

[0051] In step S505, the subtitle data generating unit 105 determines and generates a display method for subtitles of the text data associated with the subject in accordance with the changed subtitle selection conditions.

[0052] In step S506, the subtitle data generating unit 105 determines and generates a display method for subtitles of text data associated with the subject in accordance with the immediately preceding subtitle selection condition.

[0053] In step S507, the subtitle data generation unit 105 determines the placement of the subtitles from the subtitles determined in the previous step (step S505 or step S506), the virtual viewpoint information (the viewing position of the viewing user), and the position of the three-dimensional shape data of the virtual viewpoint video.

[0054] In step S508, the virtual viewpoint video generation unit 106 generates a virtual viewpoint video with subtitles based on the data received from the audio data reproduction unit 103, the three-dimensional shape data generation device 114, and the subtitle data generation unit 105. The virtual viewpoint video generation unit 106 outputs the virtual viewpoint video with subtitles to the output device 115. The virtual viewpoint video output to the output device 115 is viewed by the user.

[0055] The above configuration and operation allow the user to easily recognize what is being said by the speaker in the virtual space when viewing the virtual viewpoint video. In addition, the user can recognize the subject's audio data as subtitles according to the subject's position and the viewing user's position in the virtual viewpoint video, providing a sense of immersion in the virtual viewpoint video.

[0056] <Second embodiment> In this embodiment, the video generation device 1 described in the first embodiment further analyzes the audio level of the audio data, and displays subtitles on the virtual viewpoint video based on the results of the analysis.

[0057] [composition] The functional configuration of the image generation device 1 in this embodiment is the same as that of the first embodiment described with reference to Fig. 1. Below, differences from the first embodiment will be described.

[0058] The audio data determination unit 102 further analyzes audio information such as the audio level of the audio data. The audio information may be, but is not limited to, an average audio level or power. In the video production device 1, thresholds are set in advance to determine the background noise at the time of sound collection and a sudden increase or decrease in audio level as a subtitle display condition. The audio level of the background noise is set to a predetermined value. The audio level of the background noise may also be determined from the results of monitoring the audio level at the start of sound collection by the audio data acquisition unit 101. This determines whether the acquired audio data has an audio level equivalent to the background noise. The audio data determination unit 102 measures, as a subtitle selection condition, how long the audio level of the audio data remains undetectable or equivalent to the background noise.

[0059] The audio data determination unit 102 also obtains and records the audio level from the obtained audio data each time. Thereafter, the audio data determination unit 102 continuously compares the audio level of the next frame and thereafter with the recorded audio level, and determines whether the audio level between the previous and next frames has increased by more than a threshold value.

[0060] The audio level comparison process in this embodiment is executed for each predetermined frame or at a predetermined timing.

[0061] The subtitle data generation unit 105 also has a function of changing the subtitle display method based on the results of analysis by the audio data determination unit 102. The subtitle display of this embodiment is added as a condition to the subtitle selection conditions set in the first embodiment, and the subtitles are changed accordingly. Alternatively, the subtitle data generation unit 105 may superimpose the subtitles on top of the subtitles displayed in the first embodiment. When superimposing, the subtitles are emphasized by, for example, using convergent lines or overlaying set subtitles.

[0062] Fig. 6 illustrates an example of a subtitle selection and display method by the subtitle data generation unit. In Fig. 6(a), a viewing area 601 is a virtual viewpoint image generated based on the virtual viewpoint 201 before the audio level is determined. Subject audio data 602 is an example of audio data generated from audio collected from a subject. Subject subtitles 603 are subtitles that display spoken content according to the subtitle selection conditions.

[0063] In FIG. 6(b), the viewing area 604 is a virtual viewpoint image generated based on the virtual viewpoint 201 when the subject is not speaking. The subject audio data 605 is audio data generated from audio picked up from the subject. This audio data does not capture the content of the subject's speech, or has an audio level equivalent to background noise. In this case, as shown in the viewing area 604, no subtitles are displayed for the subject.

[0064] In FIG. 6(c), the viewing area 606 is a virtual viewpoint image generated based on the virtual viewpoint 201 when the audio level of the audio picked up from the subject becomes higher than immediately before by exceeding a threshold. Subject audio data 607 represents audio data that shows a sudden increase in the subject's audio level. Subject subtitles 608 are subject subtitles 603 superimposed with a display that emphasizes the subtitles. In this way, when there is a sudden increase in the audio level of the subject's audio data in the subject audio data 607, the subtitles are changed to subtitles that give the impression of a loud voice.

[0065] [Operation flow] The operational flow of the method for determining the display format of subtitles will be described using the flow diagram shown in Fig. 7. However, since step S501 is the same as in the first embodiment, the description thereof will be omitted.

[0066] In step S701, the sound collection device 110 transmits the generated sound data of the subject to the sound data acquisition unit 101. The generated sound data is then transmitted to the sound data determination unit 102 and the sound data reproduction unit 103.

[0067] In step S702, the audio data determination unit 102 generates text data from the audio data. The audio data determination unit 102 also analyzes audio information, such as the audio level, for making a determination according to the subtitle selection conditions for the audio data.

[0068] In step S703, the audio data determination unit 102 compares the audio level of the audio data between the previous and next frames based on the analysis result in step S702, and determines whether the audio data is equal to or lower than background noise and has not changed for a predetermined period of time or more. If there is no change, the process of step S703 is performed, and if there is a change, the process of step S704 is performed.

[0069] In step S704, the audio data determination unit 102 transmits data indicating that the audio level is equal to or lower than background noise and has not changed for a predetermined time or longer to the subtitle data generation unit 105. Based on the received information, the subtitle data generation unit 105 applies the subtitle selection conditions and determines not to display subtitles in the virtual viewpoint video.

[0070] In step S705, the audio data determination unit 102 determines whether or not there is an increase in audio level from the audio data of the immediately preceding frame that is equal to or greater than a set threshold. If there is an increase in audio level that is equal to or greater than the threshold, the process of step S706 is performed, and if there is no increase in audio level that is equal to or greater than the threshold, the process of step S707 is performed. This audio data of the immediately preceding frame is not limited to the audio data of the immediately preceding frame, but may be the average value or maximum value of the audio data of frames within a predetermined number of frames from the frame in question.

[0071] In step S706, the audio data determination unit 102 transmits information indicating that the audio level has increased to or above the threshold to the subtitle data generation unit 105. Based on the received information, the subtitle data generation unit 105 applies the subtitle selection conditions and determines to change the subtitles to emphasize them more. For example, the subtitle data generation unit 105 determines to add so-called convergent lines to the subtitles.

[0072] In step S707, the audio data determination unit 102 transmits information indicating that the audio level has not increased beyond the threshold to the subtitle data generation unit 105. The subtitle data generation unit 105 determines subtitle selection conditions based on the audio data and the received information. In this case, the subtitle data generation unit 105 determines the display format of the subtitles under the same conditions as in the first embodiment.

[0073] With the above configuration and operation, the subject's speech is displayed as subtitles in the virtual viewpoint video according to the audio level of the audio data. By making the display format more responsive to the audio data than in the first embodiment, the user's sense of immersion in the virtual viewpoint video can be further enhanced.

[0074] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0076] In the above embodiment, at least one of A and B may be only A, only B, or both A and B.

[0077] The disclosure of this embodiment includes the following configurations and methods.

[0078] (Configuration 1) a first acquisition means for acquiring three-dimensional shape data and position information of the three-dimensional shape data; a second acquiring means for acquiring voice data corresponding to the three-dimensional shape data and text data generated from the voice data; a third acquisition means for acquiring virtual viewpoint information; generating means for generating a virtual viewpoint video based on the audio data, the text data, the three-dimensional shape data, the position information, and the virtual viewpoint information; The generating means generates the virtual viewpoint video so that the text data generated from the audio data corresponding to the three-dimensional shape data is displayed at a position based on the position information even when the three-dimensional shape data is not displayed in the virtual viewpoint video. 1. An information processing device comprising:

[0079] (Configuration 2) The information processing device described in configuration 1 is characterized in that, when the three-dimensional shape data is displayed on the virtual viewpoint image, the generation means generates the virtual viewpoint image based on the position information so that the text data is displayed around the three-dimensional shape data.

[0080] (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the generating means displays the text data in a display form according to the distance from the central visual field of the virtual viewpoint image to the subject.

[0081] (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the generating means generates the virtual viewpoint image so that, when the position of the three-dimensional shape data changes, the text data is displayed at a position corresponding to the change.

[0082] (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the generating means displays the text data in a display format according to the audio level of the audio picked up in the audio data.

[0083] (Configuration 6) The information processing device described in any one of configurations 1 to 5, characterized in that the generation means generates the virtual viewpoint video without using the text data when the audio level of the audio picked up in the audio data remains low.

[0084] (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the generating means displays the text data in a balloon.

[0085] (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the generating means generates the virtual viewpoint image so that the text data is displayed perpendicular to a line of sight of the virtual viewpoint image.

[0086] (Configuration 9) a first acquisition means for acquiring a captured image and position information of a subject; a second acquisition means for acquiring text data based on audio data generated from the audio collected from the subject; a third acquisition means for acquiring virtual viewpoint information; generating means for generating a virtual viewpoint video based on the captured image, the position information, the text data, and the virtual viewpoint information; The generating means generates the virtual viewpoint video so that the text data is displayed at a position based on the position information even when the subject is not displayed. 1. An information processing device comprising:

Claims

1. a first acquisition means for acquiring three-dimensional shape data and position information of the three-dimensional shape data; a second acquiring means for acquiring voice data corresponding to the three-dimensional shape data and text data generated from the voice data; a third acquisition means for acquiring virtual viewpoint information; generating means for generating a virtual viewpoint video based on the audio data, the text data, the three-dimensional shape data, the position information, and the virtual viewpoint information; The generating means generates the virtual viewpoint video so that the text data generated from the audio data corresponding to the three-dimensional shape data is displayed at a position based on the position information even when the three-dimensional shape data is not displayed in the virtual viewpoint video.

1. An information processing device comprising:

2. The information processing device according to claim 1, characterized in that, when the three-dimensional shape data is displayed on the virtual viewpoint image, the generation means generates the virtual viewpoint image based on the position information so that the text data is displayed around the three-dimensional shape data.

3. 2. The information processing apparatus according to claim 1, wherein the generating means displays the text data in a display form according to a distance from a central visual field of the virtual viewpoint image to a subject.

4. 2. The information processing apparatus according to claim 1, wherein said generating means generates said virtual viewpoint image so that, when the position of said three-dimensional shape data changes, said text data is displayed at a position corresponding to the change.

5. 2. The information processing apparatus according to claim 1, wherein said generating means displays said text data in a display form according to the audio level of the audio data.

6. The information processing device according to claim 1, characterized in that the generating means generates the virtual viewpoint image without using the text data when the audio level of the audio picked up in the audio data remains low.

7. 2. The information processing apparatus according to claim 1, wherein the generating means displays the text data in a balloon.

8. 2. The information processing apparatus according to claim 1, wherein the generating means generates the virtual viewpoint image so that the text data is displayed perpendicular to a line of sight of the virtual viewpoint image.

9. a first acquisition means for acquiring a captured image and position information of a subject; a second acquisition means for acquiring text data based on audio data generated from the audio collected from the subject; a third acquisition means for acquiring virtual viewpoint information; generating means for generating a virtual viewpoint video based on the captured image, the position information, the text data, and the virtual viewpoint information; The generating means generates the virtual viewpoint video so that the text data is displayed at a position based on the position information even when the subject is not displayed.

1. An information processing device comprising:

10. a first acquisition step of acquiring three-dimensional shape data and position information of the three-dimensional shape data; a second acquisition step of acquiring voice data corresponding to the three-dimensional shape data and text data generated from the voice data; a third acquisition step of acquiring virtual viewpoint information; a generating step of generating a virtual viewpoint video based on the audio data, the text data, the three-dimensional shape data, the position information, and the virtual viewpoint information, The generating step generates the virtual viewpoint video so that the text data generated from the audio data corresponding to the three-dimensional shape data is displayed at a position based on the position information even when the three-dimensional shape data is not displayed in the virtual viewpoint video.

2. A method for controlling an information processing apparatus comprising:

11. a first acquisition step of acquiring three-dimensional shape data and position information of the three-dimensional shape data; a second acquisition step of acquiring voice data corresponding to the three-dimensional shape data and text data generated from the voice data; a third acquisition step of acquiring virtual viewpoint information; a generating step of generating a virtual viewpoint video based on the audio data, the text data, the three-dimensional shape data, the position information, and the virtual viewpoint information, The generating step generates the virtual viewpoint video so that the text data generated from the audio data corresponding to the three-dimensional shape data is displayed at a position based on the position information even when the three-dimensional shape data is not displayed in the virtual viewpoint video. A program for causing a computer to execute a control method characterized by the above.

Images