Information processing system, information processing method, and program

Abstract

In recent years, there has been a growing demand for providing even more immersive video content. [Solution] The information processing system 100 has the following configuration: an acquisition means for acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the generation location of the acoustic signal; and a control means for performing control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information.

Description

Technical Field

[0001] The present invention relates to an information processing system for displaying a virtual viewpoint image.

Background Art

[0002] There is a virtual viewpoint image generation system that can generate an image seen from a virtual viewpoint specified by a user from an image captured by a photographing system using a plurality of cameras and reproduce it as a virtual viewpoint image.

[0003] In Patent Document 1, a method for automatically imparting a vibration effect to an image based on an analysis result of content data including image data and acoustic data is disclosed. For example, a vibration effect can be created based on an amplitude change of acoustic data extracted from content data and imparted to the image data. Thereby, a more immersive video can be provided to viewers.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, there has been a demand for providing more immersive videos.

[0006] The present disclosure aims to provide a highly immersive virtual viewpoint image.

Means for Solving the Problems

[0007] An information processing system according to one aspect of the present disclosure has the following configuration: an acquisition means for acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the location where the acoustic signal is generated; and a control means for performing control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide highly immersive virtual viewpoint images. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of the information processing system 100 in Example 1. [Figure 2] This figure shows an example of the hardware configuration of the device that constitutes the virtual viewpoint image generation unit 112. [Figure 3] This is a flowchart showing the processing of the vibration information generation unit 114 in Example 1. [Figure 4] This figure shows the structure of object data stored in the storage unit 111 in Example 1. [Figure 5] This is a flowchart showing the processing of the vibration control unit 115 in Example 1. [Figure 6] This is a diagram illustrating the calculation of the virtual viewpoint vibration level in Example 1. [Figure 7] This is a diagram illustrating the calculation of virtual viewpoint vibration in Example 1. [Figure 8] This figure shows an example configuration of the information processing system 100 in Example 2. [Figure 9] This is a flowchart showing the processing of the vibration information generation unit 114 in Example 2. [Figure 10] This is a flowchart showing the processing of the vibration control unit 115 in Example 2. [Modes for carrying out the invention]

[0010] <Embodiment> According to a preferred embodiment of the present invention, the information processing system includes acquisition means for acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the location where the acoustic signal is generated. The information processing system also includes control means for performing control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information.

[0011] The amplitude of the vibration may be determined, for example, according to the distance between the virtual viewpoint and the location where the acoustic signal is generated. The amplitude may be smaller if the distance is large, and larger if the distance is small. The amplitude of the vibration may also be determined according to the volume level contained in the acoustic signal. Furthermore, the amplitude of the depth may be determined according to the volume contained in the acoustic signal. In addition, the amplitude of the vibration may be determined by combining these factors. Moreover, if an object exists between the virtual viewpoint and the location where the acoustic signal is generated, the amplitude of the vibration may be determined to be smaller. The location where the acoustic signal is generated may be the location of the microphone that picked up the acoustic signal, or it may be the location of an object such as a player.

[0012] There are two methods for reflecting vibration in the aforementioned virtual viewpoint image: one that applies a vibration effect to the virtual viewpoint, and another that applies a vibration effect to the display position of the virtual viewpoint image. In the method that applies a vibration effect to the virtual viewpoint, vibration is reflected in the virtual viewpoint image by applying a displacement amount equivalent to vibration to the position of the virtual viewpoint. In the method that applies a vibration effect to the display position of the virtual viewpoint image, vibration is reflected in the virtual viewpoint image by applying a displacement amount equivalent to vibration to the display position.

[0013] This embodiment makes it possible to provide viewers with highly immersive virtual viewpoint images. For example, it is possible to provide virtual viewpoint images in which vibrations change depending on the positional relationship between the virtual viewpoint and the location where the sound signal is generated, as well as the loudness of the sound.

[0014] Further, the vibration may be determined based on information in a specific frequency band included in the acoustic signal. For example, it is determined based on any one of information in the low frequency range, information in the middle frequency range, and information in the high frequency range included in the acoustic signal. Specifically, it is determined based on information in the frequency range with the largest difference in volume level from the previous frame among the information in the low frequency range, information in the middle frequency range, and information in the high frequency range included in the acoustic signal. The vibration is determined according to the type of frequency range and the volume level with the largest difference in volume level from the previous frame. For example, the frequency of the vibration is determined based on the frequency range, and the amplitude of the vibration is determined based on the volume level. For each of the low frequency range, middle frequency range, and high frequency range, vibrations of low frequency, middle frequency, and high frequency are set. Alternatively, the direction of the vibration may be determined based on the frequency range. In the case of the low frequency range, the direction of the vibration may be the one-axis direction, in the case of the middle frequency range, the direction of the vibration may be the two-axis direction, and in the case of the high frequency range, the direction of the vibration may be the three-axis direction.

[0015] According to this aspect, since the vibration is determined based on the details of the acoustic signal, a more immersive virtual viewpoint image can be provided.

[0016] Further, in the present embodiment, the virtual viewpoint image is generated based on a plurality of captured images.

[0017] According to another preferred embodiment of the present embodiment, the information processing method includes an acquisition step of acquiring viewpoint information indicating the position of the virtual viewpoint, an acoustic signal, and position information indicating the generation position of the acoustic signal. The information processing method further includes a control step of performing control to display a virtual viewpoint image in which vibration determined based on the viewpoint information, the acoustic signal, and the position information is reflected.

[0018] According to another preferred embodiment of the present embodiment, the program is for causing a computer to execute each function of the above-described information processing system.

[0019] <Example> The embodiments will be described in detail below with reference to the attached drawings. While the embodiments describe several features, not all of these features are necessarily essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0020] The information processing system 110 is a system that generates a virtual viewpoint image representing the view from a specified virtual viewpoint, based on multiple images obtained from multiple imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint image, but it is not limited to images corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, images corresponding to a viewpoint selected by the user from multiple candidates are also included in the virtual viewpoint image. Furthermore, although this embodiment mainly describes the case where the virtual viewpoint is specified by user operation, the virtual viewpoint may be specified automatically based on the results of image analysis, etc. Furthermore, although this embodiment mainly describes the case where the virtual viewpoint image is a video, the virtual viewpoint image may also be a still image.

[0021] The viewpoint information used to generate a virtual viewpoint image is information indicating the position and orientation of the virtual viewpoint (the direction of the line of sight from the virtual viewpoint). Specifically, the viewpoint information is a parameter set that includes parameters representing the three-dimensional position of the virtual viewpoint and parameters representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. However, the content of the viewpoint information is not limited to the above. For example, the parameter set as viewpoint information may include a parameter representing the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames that make up a video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive points in time.

[0022] The information processing system 110 may have multiple imaging devices that capture the imaging area from multiple directions. The imaging area may be, for example, a stadium where sports such as soccer or karate are played, or a stage where concerts or plays are performed. The multiple imaging devices are installed at different positions surrounding the imaging area and perform imaging synchronously. The multiple imaging devices do not have to be installed around the entire circumference of the imaging area; depending on the limitations of the installation location, they may be installed only around a part of the perimeter of the imaging area. Furthermore, the number of imaging devices is not limited to the example shown in the figure; for example, if the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. In addition, imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may be installed.

[0023] In this embodiment, each of the multiple imaging devices is assumed to be a camera with an independent housing capable of capturing images from a single viewpoint. However, this is not limited to this configuration, and two or more imaging devices may be configured within the same housing. For example, a single camera equipped with multiple lens groups and multiple sensors, capable of capturing images from multiple viewpoints, may be installed as multiple imaging devices.

[0024] A virtual viewpoint image is generated, for example, by the following method. First, multiple images (multiple viewpoint images) are obtained by capturing images from different directions using multiple imaging devices. Next, a foreground image is obtained by extracting the foreground region corresponding to a predetermined object such as a person or a ball, and a background image is obtained by extracting the background region other than the foreground region. In addition, a foreground model representing the three-dimensional shape of the predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring the background model representing the three-dimensional shape of the background, such as a stadium, is generated based on the background image. Finally, the texture data is mapped to the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information to generate a virtual viewpoint image. However, the method of generating a virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of captured images without using a three-dimensional model.

[0025] A foreground image is an image extracted from an image captured by an imaging device, specifically the area of ​​an object (foreground region). An object extracted as the foreground region refers to a dynamic object (a moving body) that is in motion (its absolute position and shape may change) when images are taken from the same direction over time. Examples of such objects include athletes and referees on the field during a sporting event, a ball in a ball game, or singers, musicians, performers, and presenters in a concert or entertainment event.

[0026] A background image is an image of an area (background region) that is at least different from the foreground object. Specifically, a background image is an image obtained by removing the foreground object from the captured image. The background refers to an object that remains stationary or nearly stationary when images are taken from the same direction over time. Such objects include, for example, a stage for a concert, a stadium for an event such as a sports competition, structures such as goals used in ball games, and a field. However, the background is at least an area different from the foreground object, and the object being captured may include other objects besides the foreground object and background.

[0027] Each embodiment will be explained using a horse race as an example of the subject to be filmed and sound recorded. Specifically, the racehorses and jockeys running on the racecourse will be the main subjects of filming and sound recording, and the explanation will focus on examples of reflecting vibrations in a virtual viewpoint image. However, the subject is not limited to horse races, and may be other sports, dance performances, theatrical performances, etc.

[0028] Here, the acoustic data (acoustic signal) is assumed to include race sounds such as the hooves of racehorses and the sound of whipping, as well as ambient sounds such as cheers from spectators and the sound of wind. Race sounds may be captured by microphones attached to the jockeys, or ambient sounds and race sounds may be separated and acquired from the acoustic signals captured by microphones or microphone arrays placed around the course using sound source separation processing, and each may be treated as acoustic data. The acquired acoustic data, including race sounds and ambient sounds, will be treated as virtual sound sources placed in the virtual space when generating virtual viewpoint sound. In addition, 3D models of subjects and virtual sound sources will be placed as objects in the virtual space. Therefore, the position of the virtual sound source can also be said to be the location where the acoustic signal is generated. Hereafter, the position of an object will be referred to as the object position. The object position may be obtained by identifying and tracking each racehorse and jockey from the virtual viewpoint content configuration data, or it may be obtained by an external sensor.

[0029] <Example 1> This embodiment describes a method for generating vibration information by analyzing acoustic data, and then applying vibration to a virtual viewpoint based on the generated vibration information, object position, and virtual viewpoint.

[0030] [System Configuration] Figure 1 is a block diagram showing an example configuration of the information processing system 100 according to this embodiment. The information processing system 100 consists of a storage unit 111, a virtual viewpoint image generation unit 112, a virtual viewpoint sound generation unit 113, a vibration information generation unit 114, and a vibration control unit 115. The information processing system 100 may consist of multiple devices or a single device. In this embodiment, each part of the information processing system 100 is assumed to be handled by one device.

[0031] The storage unit 111 stores virtual viewpoint content configuration data, which is data for generating virtual viewpoint images and virtual viewpoint sounds corresponding to the virtual viewpoint. The virtual viewpoint content configuration data includes image data and object data. The image data here refers to multiple captured images acquired by multiple imaging devices capturing images in synchronous manner. The object data includes time information, object ID, object position, sound data, virtual sound source position, and vibration information. The vibration information includes vibration type and vibration level. The virtual viewpoint content configuration data is generated for each time piece, and the corresponding virtual viewpoint content configuration data can be obtained by specifying the time piece. The vibration information is obtained from the vibration information generation unit 114.

[0032] The virtual viewpoint image generation unit 112 generates a virtual viewpoint image based on viewpoint information indicating the position of the virtual viewpoint and the direction of line of sight from the virtual viewpoint, acquired from the vibration control unit 115, and virtual viewpoint content data acquired from the storage unit 111, and outputs the generated virtual viewpoint image. Specifically, it outputs time information acquired from the vibration control unit 115 to the storage unit 111, and acquires virtual viewpoint content configuration data corresponding to the time information as a response. Next, it generates a virtual viewpoint image based on the virtual viewpoint content data acquired from the storage unit 111 and the viewpoint information acquired from the vibration control unit 115.

[0033] The virtual viewpoint sound generation unit 113 acquires sound data and information indicating the virtual sound source position from the storage unit 111 based on time information acquired from an external source. Then, it generates virtual viewpoint sound based on the acquired sound source data, virtual sound source position, and viewpoint information, and outputs the generated virtual viewpoint sound. For the processing used to generate virtual viewpoint sound, known surround sound processing techniques such as VBAP (Vector Base Amplitude Panning) may be used. Alternatively, binaural processing based on head-related transfer functions (HRTF) may be used, or a combination of these may be used.

[0034] The vibration information generation unit 114 acquires acoustic data stored in the storage unit 111, analyzes the acoustic characteristics for each channel of the acquired acoustic data, generates vibration information associated with each virtual sound source, and outputs the generated vibration information to the storage unit 111.

[0035] The vibration control unit 115 determines the vibration to be applied to the virtual viewpoint based on information indicating the position of the virtual viewpoint and the direction of line of sight from the virtual viewpoint acquired from an external source, and object data acquired from the storage unit 111, applies the generated vibration to the virtual viewpoint, and outputs the vibrated virtual viewpoint.

[0036] [Hardware configuration] Figure 2 shows the hardware configuration of the device constituting the virtual viewpoint image generation unit 112. Note that the devices constituting other parts of the information processing system 100 have a similar hardware configuration, and their explanation is omitted. The device constituting the virtual viewpoint image generation unit 112 includes a CPU 211, ROM 212, RAM 213, auxiliary storage device 214, image display unit 215, sound output unit 216, operation unit 217, communication I / F 218, and bus 219. The CPU 211 implements the functions of the virtual viewpoint image generation unit 112 using computer programs and data stored in the ROM 212 and RAM 213. Note that there may be one or more dedicated hardware components different from the CPU 211, and at least a portion of the processing performed by the CPU 211 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from external sources via the communication I / F 218. The auxiliary storage device 214 is composed of, for example, a hard disk drive and stores various types of data such as image data, audio data, metadata, and configuration information.

[0037] The image display unit 215 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate the system. The sound output unit 216 is composed of, for example, a D / A converter, power amplifier, speaker or headphones, and outputs the generated virtual viewpoint sound. The operation unit 217 is composed of, for example, a keyboard, mouse, joystick, or touch panel, and receives various instructions from the user and inputs them to the CPU 211. The CPU 211 operates as a display control unit that controls the image display unit 215, and as an operation control unit that controls the operation unit 217.

[0038] The communication interface 218 is used for communication with devices outside the system. For example, if each system is connected to an external device by wire, a communication cable is connected to the communication interface 218. If the system has the function of wirelessly communicating with an external device, the communication interface 218 is equipped with an antenna. Bus 219 connects the various parts and transmits information.

[0039] [Operation Flow] The operation of the information processing system 100 according to this embodiment will be described below using the flowcharts shown in Figures 3 and 5. Figure 3 is a flowchart of the processing of the vibration information generation unit 114. Figure 5 is a flowchart of the processing of the vibration control unit 115. Each process is started when the information processing system 100 receives an instruction to start the process related to recording virtual viewpoint content configuration data. The instruction to start the process may be made by user operation via the operation unit 217 of the information processing system 100, or the instruction may be input from another device. However, the execution timing of the processes shown in Figures 3 and 5 is not limited to the above timings.

[0040] The processes shown in Figures 3 and 5 are implemented by the CPU 211 loading the program stored in ROM 212 into RAM 213 and executing it. Note that at least a portion of each process may be implemented by one or more dedicated hardware components separate from the CPU 211.

[0041] The following describes each step in the processing flow of the vibration information generation unit 114 shown in Figure 3.

[0042] In S301, the vibration information generation unit 114 initializes the parameters used to generate vibration information. The parameters to be initialized are, for example, the object ID used to specify the acoustic data to be analyzed. The initial value of the object ID may be 0 or a predetermined value. The acoustic data obtained with object ID = i is referred to as acoustic data i.

[0043] Figure 4 shows the structure of object data stored in the storage unit 111. The object data includes time information, object ID, object position, acoustic data, and vibration information, and any data can be obtained by specifying the time information and object ID. For example, to obtain the acoustic data of object 1 at 13:30:50;12 frames, the corresponding acoustic data 402 can be obtained by specifying the time information as 13:30:50;12 and the object ID as 1. In this embodiment, the number of objects is n.

[0044] By repeating steps S302 to S316 n times, n vibration information points corresponding to n objects can be obtained.

[0045] In S302, the vibration information generation unit 114 acquires the acoustic data to be analyzed from the storage unit 111. Specifically, it specifies the time code for the start of extraction and extracts one frame's worth of acoustic data. If the image data frame rate is 60Hz and the acoustic data is stored in the storage unit 111 at 48kHz, the sample length of the acoustic data i will be 48000 / 60 = 800 samples.

[0046] In S303, the vibration information generation unit 114 extracts the low-frequency components of the acoustic data i using a bandpass filter (BPF). Here, the passband of the BPF in S304 is set to 20-600Hz, which is generally considered to be the low-frequency range.

[0047] In S304, the vibration information generation unit 114 calculates the volume level of the low-frequency range of the acoustic data i and outputs it as the current frame low-frequency level. Here, the amplitude of the acoustic data is squared for each sample, the arithmetic mean is taken, and the square root is taken to obtain the volume level, which is in decibels [dB]. Furthermore, if the quantization bit count of the acoustic data is 24 bits, the volume level is expressed from -144 dB to 0 dB.

[0048] In S305, the vibration information generation unit 114 calculates the difference between the current frame's low-frequency level and the previous frame's low-frequency level and outputs it as the low-frequency level difference. For example, if the current frame's low-frequency level is -20dB and the previous frame's low-frequency level is -50dB, the low-frequency level difference will be -20 - (-50) = 30.

[0049] In S306, the vibration information generation unit 114 updates the previous frame's low-frequency level with the value of the current frame's low-frequency level.

[0050] In S307, the vibration information generation unit 114 extracts the mid-range component of the acoustic data i using a BPF. Here, the passband of the BPF in S306 is set to 800 to 2kHz, which is generally considered to be the mid-range.

[0051] In S308, the vibration information generation unit 114 calculates the volume level of the mid-range acoustic data i and outputs it as the current frame mid-range level. The method for calculating the volume level is the same as in S304, so the explanation is omitted.

[0052] In S309, the vibration information generation unit 114 calculates the difference between the current frame's midrange level and the previous frame's midrange level and outputs it as the midrange level difference. The method for calculating the level difference is the same as in S305, so the explanation is omitted.

[0053] In S310, the vibration information generation unit 114 updates the mid-range level of the previous frame with the value of the mid-range level of the current frame.

[0054] In S311, the vibration information generation unit 114 extracts the high-frequency components of the acoustic data i using a BPF. Here, the passband of the BPF in S309 is set to 800 to 2kHz, which is generally considered to be the high-frequency range.

[0055] In S312, the vibration information generation unit 114 calculates the volume level of the high-frequency range of the acoustic data i and outputs it as the current frame high-frequency level. The method for calculating the volume level is the same as in S304, so the explanation is omitted.

[0056] In S313, the vibration information generation unit 114 calculates the difference between the high-frequency level of the current frame and the high-frequency level of the previous frame, and outputs it as the high-frequency level difference. The method for calculating the level difference is the same as in S305, so the explanation is omitted.

[0057] In S314, the vibration information generation unit 114 updates the high-frequency level of the previous frame with the value of the high-frequency level of the current frame.

[0058] In S315, the vibration information generation unit 114 determines the vibration type i and vibration level i as vibration information i corresponding to the acoustic data i. In this embodiment, as an example, the vibration type parameter is one of "LOW", "MID", or "HIGH". The vibration level parameter is set to a numerical value indicating the vibration level. Here, the vibration type i and vibration level i are selected from the lowest frequency level difference, mid-frequency level difference, and high-frequency level difference of each band, and the largest of these is adopted. For example, if the lowest frequency level difference is the largest of the lowest frequency level difference, mid-frequency level difference, and high-frequency level difference, the vibration type is set to "LOW". Then, the low-frequency vibration level is recorded as the vibration level. In Figure 4, the vibration type 403 is set to "LOW", and the vibration level 404 is set to the low-frequency vibration level of -20.

[0059] In S316, the vibration information generation unit 114 increments i to select the next acoustic data.

[0060] By repeating steps S302 to S316 n times, n vibration types and vibration levels corresponding to n objects can be obtained.

[0061] In S317, the vibration information generation unit 114 stores the obtained vibration type and vibration level in the storage unit 111. Figure 4 shows an image of the configuration of vibration type and vibration level.

[0062] Next, we will explain each step of the operation flow of the vibration control unit 115 shown in Figure 5.

[0063] In S501, the vibration control unit 115 acquires viewpoint information from an external terminal indicating the position of the virtual viewpoint and the direction of line of sight from the virtual viewpoint. Alternatively, viewpoint information indicating the position of the virtual viewpoint and the direction of line of sight from the virtual viewpoint may be acquired from a device mounted on the information processing system 100. The viewpoint information includes the position of the virtual viewpoint, the direction of line of sight from the virtual viewpoint, the field of view, and time information for specifying the playback point of the virtual viewpoint content. As an example, when the position of the virtual viewpoint is expressed in three-dimensional coordinates [X, Y, Z], values ​​such as X=4.0, Y=9.0, and Z=1.5 are calculated. In this embodiment, the unit of the position of the virtual viewpoint is meters [m], and the origin position is the center of the virtual space. When the X-axis is specified to be parallel to the ground surface, the Y-axis is specified to be parallel to the ground surface and perpendicular to the X-axis, and the Z-axis is specified to be perpendicular to the ground surface. Furthermore, when expressing the line of sight from the virtual viewpoint using three angles [Pan, Tilt, Roll], values ​​such as Pan=20.0, Tilt=10.0, and Roll=2.0 are calculated. In this embodiment, the unit of angle is [degrees], and the value range is from -180 to +180. Note that Pan is the angle of rotation parallel to the ground surface, Tilt is the angle of rotation perpendicular to the ground surface, and Roll is the angle of rotation relative to the optical axis of the virtual viewpoint. Also, when expressing the field of view of the virtual viewpoint using focal length zoom, values ​​such as Zoom=6.0 are calculated. In this embodiment, the unit of field of view is [mm].

[0064] In S502, the vibration control unit 115 obtains the object position included in the object data corresponding to the time information included in the viewpoint information from the storage unit 111.

[0065] In S503, the vibration control unit 115 acquires vibration information contained in object data corresponding to time information contained in viewpoint information from the storage unit 111.

[0066] In S504, the vibration control unit 115 acquires the user settings specified in the operation unit. The user settings may include a vibration level user setting that sets the level of vibration, and the vibration level user setting may be a value such as "weak", "medium", "strong", or "OFF".

[0067] In S505, the vibration control unit 115 initializes the parameters used for determining the maximum virtual viewpoint vibration level, as shown in S506 to S510. The parameters to be initialized are, for example, the object ID to be determined and the virtual viewpoint vibration level max. The object ID is a parameter for specifying the vibration information to be determined for the virtual viewpoint vibration level, and the initial value of the object ID may be 0 or a predetermined value. Hereinafter, the object data obtained when object ID = i will be the object position i, vibration type i, and vibration level i, respectively. The virtual viewpoint vibration level max is the maximum value among the virtual viewpoint vibration levels obtained at the virtual viewpoint position for the vibration of each object, and here it is initialized to 0.

[0068] By repeating steps S506 to S511 n times, the object ID that is the source of vibration applied to the virtual viewpoint in that frame can be obtained.

[0069] In S506, the vibration control unit 115 obtains the vibration level i from the object data.

[0070] In S507, the virtual viewpoint vibration level i is calculated. Let A be the vibration level i obtained at object position i, r be the distance from object position i to the virtual viewpoint, and Av be the virtual viewpoint vibration level i obtained at the virtual viewpoint. Av is calculated using the formula Av = A - 20log(r). Figure 6 is a diagram illustrating the calculation of the virtual viewpoint vibration level in this embodiment. Figure 6 shows an example where a virtual viewpoint 602 and objects 603-605 exist in the virtual space 601. If the vibration level of object 604 is -20dB and the distance to the virtual viewpoint is 9.2m, then Av = -20 - 20log(9.2) ≈ -39.2dB.

[0071] In S508, the vibration control unit 115 determines whether the virtual viewpoint vibration level i is equal to the virtual viewpoint vibration level max. Specifically, it compares the virtual viewpoint vibration level i with the virtual viewpoint vibration level max, and if the virtual viewpoint vibration level i exceeds the virtual viewpoint vibration level max, it proceeds to S509. If the virtual viewpoint vibration level i is less than the virtual viewpoint vibration level max, it proceeds to S511.

[0072] In S509, the vibration control unit 115 sets the virtual viewpoint vibration level i to the virtual viewpoint vibration level max.

[0073] In S510, the vibration control unit 115 holds the object ID of the object that has been determined to have the maximum virtual viewpoint vibration level. Specifically, it updates the identifier t, which indicates the object ID of the object that has been determined to have the maximum virtual viewpoint vibration level, with the object ID i. This process makes it possible to identify the object with the highest virtual viewpoint vibration level from among all objects.

[0074] In S511, the vibration control unit 115 increments the object ID, i.e., i, in order to select the next vibration information to be determined.

[0075] By repeating steps S506 to S511 n times, the object ID of the vibration that will be applied to the virtual viewpoint in that frame can be obtained.

[0076] In S512, the vibration control unit 115 calculates the virtual viewpoint vibration corresponding to the object ID obtained above. Specifically, it calculates the virtual viewpoint vibration to be applied to the virtual viewpoint from the object position, vibration type, vibration level, virtual viewpoint position, and user settings corresponding to the object ID. Here, as an example, the virtual viewpoint vibration for vibration type LOW is calculated using a logarithmic spiral. Here, if the plane directly facing the virtual viewpoint is the polar coordinate plane, and a is the scaling coefficient and b is the rotation rate, then the distance r from the origin of the polar coordinate plane, i.e., the position of the virtual viewpoint, can be calculated as r = a × exp(b × θ). Furthermore, the x and y coordinates in the polar coordinate plane can be calculated as x = r × cos(θ) and y = r × sin(θ), respectively. θ is the rotation angle in the polar coordinate plane, and its unit is radians [rad]. When b is positive, as θ decreases, r also decreases; that is, by decreasing θ over time, the convergence of vibration can be represented. Furthermore, by adjusting the scaling coefficient a according to the vibration level and user settings, the strength of the vibration can be represented. Figure 7 is a diagram illustrating the virtual viewpoint vibration in this embodiment. Figure 7 shows a logarithmic spiral graph when Aa=0.1 and b=0.2, and θ is varied from 10π [rad] to 0 [rad].

[0077] In this embodiment, the x and y values ​​calculated above are used as virtual viewpoint vibrations, and virtual viewpoint vibrations are realized by applying them as offset amounts to the position of the virtual viewpoint. Randomness may be added to the calculation of virtual viewpoint vibrations. The same calculation method may be used for vibration types MID and HIGH, or different calculation methods may be used, such as virtual viewpoint vibrations in the x direction only or the y direction only.

[0078] In S513, the vibration control unit 115 applies the virtual viewpoint vibration calculated in S511 to the virtual viewpoint. For example, if the virtual viewpoint coordinates [X, Y, Z] are [5.9, 2.7, 1.5] and the virtual viewpoint vibration [x, y, z] is [0.1, 0.2, 0.0], the virtual viewpoint coordinates to which the virtual viewpoint vibration is applied will be [6.0, 2.9, 1.5].

[0079] With the above configuration, it is possible to analyze acoustic data, which is one of the virtual viewpoint content configuration data, to generate vibration information, and then apply vibration to the virtual viewpoint based on the generated vibration information, object position, and virtual viewpoint.

[0080] <Example 2> In this embodiment, instead of generating and storing vibration information as in Embodiment 1, a method is described in which vibration information is generated by analyzing the generated virtual viewpoint acoustics, and vibration is applied to the virtual viewpoint based on the generated vibration information, object position, and virtual viewpoint.

[0081] [System Configuration] Figure 8 is a block diagram showing an example configuration of the information processing system 100 according to this embodiment. In this embodiment, the operation of the vibration information generation unit 117 and the vibration control unit 118 is different from that of Embodiment 1. Blocks that operate the same as in Embodiment 1 will not be described.

[0082] The vibration information generation unit 117 acquires the virtual viewpoint sound generated by the virtual viewpoint sound generation unit 116, analyzes the acoustic characteristics of the acquired virtual viewpoint sound, generates vibration information from the analysis results, and outputs the generated vibration information to the vibration control unit 118.

[0083] The vibration control unit 118 acquires vibration information from the vibration information generation unit 117, determines the vibration to be applied to the virtual viewpoint based on the vibration information, applies the generated vibration to the virtual viewpoint, and outputs the virtual viewpoint with vibration.

[0084] [Operation Flow] In the following, the operation of the information processing system 100 according to this embodiment will be explained using the flowchart shown in Figure 9. Steps that are the same as those in Embodiment 1 will be omitted from the explanation.

[0085] Figure 9 is a flowchart showing the processing of the vibration information generation unit 114 in this embodiment.

[0086] In S318, the vibration information generation unit 114 acquires the acoustic data to be analyzed from the virtual viewpoint sound generation unit 113. At each frame timing, the most recent frame of the virtual viewpoint sound is extracted and used as the acoustic data to be analyzed.

[0087] In S319, the vibration information generation unit 114 mixes the acoustic data acquired in S318 into a monaural format. Here, since the acoustic data may be in a multi-channel configuration for surround speaker output or a 2-channel configuration for binaural output, the amplitude of the acoustic data may be normalized by dividing the mixed result by the number of channels, depending on the channel configuration.

[0088] Subsequently, the extracted acoustic data is analyzed in the same manner as in Example 1 to determine the vibration information. In Example 1, vibration information was determined for each of the n acoustic data points, but in this example, there is only one target acoustic data point, which corresponds to the case of n=1 in Example 1, so the explanation is omitted.

[0089] Next, we will explain each step of the processing flow of the vibration control unit 115 shown in Figure 10.

[0090] In S514, the vibration control unit 115 acquires vibration information from the vibration information generation unit 114.

[0091] In S515, the vibration control unit 115 acquires the user settings specified in the operation unit. These user settings include vibration level user settings, etc.

[0092] In S516, the vibration control unit 115 calculates virtual viewpoint vibrations to be applied to the virtual viewpoint based on vibration information and user settings. The method for calculating virtual viewpoint vibrations may be the same as in Example 1.

[0093] In S517, the vibration control unit 115 applies the virtual viewpoint vibration calculated in S516 to the virtual viewpoint.

[0094] With the configuration described above, it is possible to analyze the acoustics of the virtual viewpoint to generate vibration information, and then apply vibration to the virtual viewpoint based on the generated vibration information. Furthermore, since vibration can be applied to the virtual viewpoint without generating and storing vibration information as in Example 1, it may be possible to reduce system costs.

[0095] Furthermore, the disclosure of this embodiment includes the following configuration, method, and program. (Composition 1) An acquisition means for acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the location where the acoustic signal originated. Control means for performing control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information, An information processing system characterized by having the following features. (Configuration 2) The information processing system according to configuration 1, characterized in that the control means performs control to display the virtual viewpoint image generated by applying the vibration to the virtual viewpoint. (Composition 3) The information processing system according to configuration 1 or 2, characterized in that the vibration is determined based on the distance between the position of the virtual viewpoint and the location where the acoustic signal is generated. (Composition 4) The information processing system according to any one of configurations 1 to 3, characterized in that the vibration is determined based on the volume level included in the acoustic signal. (Composition 5) The information processing system according to any one of configurations 1 to 4, characterized in that the vibration is determined based on information in a specific frequency band included in the acoustic signal. (Composition 6) The information processing system according to configuration 5, characterized in that the vibration is determined based on any of the low-frequency information, mid-frequency information, or high-frequency information contained in the acoustic signal. (Composition 7) The information processing system according to configuration 6, characterized in that the vibration is determined based on the information in the frequency range where the difference in volume level with the previous frame is the largest, among the low-frequency information, mid-frequency information, and high-frequency information included in the acoustic signal. (Composition 8) The information processing system according to any one of configurations 1 to 7, characterized in that the virtual viewpoint image is generated based on a plurality of captured images. (method) An acquisition step involves acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the location where the acoustic signal originated. A control step that performs control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information, An information processing method characterized by having the following features. (program) A program to cause a computer to function as an information processing system described in any one of configurations 1 through 8. [Explanation of Symbols]

[0096] 100 Information Processing Systems 111 Storage section 112 Virtual viewpoint image generation unit 113 Virtual Viewpoint Sound Generation Unit 114 Vibration information generation section 115 Vibration Control Unit

Claims

1. An acquisition means for acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the location where the acoustic signal originated. Control means for performing control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information, An information processing system characterized by having the following features.

2. The information processing system according to claim 1, characterized in that the control means performs control to display the virtual viewpoint image generated by applying the vibration to the virtual viewpoint.

3. The information processing system according to claim 1, characterized in that the vibration is determined based on the distance between the position of the virtual viewpoint and the location where the acoustic signal is generated.

4. The information processing system according to claim 1, characterized in that the vibration is determined based on the volume level included in the acoustic signal.

5. The information processing system according to claim 1, characterized in that the vibration is determined based on information in a specific frequency band included in the acoustic signal.

6. The information processing system according to claim 5, characterized in that the vibration is determined based on any of the low-frequency information, mid-frequency information, or high-frequency information contained in the acoustic signal.

7. The information processing system according to claim 6, characterized in that the vibration is determined based on the information in the frequency range where the difference in volume level with the previous frame is the largest, among the low-frequency information, mid-frequency information, and high-frequency information included in the acoustic signal.

8. The information processing system according to claim 1, characterized in that the virtual viewpoint image is generated based on a plurality of captured images.

9. An acquisition step involves acquiring viewpoint information indicating the position of a virtual viewpoint, an acoustic signal, and position information indicating the location where the acoustic signal originated. A control step that performs control to display a virtual viewpoint image that reflects vibrations determined based on the viewpoint information, the acoustic signal, and the position information, An information processing method characterized by having the following features.

10. A program for causing a computer to function as an information processing system according to any one of claims 1 to 8.

Images