Image processing device, image processing method, and program

Abstract

The present invention relates to an image processing device, an image processing method and a program, with which it is possible to more appropriately switch viewpoints. Provided is an image processing device comprising: a virtual image generation unit for generating, in accordance with band information pertaining to the transmission of viewpoint images, a virtual image based on viewpoint image-associated information associated with a first viewpoint image and model data that corresponds to the subject of a viewpoint image; and a display control unit for performing control so as to display the generated virtual image before displaying the first viewpoint image. The present invention can be applied, for example, to a reproduction device for reproducing VR content.

Description

[Technical Field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and more particularly to an image processing device, an image processing method, and a program that enable more appropriate viewpoint switching. [Background technology]

[0002] In recent years, much research and development has been done to provide new experiences by allowing users to perceive computer-generated spaces, such as virtual reality (VR), as reality.

[0003] For example, Patent Document 1 discloses a technique relating to a playback device that plays back free viewpoint image data that enables playback of images from any viewpoint. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-187797 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when any viewpoint can be selected, when a user selects a new viewpoint, a waiting time occurs until the image from the new viewpoint can be played back, so it is necessary to switch viewpoints more appropriately, taking this waiting time into consideration.

[0006] The present technology has been developed in light of such circumstances, and makes it possible to switch viewpoints more appropriately. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present technology is an image processing device including: a virtual image generation unit that generates a virtual image based on viewpoint image-related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image; and a display control unit that controls the generated virtual image to be displayed before displaying the first viewpoint image.

[0008] An image processing method according to one aspect of the present technology is an image processing method in which an image processing device generates a virtual image based on viewpoint image-related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image, and controls the image processing device to display the generated virtual image before displaying the first viewpoint image.

[0009] A program according to one aspect of the present technology is a program that causes a computer to function as a virtual image generation unit that generates a virtual image based on viewpoint image-related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image, and a display control unit that controls the generated virtual image to be displayed before displaying the first viewpoint image.

[0010] In an image processing device, an image processing method, and a program according to one aspect of the present technology, a virtual image is generated based on viewpoint image-related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to the transmission of the viewpoint image, and the generated virtual image is controlled to be displayed before the first viewpoint image is displayed.

[0011] The image processing device according to one aspect of the present technology may be an independent device or an internal block constituting a single device. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of a content distribution system to which the present technology is applied. [Figure 2]FIG. 10 is a diagram illustrating an example of switching between multiple viewpoints. [Figure 3] FIG. 2 is a diagram showing an example of a viewpoint image according to a first viewpoint. [Figure 4] FIG. 10 is a diagram showing an example of a viewpoint image according to a second viewpoint. [Figure 5] 10A and 10B are diagrams illustrating examples of virtual images displayed when switching between multiple viewpoints. [Figure 6] FIG. 10 is a diagram illustrating an example of setting a transition path according to a band when switching between multiple viewpoints. [Figure 7] FIG. 10 is a diagram showing an example of a transition route displayed on a map of an event venue. [Figure 8] 1 is a diagram illustrating an example of the configuration of each device in a content distribution system to which the present technology is applied. [Figure 9] FIG. 2 is a diagram showing the flow of data between devices in a content distribution system. [Figure 10] 10 is a flowchart showing a first example of the flow of a viewpoint switching process. [Figure 11] 10 is a flowchart showing a second example of the flow of a viewpoint switching process. [Figure 12] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0013] <1. Embodiments of the present technology>

[0014] (System configuration example) FIG. 1 shows an example of the configuration of a content distribution system to which this technology is applied.

[0015] The content distribution system 1 is a system that provides a service of distributing content such as VR video (VR content) and allowing users to view the content.

[0016] 1, the content distribution system 1 is made up of cameras 10-1 to 10-N, a metadata server 20, a sensor 21, a distribution server 30, and playback devices 40-1 to 40-M, where N and M are integers equal to or greater than 1.

[0017] For example, when distributing VR content such as a music concert, cameras 10-1 to 10-N and various sensors 21 are installed in an event venue such as a music concert venue. Each of cameras 10-1 to 10-N is installed in a location where it can capture images of performers on stage from various angles. The various sensors 21 are also installed in a location where they can detect a detection target, and are connected to the metadata server 20.

[0018] Camera 10-1 is composed of an optical system, an image sensor, a signal processing circuit, etc. Camera 10-1 is installed at a predetermined location in the event venue and captures images of performers on stage. The images captured by camera 10-1 can be said to be viewpoint images captured from a first viewpoint position.

[0019] Each of the cameras 10-2 to 10-N is configured similarly to the camera 10-1, and captures viewpoint images from different viewpoint positions by photographing the performers on the stage from the respective predetermined locations where the cameras are installed.

[0020] The sensor 21 includes sensors such as a physical sensor and a distance measurement sensor. The physical sensor is a sensor that detects electrical or magnetic quantities, mechanical quantities, and physical quantities such as light and temperature. The distance measurement sensor is a sensor that measures the distance to an object using a method such as ToF (Time of Flight).

[0021] The metadata server 20 generates metadata related to viewpoint images based on sensor data detected by various sensors 21 and viewpoint image data captured by the cameras 10-1 to 10-N. This metadata includes meta information such as information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image.

[0022] The viewpoint image data captured by the cameras 10-1 to 10-N and data such as metadata generated by the metadata server 20 are transmitted to the distribution server 30 via a network 50-1. The network 50-1 includes a communication network such as the Internet, an intranet, or a mobile phone network, and may be a dedicated line.

[0023] The distribution server 30 is a server (cloud server) that distributes content, and is installed in a data center, etc. The distribution server 30 receives data such as viewpoint image data and metadata via the network 50-1, and performs processing to enable distribution of VR content such as a music concert.

[0024] The playback device 40-1 is a device capable of playing back content such as a head mounted display (HMD). In response to a user's operation, the playback device 40-1 transmits a VR content distribution request to the distribution server 30 via a network 50-2. The network 50-2 includes a communication network such as the Internet, an intranet, or a mobile phone network.

[0025] When the distribution server 30 receives a distribution request from the playback device 40-1, it transmits VR content data corresponding to the distribution request to the playback device 40-1 via the network 50-2. The VR content data includes viewpoint image data, metadata, etc. As a result, the playback device 40-1 plays back VR content such as a music concert, which is then viewed by the user.

[0026] Playback devices 40-2 to 40-M are configured similarly to playback device 40-1 and can play VR content distributed from distribution server 30. In the following description, when there is no need to particularly distinguish between playback devices 40-1 to 40-M, they will be referred to as playback device 40. Furthermore, network 50-1 and network 50-2 are not limited to being different networks, and may be the same network, and will be referred to as network 50 in the following description.

[0027] In the content distribution system 1 configured as described above, performers performing on a stage in an event venue are photographed from various angles by multiple cameras 10, so that a playback device 40 that plays VR content such as a music concert can display performers according to various viewpoints as subjects in a viewpoint image.

[0028] For example, imagine a scene in which a group of three performers, PR1, PR2, and PR3, are performing a song on stage ST in an event venue, as shown in Figure 2. In the example of Figure 2, facing the stage ST, camera 10-1 is installed on the left, camera 10-2 is installed in front, and camera 10-3 is installed on the right.

[0029] The field of view A1 of camera 10-1 includes three performers as subjects, but since camera 10-1 is installed on the left side facing the stage ST, the captured image obtained is, for example, a viewpoint image PI1 such as that shown in Figure 3.

[0030] The field of view A3 of camera 10-3 includes three performers as subjects, but since camera 10-3 is installed on the right side facing the stage ST, the captured image obtained is, for example, a viewpoint image PI3 such as that shown in Figure 4.

[0031] Although not shown, camera 10-2 is installed facing the stage ST, and therefore the captured image obtained is a viewpoint image of the three performers from the front within the range of angle of view A2.

[0032] At this time, the user viewing the VR content of the music concert on the playback device 40 may position his / her viewpoint at a viewpoint P according to the angle of view A2 of the front camera 10-2. C From the left camera 10-1, the viewpoint P N When you switch to the current viewpoint P C A new perspective from P N The viewpoint transition is performed (viewpoint transition TR11 in FIG. 2).

[0033] In addition, a user viewing VR content on the playback device 40 may change his / her viewpoint to a viewpoint P according to the angle of view A2 of the front camera 10-2. C From the viewpoint P according to the angle of view A3 of the right camera 10-3, N When you switch to the current viewpoint P C A new perspective from P N The viewpoint transition is performed (viewpoint transition TR12 in FIG. 2).

[0034] In this way, when the playback device 40 allows the user viewing VR content to select from multiple viewpoints, the viewpoint P C From the viewpoint after switching P N When the viewpoint transition (viewpoint transition TR11, TR12 in Figure 2) is performed, the viewpoint P N The viewpoint image is switched after a certain waiting time has elapsed until the viewpoint image corresponding to the selected viewpoint image (viewpoint image PI1 in FIG. 3, viewpoint image PI3 in FIG. 4) becomes ready to be played back.

[0035] Therefore, in the playback device 40, when a user watching VR content can select multiple viewpoints, problems can arise such as the viewpoint transition taking a long time to complete, or it being difficult to grasp where the new viewpoint is after the viewpoint has been switched.

[0036] Therefore, the content distribution system 1 solves this problem and allows the user to switch between viewpoints more appropriately when multiple viewpoints are available for selection.

[0037] That is, in the content distribution system 1, when transmitting a viewpoint image, metadata including meta-information such as information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image, and model data corresponding to the subject in the viewpoint image are generated and transmitted to the playback device 40. The playback device 40 generates a virtual image based on the metadata and model data, and displays the virtual image until preparations for playback of the viewpoint image from the new viewpoint after the viewpoint has been switched are complete.

[0038] (Example of virtual image display) FIG. 5 shows an example of a virtual image displayed when switching between multiple viewpoints.

[0039] In FIG. 5, in the playback device 40, a user viewing VR content positions his / her viewpoint at a viewpoint P according to the angle of view A1 of the left camera 10-1. C From the viewpoint P according to the angle of view A3 of the right camera 10-3, N When you switch to P C From the viewpoint after switching P N The viewpoint transition is performed (viewpoint transition TR21 in FIG. 5).

[0040] At this time, while the viewpoint is being switched, the playback device 40 C After the operation to switch is performed, the viewpoint P N For example, the playback device 40 displays the virtual image VI until the viewpoint image at the previous viewpoint P (the viewpoint image PI3 in FIG. 4) is ready to be played back. C From the viewpoint after switching P N It is possible to display a virtual image VI according to a viewpoint (position of the virtual camera) that transitions between viewpoints up to .

[0041] The virtual image VI includes characters PV1, PV2, and PV3 corresponding to performers PR1, PR2, and PR3 on the stage ST. That is, the metadata includes information for generating the subject in the viewpoint image PI, and the model data includes an image related to a specific character associated with the subject, so that a 3D character PV can be generated that corresponds to the performer PR as the subject.

[0042] Figure 5 shows an example in which a 3D character PV corresponding to the performer PR is displayed as the virtual image VI, but the virtual image VI is not limited to a 3D character and can be any image that is associated with the subject of the viewpoint image PI, such as a live-action 3D model.

[0043] The virtual image VI is not limited to a three-dimensional image such as a 3D character, but may also be a two-dimensional image such as a 2D character. For example, considering the processing load of the playback device 40, displaying a three-dimensional virtual image VI imposes a higher load than displaying a two-dimensional virtual image VI. Therefore, depending on the processing capability (processor performance, etc.) of the playback device 40, the two-dimensional virtual image VI or the three-dimensional virtual image VI can be displayed.

[0044] Furthermore, in the content distribution system 1, a virtual image based on the metadata and model data is generated according to bandwidth information related to the transmission of viewpoint images. That is, if the network 50 is a best-effort network or the like, the bandwidth of the network 50 varies depending on the environment in which the user uses the playback device 40, and this changes the time it takes to switch viewpoints, so the bandwidth information is used.

[0045] For example, in the playback device 40, when the viewpoint transitions in response to a user's viewpoint switching operation, a virtual image VI corresponding to the viewpoint (position of the virtual camera) transitioning between the viewpoints before and after the switch can be displayed, and the transition path and transition speed of the viewpoint can be adjusted in accordance with bandwidth information.

[0046] Specifically, in the playback device 40, the viewpoint P C The viewpoint position information about the viewpoint position of and the viewpoint P N Based on the viewpoint position information about the viewpoint position of C and the viewpoint after switching P N When generating a virtual image VI according to the trajectory information obtained by the above method, the trajectory information can be determined using the bandwidth information. The trajectory information includes information about the transition path between the viewpoints before and after switching and information about the transition speed.

[0047] Furthermore, the playback device 40 determines whether the viewpoint P before switching is correct based on the bandwidth information of the network 50. C Viewpoint after switching from P NThe time required for the transition (the time required from the start of switching of the viewpoint to the completion) is calculated, and if the transition time is shorter than a predetermined value, the transition route is set to the shortest route, and if the transition time is longer than the predetermined value, a longer route can be set as the transition route.

[0048] In other words, since there is a relationship in which the greater the bandwidth, the shorter the length of the transition path, and the smaller the bandwidth, the longer the length of the transition path, there is a negative correlation between the bandwidth indicated by the bandwidth information and the length of the transition path (trajectory distance) indicated by the trajectory information.

[0049] FIG. 6 shows an example of setting a transition path according to a band when switching between multiple viewpoints.

[0050] In FIG. 6, when the transition time according to the band is shorter than a predetermined value, the viewpoint P C and the viewpoint after switching P N A viewpoint transition TR31 is set as the shortest route between the viewpoints before and after the switch. On the other hand, if the transition time according to the bandwidth is longer than a predetermined value, a viewpoint transition TR32 is set as a longer route. Regardless of whether a viewpoint transition TR31 or a viewpoint transition TR32 is set, a virtual image VI according to the viewpoint transitioning between the viewpoints before and after the switch is displayed.

[0051] 6 shows an example in which the length of the transition path is changed based on a threshold determination of the transition time according to the band, but it is also possible to change the transition speed of the viewpoint when transitioning between viewpoints. That is, with the same path for transitioning the viewpoint, if the transition time according to the band is shorter than a predetermined value, the transition speed of the viewpoint can be set to a faster speed, and if the transition time according to the band is longer than the predetermined value, the transition speed of the viewpoint can be set to a slower speed, thereby creating a difference in speed.

[0052] Furthermore, both the length of the transition path and the transition speed may be adjusted by determining a threshold value for the transition time according to the bandwidth. Furthermore, when determining a threshold value for the transition time according to the bandwidth, the processing capacity of the playback device 40 may be taken into consideration. This processing capacity includes the performance of the processor, the memory capacity, the display resolution, etc.

[0053] In addition, in the playback device 40, if a predetermined operation is performed by the user during a transition of the viewpoint in response to a viewpoint switching operation, the transition of the viewpoint may be stopped and the viewpoint may be switched to the viewpoint after the change in the viewpoint in response to the operation. Also, by darkening the surroundings of the field of view of the user using the playback device 40 in accordance with the speed of the transition of the viewpoint, so-called VR sickness can be suppressed.

[0054] Furthermore, the playback device 40 may hide the virtual image VI by instantly switching the viewpoint image PI between before and after the viewpoint is switched in response to the viewpoint switching operation. For example, the virtual image VI may be hidden when the bandwidth amount is greater than a predetermined value based on the bandwidth information of the network 50. However, even when the virtual image VI is hidden, information regarding the transition of the viewpoint is presented to the user.

[0055] For example, as shown in FIG. 7, in the playback device 40, the viewpoint P C The viewpoint position information about the viewpoint position of and the viewpoint P N Based on the viewpoint position information regarding the viewpoint position of the event venue, an image including information indicating the viewpoint transition TR21 is displayed on a map of the event venue. This allows the user to check the displayed map and recognize how the viewpoint has moved. Note that the display example in FIG. 7 is just one example, and for example, characters corresponding to the performers on the stage ST may be displayed on the map of the event venue together with information regarding the viewpoint transition.

[0056] In this way, by displaying a virtual image when switching between multiple viewpoints, the user can enjoy viewing the virtual image while waiting for the viewpoint to be switched. Also, by displaying a virtual image corresponding to the viewpoint that transitions between viewpoints before and after the switch, the user can grasp where the next viewpoint will be.

[0057] (Metadata configuration example) As mentioned above, the metadata includes meta-information such as information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image, but more specifically, the metadata includes the following information as viewpoint image-related information related to the viewpoint image:

[0058] That is, the metadata includes, as information about the viewpoint position of the viewpoint image, information about the position, posture, and angle of view of the camera 10 that captures the viewpoint image. The metadata also includes, as information about the subject in the viewpoint image, information about the position, posture, and time corresponding to the subject (performer). The information about the subject may include information that identifies the subject, such as the name of the performer.

[0059] The metadata may include information about the space in which the subject (performer) is located, such as an event venue, and information about various objects present in the space. For example, information about the event venue may include control information for spotlights and lighting. Information about various objects may include information about the performer's clothing, the type and position of musical instruments, and so on.

[0060] These metadata are generated by the metadata server 20, but information that can be analyzed by analytical processing, such as information about objects, may be added based on the results of analytical processing on the distribution server 30 or an external server (not shown).

[0061] In playback device 40, metadata may be acquired at the following times, for example. That is, playback device 40 constantly acquires metadata that is repeatedly transmitted at predetermined intervals from distribution server 30. By constantly acquiring metadata in this way, when the user switches the viewpoint, processing can be started instantly using the acquired metadata.

[0062] Furthermore, when a user performs an operation to switch the viewpoint (for example, by pressing a switch button), playback device 40 can acquire the metadata transmitted from distribution server 30 by requesting the metadata from distribution server 30. At this time, the request for metadata may include information that specifies the viewpoint to be switched, so that only the information related to necessary camera 10 is acquired.

[0063] Note that these acquisition timings may be combined, and for example, the metadata may include meta information that is constantly acquired and meta information that starts to be acquired when a viewpoint switching operation is performed. In other words, the acquisition timing can be made different for each piece of meta information included in the metadata.

[0064] (Application example of network slicing) In the content distribution system 1, network slicing technology can be used when transmitting data over the network 50. Network slicing is a technology that virtually divides network resources and controls communications according to their respective uses. The uses here include high reliability and low latency, high speed and large capacity, etc.

[0065] For example, in the content distribution system 1, in order to transmit metadata including information on the position, posture, and angle of view of the camera 10 and information on the position, posture, etc. of the subject (performer) of the viewpoint image in a highly reliable slice, the metadata server 20 associates slice-related information for specifying a highly reliable slice to be assigned to each piece of meta information included in the metadata. Then, a base station (not shown) via which the metadata server 20 and the distribution server 30 are connected assigns each piece of meta information to a slice specified based on the slice-related information, and transmits each piece of meta information to the distribution server 30 using a highly reliable slice.

[0066] Here, the slice-related information is not particularly limited as long as it can identify the slice to be assigned, and examples include naming each piece of meta information, assigning an ID corresponding to the slice to be assigned to each piece of meta information, etc. Furthermore, although high reliability and low latency and high speed and large capacity have been exemplified as slices, the slice is not limited to these, and any slice set by virtually dividing may be used.

[0067] Similarly, network slicing technology can be used when sending various types of metadata from the distribution server 30 to the playback device 40. In this case, slice-related information is associated with each piece of meta information, just as in the case of transmitting various types of meta information from the metadata server 20 to the distribution server 30. Then, in a base station (not shown) via which the distribution server 30 and the playback device 40 are connected, each piece of meta information is transmitted using a slice identified by the slice-related information associated with each piece of meta information.

[0068] Here, all meta information may be sent as a highly reliable slice, or, for example, only the minimum required meta information (for example, information on the position and posture of the subject and the position of the viewpoint after switching) may be controlled to be sent as a highly reliable slice. Alternatively, in a case where metadata is constantly being sent, control may be exercised to always continue sending meta information as a highly reliable slice. Furthermore, control may be exercised to send as a highly reliable slice only when a user instructs to switch the viewpoint.

[0069] In this way, by using network slicing technology to transmit data such as viewpoint image data while reliably ensuring a bandwidth for transmitting metadata, the playback device 40 can reliably receive the metadata. This allows the playback device 40 to reduce the processing load when displaying a virtual image during viewpoint switching. Note that since the amount of data for metadata is smaller than that for viewpoint image data and the like, it is easier to ensure a bandwidth for metadata compared to other data.

[0070] In addition, when the network 50 is configured to include a wireless communication network such as a fifth generation mobile communication system (5G: 5th Generation), the distribution server 30 may be configured as a device on the RAN (Radio Access Network) side.

[0071] (Configuration of each device) FIG. 8 shows an example of the configuration of each device in a content distribution system to which the present technology is applied.

[0072] The camera system 100 is made up of cameras 10-1 to 10-N, a 3D model generation unit 101, and a communication unit .

[0073] Cameras 10-1 to 10-N are installed at predetermined locations within the event venue. Each of cameras 10-1 to 10-N captures images of performers on stage from its own location and supplies viewpoint image data from different viewpoints to 3D model generation unit 101 and communication unit 102.

[0074] The 3D model generation unit 101 generates 3D model data corresponding to the subject (performer) in the viewpoint image based on the viewpoint image data supplied from the cameras 10-1 to 10-N, and supplies the data to the communication unit 102. This 3D model data is data of a live-action 3D model.

[0075] The communication unit 102 transmits the viewpoint image data supplied from the cameras 10-1 to 10-N and the 3D model data supplied from the 3D model generation unit 101 to the distribution server 30 via the network 50.

[0076] The metadata server 20 includes an analysis unit 201 , a metadata generation unit 202 , and a communication unit 203 .

[0077] The physical sensor 21-1 is installed at a predetermined location within the event venue, detects physical quantities such as electrical or magnetic quantities, and supplies sensor data corresponding to the detection results to the metadata generation unit 202. The distance measurement sensor 21-2 is installed at a predetermined location within the event venue, and supplies sensor data corresponding to the measurement results of the distance to an object to the metadata generation unit 202.

[0078] The analysis unit 201 analyzes the viewpoint image data supplied from the camera 10, and supplies analysis data corresponding to the analysis results to the metadata generation unit 202. Note that the camera 10 is not limited to the cameras 10-1 to 10-N shown in FIG. 1 etc., but may be a camera installed elsewhere within the event venue.

[0079] The metadata generation unit 202 generates metadata based on the sensor data supplied from the physical sensor 21-1 and the distance measurement sensor 21-2 and the analysis data supplied from the analysis unit 201, and supplies the generated metadata to the communication unit 203.

[0080] The metadata generated by the metadata generation unit 202 includes meta information such as information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image. Information about the viewpoint position of the viewpoint image includes information about the position, posture, and angle of view of the camera 10. Information about the subject in the viewpoint image includes information about the position and posture corresponding to the subject (performer).

[0081] The communication unit 203 transmits the metadata supplied from the metadata generation unit 202 to the distribution server 30 via the network 50 .

[0082] The viewpoint image data and 3D model data transmitted from the camera system 100 and the metadata transmitted from the metadata server 20 are transmitted by the distribution server 30 to the playback device 40 via the network 50.

[0083] Note that part of the processing executed by the camera system 100 and the metadata server 20 may be executed by the distribution server 30 or an external server (not shown). For example, the processing for generating 3D model data may be executed by the distribution server 30 or an external server.

[0084] The playback device 40 is composed of a communication unit 401 , an image processing unit 402 , a display unit 403 , an input unit 404 , and a detection unit 405 .

[0085] The communication unit 401 is a communication module that supports wireless communication such as a wireless local area network (LAN) or cellular communication (for example, LTE-Advanced or 5G) or wired communication.

[0086] The communication unit 401 receives viewpoint image data, 3D model data, and metadata distributed from the distribution server 30 via the network 50, and supplies them to the image processing unit 402. The communication unit 401 also receives bandwidth information of the network 50 from a measurement server (not shown) or the like, and supplies it to the image processing unit 402. The bandwidth information is obtained periodically because the bandwidth of the network 50 is not always the same. The measurement server is a server that measures the bandwidth of the network 50 at that time in response to a request from a device connected to the network 50, and provides the measured bandwidth information.

[0087] The image processing unit 402 is a player that plays back content such as VR content. The image processing unit 402 has a viewpoint image processing unit 411, a transition calculation unit 412, a virtual image generation unit 413, a storage unit 414, a switching unit 415, and a playback unit 416. The viewpoint image processing unit 411, the virtual image generation unit 413, and the switching unit 415 form a display control unit 410.

[0088] The viewpoint image processing unit 411 processes the viewpoint image data supplied from the communication unit 401 and supplies the viewpoint image data corresponding to the user's viewpoint to the switching unit 415 .

[0089] The transition calculation unit 412 calculates the transition path and transition speed of the viewpoint that transitions between the viewpoints before and after switching based on the metadata and bandwidth information supplied from the communication unit 401 and the terminal position information supplied from the detection unit 405, and supplies trajectory information including the calculation results to the virtual image generation unit 413.

[0090] The virtual image generation unit 413 generates virtual image data based on the 3D model data and metadata supplied from the communication unit 401 and the trajectory information supplied from the transition calculation unit 412 , and supplies the generated data to the switching unit 415 .

[0091] Furthermore, when generating virtual image data, the virtual image generation unit 413 can use data such as 3D characters and stage maps stored in the storage unit 414. The storage unit 414 is an auxiliary storage device including a semiconductor memory such as a nonvolatile memory. The storage unit 414 may be configured as an internal storage or may be an external storage such as a memory card.

[0092] The switching unit 415 is supplied with viewpoint image data from the viewpoint image processing unit 411 and virtual image data from the virtual image generation unit 413. The switching unit 415 switches between the viewpoint image data and the virtual image data, and supplies either one of the image data to the playback unit 416.

[0093] The playback unit 416 performs playback processing using the viewpoint image data or virtual image data supplied from the switching unit 415 based on the user's viewpoint information supplied from the detection unit 405, and displays the viewpoint image or virtual image on the display unit 403. The display unit 403 is a display including an OLED (Organic Light Emitting Diode) panel, a liquid crystal panel, or the like.

[0094] The input unit 404 is configured with physical buttons, touch sensors, etc. When a user performs a viewpoint switching operation, the input unit 404 supplies operation data corresponding to the operation to the image processing unit 402. The image processing unit 402 performs viewpoint switching processing in accordance with the operation data supplied from the input unit 404.

[0095] The detection unit 405 has a sensing function using various sensors such as a gyro sensor, and an eye-tracking function. The detection unit 405 detects information about the position of the playback device 40 (terminal position information) and information about the location of the user's viewpoint (where the user is looking) (viewpoint information), and supplies the information to the image processing unit 402.

[0096] FIG. 9 shows the flow of data between the devices in the content distribution system 1 configured as above.

[0097] The viewpoint image data captured by the multiple cameras 10 and the corresponding 3D model data, and the metadata generated by the metadata server 20 are transmitted to the distribution server 30 via the network 50 (S11, S12).

[0098] In response to a request from the playback device 40, the distribution server 30 transmits the viewpoint image data to the playback device 40 via the network 50 (S13, S14). As a result, a viewpoint image corresponding to the viewpoint image data is displayed on the playback device 40. The metadata and 3D model data are transmitted at any timing, such as repeatedly at predetermined intervals or when requested by the playback device 40.

[0099] In playback device 40, it is determined whether to switch the viewpoint based on operation data from input unit 404 (S15). For example, if the user performs an operation to switch the viewpoint and it is determined that the viewpoint is to be switched ("Yes" in S15), the process proceeds to step S16.

[0100] In step S16, a viewpoint switching process is executed by the playback device 40. Details of this viewpoint switching process will be described later with reference to the flowcharts of FIGS.

[0101] (Flow of viewpoint switching process) First, a first example of the viewpoint switching process corresponding to step S16 in Fig. 9 will be described with reference to the flowchart in Fig. 10. However, in this first example, it is assumed that metadata and the like transmitted at predetermined intervals from the distribution server 30 have already been acquired.

[0102] In step S101, the transition calculation unit 412 determines whether the bandwidth amount indicated by the bandwidth information of the network 50 is equal to or less than a predetermined value. The predetermined value can be a predetermined fixed value or a variable value depending on the processing capability (processor performance, etc.) of the playback device 40.

[0103] If it is determined in the determination process of step S101 that the bandwidth amount is equal to or less than the predetermined value, the process proceeds to step S102.

[0104] In step S102, the transition calculation unit 412 calculates the transition path and transition speed of the viewpoint moving between the viewpoints before and after switching, based on the metadata, bandwidth information, and terminal position information. For example, as shown in FIG. 6 above, a viewpoint transition TR31 or a viewpoint transition TR32 is calculated based on the bandwidth information of the network 50.

[0105] In step S103, the virtual image generation unit 413 generates virtual image data based on the 3D model data, metadata, and trajectory information. For example, as shown in Fig. 6 above, a virtual image VI is generated according to the viewpoint transition TR31 or viewpoint transition TR32, which transitions between the viewpoints before and after the switching.

[0106] The metadata used when generating this virtual image data may include both meta information related to the viewpoint image after the switch and meta information related to the viewpoint image before the switch, but it is sufficient if it includes at least meta information related to the viewpoint image after the switch.

[0107] In step S104, the switching unit 415 supplies the virtual image data generated by the virtual image generating unit 413 to the playback unit 416, and controls the display unit 403 so that a virtual image corresponding to the virtual image data is displayed.

[0108] In step S105, the switching unit 415 determines whether or not preparation for switching is complete. For example, whether or not preparation for switching is complete can be determined by monitoring the processing status of viewpoint image data in the viewpoint image processing unit 411.

[0109] If it is determined in step S105 that preparation for switching is not complete, the process returns to step S102, and the subsequent steps are repeated. That is, by repeating steps S102 to S105, the playback device 40 continues to display the virtual image.

[0110] On the other hand, if it is determined in the determination process of step S105 that preparation for switching is complete, the process proceeds to step S106. In step S106, the switching unit 415 switches the image data supplied to the playback unit 416 from the virtual image data from the virtual image generation unit 413 to the viewpoint image data from the viewpoint image processing unit 411.

[0111] As a result, the playback device 40 displays a viewpoint image according to the viewpoint image data, and switches from the virtual image to the viewpoint image after the switch.

[0112] If the bandwidth amount exceeds the predetermined value in the determination process of step S101, it means that a sufficient bandwidth amount has been secured, so steps S102 to S105 are skipped and the process proceeds to step S106. In this case, the playback device 40 immediately switches from the pre-switching viewpoint image to the post-switching viewpoint image without displaying a virtual image. At this time, information about the transition of viewpoints shown in FIG. 7 above (for example, information superimposed on a map of the event venue) may be presented.

[0113] When the processing in step S106 is completed, the viewpoint switching processing ends.

[0114] Next, a second example of the viewpoint switching process corresponding to step S16 in Fig. 9 will be described with reference to the flowchart in Fig. 11. However, in this second example, metadata is not always acquired as in the first example, but is acquired when the user performs a viewpoint switching operation.

[0115] In step S201, similar to step S101 in FIG. 10, it is determined whether the bandwidth of the network 50 is equal to or less than a predetermined value, and if the bandwidth is equal to or less than the predetermined value (“Yes” in S201), the process proceeds to step S202.

[0116] In step S202, the communication unit 401 requests the metadata from the distribution server 30 via the network 50, thereby receiving the metadata transmitted from the distribution server 30. As a result, the transition calculation unit 412 acquires the metadata from the communication unit 401.

[0117] In step S203, the transition calculation unit 412 calculates the transition path and transition speed of the viewpoint that transitions between the viewpoints before and after the switch, and the metadata used in the calculation is the metadata acquired in the processing of the immediately preceding step S202.

[0118] In steps S204 to S207, similarly to steps S103 to S106 in FIG. 10, the generated virtual image is displayed until preparation for switching is completed, and when preparation for switching is completed, the virtual image is switched to the post-switch viewpoint image.

[0119] The above has described the flow of the viewpoint switching process. In this viewpoint switching process, a virtual image is generated based on the metadata and 3D model data in accordance with the bandwidth information of the network 50, and the generated virtual image is controlled to be displayed before the viewpoint image after the switch is displayed.

[0120] By performing this processing, a virtual image according to the bandwidth information of network 50 is displayed from the time the user switches the viewpoint until the post-switching viewpoint image can be displayed, thereby enabling more accurate control to display the virtual image at the timing when it should be displayed.

[0121] That is, although the time required to switch the viewpoint varies depending on the environment in which the user uses the playback device 40 because the bandwidth of the network 50 differs, by generating a virtual image according to the bandwidth information, it is possible to control the display of the virtual image more accurately. As a result, when the user performs an operation to switch the viewpoint, the viewpoint can be switched more appropriately.

[0122] Furthermore, by viewing the virtual image, the user can grasp where the viewpoint will be after the viewpoint change. Furthermore, the user can view the virtual image without being aware of the time it takes for the viewpoint change to be completed.

[0123] <2. Modifications>

[0124] In the above description, a head-mounted display has been described as an example of the playback device 40, but it may also be a mobile device such as a smartphone or tablet terminal, or a wearable device. Note that the playback device 40 may also be considered an image processing device since it has the image processing unit 402. Furthermore, the VR content played back by the playback device 40 is an example of content, and other content may also be used.

[0125] In the above explanation, for convenience of explanation, the metadata server 20 is installed at the event venue, but it may be installed in another location. For example, the metadata server 20 may be installed on the network 50, and the sensor data from the sensor 21 may be received by the metadata server 20 via the network 50.

[0126] Furthermore, although the sensor 21 has been described using the physical sensor 21-1 and the distance measurement sensor 21-2 as an example, other sensors may be used as long as they are capable of sensing spatial information and time information within the event venue. While the above description has shown an example of distributing VR content captured at an event venue such as a music concert venue, the environment is not limited to the event venue, and may be other environments such as the interior of a building or a city center, as long as multiple cameras 10 can be installed. Furthermore, the event venue is not limited to a music concert venue, and may be, for example, a venue where a sporting event or other competition is held.

[0127] <3. Computer Configuration>

[0128] The series of processes described above (the viewpoint switching processes shown in FIGS. 10 and 11) can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed in the computers of the devices.

[0129] FIG. 12 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0130] In the computer, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0131] The input unit 1006 includes a microphone, keyboard, mouse, etc. The output unit 1007 includes a speaker, display, etc. The storage unit 1008 includes a hard disk, non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable recording medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0132] In a computer configured as described above, the CPU 1001 loads a program recorded in the ROM 1002 or memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes it, thereby performing the above-mentioned series of processes.

[0133] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable recording medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0134] In a computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable recording medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0135] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed chronologically in the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or object-based processing). Furthermore, the program may be processed by one computer (processor), or may be processed in a distributed manner by multiple computers.

[0136] 10 and 11 can be executed by one device or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0137] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0138] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0139] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0140] The present technology can be configured as follows.

[0141] (1) a virtual image generating unit that generates a virtual image based on viewpoint image related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image; a display control unit that controls the generated virtual image to be displayed before the first viewpoint image is displayed; An image processing device comprising: (2) The viewpoint image related information includes information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image. The image processing device according to (1) above. (3) The virtual image generation unit generates the virtual image when the bandwidth amount indicated by the bandwidth information is equal to or less than a predetermined value. The image processing device according to (1) or (2). (4) The predetermined value is a fixed value that is set in advance, or a variable value that depends on the processing capacity of the device that displays the virtual image. The image processing device according to (3) above. (5) The virtual image generation unit generates the virtual image based on viewpoint image related information related to the second viewpoint image that was displayed before switching to the first viewpoint image. The image processing device according to any one of (1) to (4). (6) The virtual image generation unit generates the virtual image according to trajectory information determined by the first viewpoint position and the second viewpoint position, based on first viewpoint position information related to a first viewpoint position corresponding to the first viewpoint image and second viewpoint position information related to a second viewpoint position corresponding to the second viewpoint image. The image processing device according to (5) above. (7) The virtual image generation unit determines the trajectory information based on the band information. The image processing device according to (6) above. (8) The bandwidth amount indicated by the bandwidth information and the length of the trajectory distance indicated by the trajectory information have a negative correlation. The image processing device according to (7) above. (9) The display control unit adjusts a speed at which a viewpoint position corresponding to the virtual image moves on a trajectory indicated by the trajectory information, based on the bandwidth information. The image processing device according to (7) or (8). (10) When the bandwidth amount indicated by the bandwidth information exceeds a predetermined value, the display control unit does not display the virtual image and displays a map according to the first viewpoint position information and the second viewpoint position information. The image processing device according to (6) above. (11) When preparation for displaying the first viewpoint image is completed, the display control unit switches from the virtual image to the first viewpoint image. The image processing device according to any one of (1) to (10). (12) the viewpoint image related information includes generation information of the subject, The model data is an image of a specific character associated with the subject. The image processing device according to any one of (1) to (11). (13) The virtual image generation unit generates the two-dimensional or three-dimensional virtual image according to the processing capacity of a device that displays the virtual image. The image processing device according to (12) above. (14) The viewpoint image related information is acquired at predetermined intervals or in response to an instruction to switch viewpoints by a user. The image processing device according to any one of (1) to (13). (15) The acquisition timing differs for each piece of information included in the viewpoint image related information. The image processing device according to (14) above. (16) the information about the viewpoint position of the viewpoint image includes information about the position, attitude, and angle of view of a camera that captures the viewpoint image; The information about the subject in the viewpoint image includes information about the position and posture of the subject. The image processing device according to (2) above. (17) The viewpoint image related information further includes information about a space in which the subject exists and information about objects existing in the space. The image processing device according to (16) above. (18) The viewpoint image related information is transmitted in a band that ensures higher reliability than the viewpoint image. The image processing device according to any one of (1) to (17). (19) The image processing device generating a virtual image based on viewpoint image-related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image; Control is performed so that the generated virtual image is displayed before the first viewpoint image is displayed. Image processing methods. (20) Computer, a virtual image generating unit that generates a virtual image based on viewpoint image related information related to a first viewpoint image and model data corresponding to a subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image; a display control unit that controls the generated virtual image to be displayed before the first viewpoint image is displayed; A program that makes it work. (twenty one) a generating unit that generates viewpoint image related information related to a first viewpoint image based on a camera that captures viewpoint images and sensor data according to a detection result related to a subject of the viewpoint images; The first viewpoint image is displayed after displaying the viewpoint image related information and a virtual image based on model data corresponding to the subject of the viewpoint image in accordance with bandwidth information related to transmission of the viewpoint image. Image processing device. (twenty two) The viewpoint image related information includes information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image. The image processing device according to (21) above. (twenty three) the information about the viewpoint position of the viewpoint image includes information about the position, attitude, and angle of view of a camera that captures the viewpoint image; The information about the subject in the viewpoint image includes information about the position and posture of the subject. The image processing device according to (22) above. [Explanation of symbols]

[0142] 1 Content distribution system, 10, 10-1 to 10-N Camera, 20 Metadata server, 21 Sensor, 21-1 Physical sensor, 21-2 Distance measurement sensor, 30 Distribution server, 40 Playback device, 50, 50-1, 50-2 Network, 100 Camera system, 101 3D model generation unit, 102 Communication unit, 201 Analysis unit, 202 Metadata generation unit, 203 Communication unit, 401 Communication unit, 402 Image processing unit, 403 Display unit, 404 Input unit, 405 Detection unit, 410 Display control unit, 411 Viewpoint image processing unit, 412 Transition calculation unit, 413 Virtual image generation unit, 414 Memory unit, 416 Playback unit, 1001 CPU

Claims

1. a virtual image generation unit that generates a virtual image based on viewpoint image-related information related to a first viewpoint image and a second viewpoint image that was displayed before switching to the first viewpoint image, and model data corresponding to a subject of the viewpoint image, in accordance with bandwidth information related to transmission of the viewpoint images; a display control unit that controls the generated virtual image to be displayed before the first viewpoint image is displayed; Equipped with The virtual image generation unit determining trajectory information defined by a first viewpoint position corresponding to the first viewpoint image and a second viewpoint position corresponding to the second viewpoint image based on the bandwidth information; generating the virtual image according to the trajectory information based on first viewpoint position information relating to the first viewpoint position and second viewpoint position information relating to the second viewpoint position; Image processing device.

2. The viewpoint image related information includes information about the viewpoint position of the viewpoint image and information about the subject in the viewpoint image. The image processing device according to claim 1 .

3. The virtual image generation unit generates the virtual image when the bandwidth amount indicated by the bandwidth information is equal to or less than a predetermined value. The image processing device according to claim 1 .

4. The predetermined value is a fixed value that is set in advance, or a variable value that depends on the processing capacity of the device that displays the virtual image. The image processing device according to claim 3 .

5. The bandwidth amount indicated by the bandwidth information and the length of the trajectory distance indicated by the trajectory information have a negative correlation. The image processing device according to claim 1 .

6. The display control unit adjusts a speed at which a viewpoint position corresponding to the virtual image moves on a trajectory indicated by the trajectory information, based on the bandwidth information. The image processing device according to claim 1 .

7. When the bandwidth amount indicated by the bandwidth information exceeds a predetermined value, the display control unit hides the virtual image and displays a map according to the first viewpoint position information and the second viewpoint position information. The image processing device according to claim 1 .

8. When preparation for displaying the first viewpoint image is completed, the display control unit switches from the virtual image to the first viewpoint image. The image processing device according to claim 1 .

9. the viewpoint image related information includes generation information of the subject, The model data is an image of a specific character associated with the subject. The image processing device according to claim 1 .

10. The virtual image generation unit generates the two-dimensional or three-dimensional virtual image according to the processing capacity of a device that displays the virtual image. The image processing device according to claim 9 .

11. The viewpoint image related information is acquired at predetermined intervals or in response to an instruction to switch viewpoints by a user. The image processing device according to claim 1 .

12. The acquisition timing differs for each piece of information included in the viewpoint image related information. The image processing device according to claim 11 .

13. the information about the viewpoint position of the viewpoint image includes information about the position, attitude, and angle of view of a camera that captures the viewpoint image; The information about the subject in the viewpoint image includes information about the position and posture of the subject. The image processing device according to claim 2 .

14. The viewpoint image related information further includes information about a space in which the subject exists and information about objects existing in the space. The image processing device according to claim 13.

15. The viewpoint image related information is transmitted in a band that ensures higher reliability than the viewpoint image. The image processing device according to claim 11 .

16. The image processing device generating a virtual image based on viewpoint image-related information relating to a first viewpoint image and a second viewpoint image that was displayed before switching to the first viewpoint image, and model data corresponding to a subject of the viewpoint image, in accordance with bandwidth information relating to transmission of the viewpoint images; controlling the generated virtual image to be displayed before displaying the first viewpoint image; Including, determining trajectory information defined by a first viewpoint position corresponding to the first viewpoint image and a second viewpoint position corresponding to the second viewpoint image based on the band information; generating the virtual image according to the trajectory information based on first viewpoint position information related to the first viewpoint position and second viewpoint position information related to the second viewpoint position; The image processing method further comprises:

17. Computer, a virtual image generation unit that generates a virtual image based on viewpoint image-related information related to a first viewpoint image and a second viewpoint image that was displayed before switching to the first viewpoint image, and model data corresponding to a subject of the viewpoint image, in accordance with bandwidth information related to transmission of the viewpoint images; a display control unit that controls the generated virtual image to be displayed before the first viewpoint image is displayed; Equipped with The virtual image generation unit determining trajectory information defined by a first viewpoint position corresponding to the first viewpoint image and a second viewpoint position corresponding to the second viewpoint image based on the bandwidth information; generating the virtual image according to the trajectory information based on first viewpoint position information relating to the first viewpoint position and second viewpoint position information relating to the second viewpoint position; A program that functions as an image processing device.

Images