Information processing device, information processing method, and display device

Abstract

An information processing device according to an embodiment of the present technology is provided with an image capture control unit. In accordance with the positional relationship between a target object which operates in a virtual space, and a virtual camera which moves through the virtual space and captures images of the target object, the image capture control unit executes collision avoidance processing to avoid a collision of the virtual camera with the target object.

Description

Technical Field

[0001] This technology relates to an information processing apparatus, an information processing method, and a display apparatus applicable to image display in a virtual space.

Background Art

[0002] Patent Document 1 describes a content distribution server that distributes live content using virtual characters. In this server, a virtual character that operates in accordance with the movements of a distributor is placed in a virtual space, and an image within the virtual space is captured from the viewpoint of a virtual camera. The shooting position of the virtual camera and the like are controlled according to the instructions of viewers participating as cameramen. The image captured by the virtual camera is distributed to viewer terminals as live content (paragraphs

[0018]

[0038]

[0045] of the specification of Patent Document 1, FIG. 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Technologies for distributing live content and the like using a virtual space are expected to be applied in various fields such as entertainment and education, and technologies for appropriately capturing an object in a virtual space are required.

[0005] In view of the above circumstances, an object of this technology is to provide an information processing apparatus, an information processing method, and a display apparatus capable of appropriately capturing an object in a virtual space.

Means for Solving the Problems

[0006] To achieve the above object, an information processing apparatus according to one aspect of this technology includes a shooting control unit. The aforementioned shooting control unit executes collision avoidance processing to avoid collisions between the virtual camera and the target object, depending on the positional relationship between the target object operating in the virtual space and the virtual camera moving in the virtual space to photograph the target object.

[0007] This information processing device performs collision avoidance processing based on the positional relationship between the target object in the virtual space and the virtual camera that moves through the virtual space and photographs the target object. As a result, collisions between the target object and the virtual camera are avoided in advance, making it possible to properly photograph the target in the virtual space.

[0008] The shooting control unit may detect the proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and if the proximity state is detected, it may execute the collision avoidance process.

[0009] The shooting control unit may detect a state in which the relative distance between the target object and the virtual camera is below a predetermined threshold as the proximity state.

[0010] The aforementioned relative distance may include the current value or the predicted value.

[0011] The virtual camera may photograph the target object along a pre-set shooting path. In this case, the shooting control unit may change the shooting path as a collision avoidance process when the proximity state is detected.

[0012] The aforementioned shooting path may be a path configured such that the virtual camera passes through a plurality of relay points in sequence.

[0013] The shooting control unit may change the relay point that serves as the virtual camera's movement target when the proximity state is detected, so as to avoid a collision between the target object and the virtual camera.

[0014] The shooting control unit may set the virtual camera's target to the nearest intermediate point in sequence that allows for collision avoidance between the target object and the virtual camera.

[0015] When the proximity state is detected, the shooting control unit may modify at least a portion of the path to the relay point that serves as the virtual camera's target so as to avoid a collision between the target object and the virtual camera.

[0016] The shooting control unit may move the virtual camera along a detour path that bypasses the target object, starting from the point where the proximity state is detected.

[0017] The aforementioned detour route may be a route that maintains a constant relative distance between the target object and the virtual camera.

[0018] The aforementioned shooting path may be a path in which the passage time of the virtual camera is set for each of the plurality of relay points. In this case, the shooting control unit may adjust the movement speed of the virtual camera moving along the modified shooting path based on the passage time set for the relay points.

[0019] The plurality of relay points may include at least one important relay point. In this case, the shooting control unit may adjust the movement speed of the virtual camera to match the time of passage of the important relay point included in the modified shooting path.

[0020] When the proximity state is detected, the shooting control unit may, as part of the collision avoidance process, switch the display image for displaying the target object from an image captured by the virtual camera to an image captured by another virtual camera.

[0021] The shooting control unit may, as part of the collision avoidance process, move the virtual camera so that the relative distance between the target object and the virtual camera remains constant.

[0022] The imaging control unit may delay the timing of the speed change of the virtual camera with respect to the timing of the speed change of the target object so that the relative distance falls within a predetermined range.

[0023] The target object may include a three-dimensional real model of a performer.

[0024] The imaging control unit may distribute the image captured by the virtual camera in real time.

[0025] An information processing method according to an aspect of the present technology is an information processing method executed by a computer system, including executing collision avoidance processing for avoiding a collision of the virtual camera with respect to the target object according to the positional relationship between the target object operating in the virtual space and the virtual camera that moves in the virtual space to capture the target object.

[0026] A display device according to an aspect of the present technology includes an image acquisition unit and a display unit. The image acquisition unit acquires an image captured by the virtual camera that operates according to collision avoidance processing for avoiding a collision of the virtual camera with respect to the target object, which is executed according to the positional relationship between the target object operating in the virtual space and the virtual camera that moves in the virtual space to capture the target object. The display unit displays the image captured by the virtual camera.

Brief Description of the Drawings

[0027] [Figure 1] It is a schematic diagram showing a configuration example of a distribution system according to an embodiment of the present technology. [Figure 2] It is a schematic diagram showing an example of a virtual space configured by a distribution system. [Figure 3] It is a schematic diagram for explaining imaging by a virtual camera. [Figure 4] It is a schematic diagram showing an example of a virtual camera video. [Figure 5]This block diagram shows an example of the functional configuration of a shooting system and a distribution server. [Figure 6] This is a schematic diagram showing an example of the configuration of the imaging unit. [Figure 7] This is a schematic diagram showing an example of a photography studio. [Figure 8] This block diagram shows an example of a client terminal's functional configuration. [Figure 9] This is a schematic diagram illustrating an example of a proximity state. [Figure 10] This is a schematic diagram illustrating an example of collision avoidance processing that alters the shooting path. [Figure 11] Figure 10 shows a flowchart of the collision avoidance process. [Figure 12] This is a schematic diagram illustrating another example of collision avoidance processing that alters the shooting path. [Figure 13] Figure 12 shows the flowchart for collision avoidance processing. [Figure 14] This is a schematic diagram illustrating another example of collision avoidance processing for a virtual camera. [Figure 15] Figure 14 shows the flowchart for collision avoidance processing. [Figure 16] This is a schematic diagram illustrating an example of a collision between the subject being filmed and the virtual camera. [Modes for carrying out the invention]

[0028] The embodiments of this technology will be described below with reference to the drawings.

[0029] [Overview of the distribution system] Figure 1 is a schematic diagram showing an example configuration of a distribution system according to one embodiment of this technology. The distribution system 100 is a system that distributes live content using a virtual space in real time. The distribution system 100 includes a distribution server 10, a shooting system 11, and at least one client terminal 12. In the configuration shown in Figure 1, the distribution server 10 and each client terminal 12 are connected to each other so that they can communicate via the Internet 13. Alternatively, the distribution server 10 and each client terminal 12 may be connected via a dedicated network line such as a private network. The following describes the overview of the distribution system 100, using the example of a music live performance or similar event taking place in a virtual space.

[0030] The distribution server 10 generates live content using a virtual space and distributes the generated live content to each client terminal 12 via the internet 13. In other words, the distribution server 10 provides a live content distribution service. The virtual space contains computer graphics (CG) objects such as a virtual model operated by performer 1, who is the lead in the live content, as well as stage sets and backgrounds. Performer 1 can perform various acts in the virtual space, such as singing, dancing, and playing musical instruments, by controlling their own virtual model. The distribution server 10 configures such a virtual space and generates live content, including images of the virtual space, audio from the virtual space, and data related to virtual objects placed in the virtual space (such as a virtual model of performer 1 or other objects). In this disclosure, "image" includes both still images and moving images. In the following, moving images (video) of virtual spaces will be used primarily as live content.

[0031] The shooting system 11 is a system that shoots and senses performer 1 and generates the data necessary to create a virtual model of performer 1. In the distribution system 100, the distribution server 10 generates a virtual model of performer 1 based on the data generated by the shooting system 11 (see Figure 5). The virtual model of performer 1 is an example of a target object that will be captured by the virtual camera described later. In this embodiment, a 3D live-action model (volumetric model) of performer 1 is used as the virtual model (target object) of performer 1. The volumetric model is a virtual model that reproduces performer 1 in real space as 3D computer graphics. By using a volumetric model, it is possible to reproduce performer 1's performance in real space directly in the virtual space. Alternatively, the virtual model of performer 1 may be generated as a stereo image captured from different left and right viewpoints. This makes it possible to view performer 1 in real space in 3D.

[0032] The client terminal 12 is a terminal device used by viewer 2 who uses the live content distribution service. The client terminal 12 plays video, audio, etc., from the virtual space based on the live content distributed from the distribution server 10. For example, a device equipped with an HMD 14 (Head Mounted Display) can be used as the client terminal 12. By using the HMD 14, it becomes possible to display images of the virtual space so as to cover the viewer's field of vision 2. In addition, the client terminal 12 may also be a device equipped with wearable AR glasses (transparent HMD), a stationary display, or a portable terminal device such as a tablet or smartphone.

[0033] The client terminal 12 is also equipped with a motion sensor to detect the movements of viewer 2, a microphone to detect viewer 2's voice, and input devices such as a keyboard to accept text input from viewer 2. Data entered via the input devices is transmitted to the distribution server 10. The distribution server 10 generates live content that reflects the actions, voice, and text input of viewer 2 based on the data transmitted from each client terminal 12. This allows viewers to participate in the live performance by performer 1 held in a virtual space.

[0034] Figure 2 is a schematic diagram showing an example of a virtual space composed of the distribution system 100. The virtual space 3 can be described as a shared space shared by multiple viewers 2 during their virtual experience. In virtual space 3, performer model 4, which is a virtual model of performer 1, is placed. As described above, performer model 4 is a volumetric model that reproduces performer 1 in real space and operates in the same way as performer 1's movements. In the example shown in Figure 2, performer model 4 placed on the stage in virtual space 3 is schematically illustrated.

[0035] Furthermore, a viewer model 5, which is a virtual model of viewer 2, is placed in the virtual space 3. The viewer model 5 is a virtual avatar that operates in response to the movements of viewer 2 detected, for example, by a motion sensor on the client terminal 12. In the example shown in Figure 2, multiple viewer models 5 are schematically illustrated, arranged to surround the stage.

[0036] Furthermore, in virtual space 3, the voices and comments of performer 1 and viewer 2 are shared. Figure 2 schematically illustrates icons 6 representing voices and comments. For example, icons 6 containing comments entered by performer 1 or viewer 2 are placed corresponding to the virtual models of those individuals so that the person who entered the comment can be identified. Alternatively, when performer 1 or viewer 2 speaks, icons 6 may be placed to indicate who is speaking.

[0037] For example, on the client terminal 12 of viewer 2, who is participating in a live event in virtual space 3 using viewer model 5, the video corresponding to viewer model 5's field of view in virtual space 3 (field of view video) is displayed. This makes it possible to provide viewer 2 with a realistic virtual experience, as if they were actually at the live venue in virtual space 3.

[0038] Furthermore, in the distribution system 100, the distribution server 10 generates video footage of the virtual space 3 from the viewpoint of a virtual camera, which is then distributed as live content. In the following, the video captured by the virtual camera (video captured from the virtual camera's viewpoint) will be referred to as the virtual camera video. The virtual camera video is different from the field of view video described above.

[0039] [Virtual camera footage] Figure 3 is a schematic diagram illustrating the process of capturing images using a virtual camera. Figure 4 is a schematic diagram showing an example of a virtual camera image. The virtual camera 20 is a camera virtually configured to move within the virtual space 3 and photograph the virtual space 3. The shooting position and direction of the virtual camera 20 can be freely set within the virtual space 3. Figure 3 schematically illustrates a virtual camera 20 that photographs the virtual space 3, and the shooting path 21 of the virtual camera 20 (dotted line in the figure). Note that when actually constructing the virtual space 3, the objects representing the virtual camera 20 and the shooting path 21 are not displayed.

[0040] The target of the virtual camera 20 is the performer model 4 operating in the virtual space 3. In other words, the virtual camera 20 is a camera that moves around the virtual space 3 to photograph the performer model 4. In the example shown in Figure 3, the virtual camera 20 moves along a shooting path 21 that circles around from the left rear of the performer model 4 to the front. At this time, the shooting direction and magnification of the virtual camera 20 are appropriately controlled so that the performer model 4 is within the shooting range of the virtual camera 20. For example, as shown in Figure 3, when the virtual camera 20 passes in front of the performer model 4, a virtual camera image 22 of the performer model 4 taken from the front is generated, as shown in Figure 4.

[0041] Thus, the virtual camera footage 22 is footage of the performer model 4, captured by changing the shooting position and direction in the virtual space 3 over time. When shooting with the virtual camera 20, there are fewer constraints on shooting position and direction. Therefore, it becomes easy to implement camera work that would be difficult to achieve in real space, for example. As a result, dynamic visual expression can be easily realized. In this embodiment, images of the performer model 4 captured by the virtual camera 20 (virtual camera footage 22) are distributed as live content. This allows each viewer 2 to watch music concerts and other events taking place in the virtual space 3 in real time.

[0042] In such live streaming, when the virtual camera 20 is brought close to the performer model 4 for close-up shooting, if the camera path (shooting route 21) of the virtual camera 20 is a fixed route, there is a possibility of collision between the performer model 4 and the virtual camera 20. For example, if the performer model 4 (performer 1) moves more than planned, a collision between the performer model 4 and the virtual camera 20 is possible. In this case, there is a risk that "penetration" or "penetration" of the performer model 4 will be live-streamed as the virtual camera footage 22.

[0043] In this embodiment, the positional relationship between the performer model 4 and the virtual camera 20 is monitored. Then, depending on the positional relationship between the performer model 4 and the virtual camera 20, collision avoidance processing is performed to prevent the virtual camera 20 from colliding with the performer model 4. Collision avoidance processing typically involves controlling the behavior of the virtual camera 20 to cause it to perform actions (collision avoidance actions) that avoid collisions with the performer model 4. This prevents situations such as the virtual camera 20 colliding with the performer model 4, making it possible to properly film the performer model 4 without generating "penetration footage" or "penetration footage." The collision avoidance process will be explained in detail later.

[0044] Figure 5 is a block diagram showing an example of the functional configuration of the shooting system 11 and the distribution server 10. The shooting system 11 and the distribution server 10 are distribution-side systems used by the distributor to distribute live content in the distribution system 100. For example, performer 1 who appears in the live performance, or a business operator that plans the live content, etc., would be the distributor.

[0045] [Shooting system configuration] The imaging system 11 generates the data necessary to generate a volumetric model of performer 1. As shown in Figure 5, the imaging system 11 has an imaging unit 30 and an imaging processing unit 31. The shooting unit 30 includes a group of devices for shooting and sensing the performer 1, and is used, for example, in a shooting studio for shooting the performer 1. In this embodiment, the shooting unit 30 has a multi-view video shooting unit 32 and a body position sensing unit 33.

[0046] Figure 6 is a schematic diagram showing an example of the configuration of the imaging unit 30. The multi-view video shooting unit 32 is a multi-camera unit that includes multiple cameras. Figure 6 schematically illustrates the N cameras (Cam(1) to Cam(N)) that make up the multi-view video shooting unit 32. Each camera is positioned differently from the others to, for example, capture performer 1 from all directions. This makes it possible to generate a volumetric model that synthesizes the entire body of performer 1. Thus, the multi-view video shooting unit 32 can be described as a multi-view camera for volumetric photography.

[0047] The body position sensing unit 33 includes sensors (motion sensors) for detecting the position of various parts of the performer's body. In the example shown in Figure 6, the body position sensing unit 33 uses a wearable position sensor 33a, a depth camera 33b, and an infrared camera 33c. Furthermore, the type of sensor used as the body position sensing unit 33 is not limited, and other motion sensors may be used. These sensors may also be used individually, or all or some of them may be used in combination.

[0048] The wearable position sensor 33a is a sensor used by the performer 1, worn on the body, to detect the position of the part to which it is worn. For example, in the example shown in Figure 6, the wearable position sensors 33a are attached to the left and right wrists of the performer 1, respectively. In this case, the position of the performer 1's wrists is detected, making it possible to detect the movement of the performer 1's hands. The depth camera 33b is a camera that captures depth images of the subject (performer 1). A depth image is an image in which the distance (depth) to the subject is detected for each pixel. By using the depth image, it is possible to detect the position of each part of performer 1's body. For example, a ToF camera can be used as the depth camera 33b. The infrared camera 33c is a camera that captures infrared images by irradiating a target with infrared light. When using the infrared camera 33c, for example, an infrared marker 34 that selectively reflects infrared light is attached to the body of performer 1, and an infrared image of performer 1 is captured. From the state of the infrared marker 34 in the infrared image thus captured, it is possible to detect the position of the attachment site.

[0049] Figure 7 is a schematic diagram showing an example of a filming studio. The filming studio is, for example, a green screen studio that uses green materials for the background and floor. In the filming studio, poles are installed to fix each camera of the multi-view video shooting unit 32, for example. In the example shown in Figure 7, Cam(1) and Cam(2) are fixed to the upper and lower sections of the left pole in the figure, and Cam(3) and Cam(4) are fixed to the upper and lower sections of the right pole. An infrared camera 33c is also placed at the top of each pole. Furthermore, each pole is equipped with a light 35 for illumination. Multiple such poles are arranged, for example, to surround the performer 1. This makes it possible to film the performer 1 from all directions. Additionally, a depth camera 33b is positioned in the shooting studio. The distance between the depth camera 33b and performer 1 is set to a predetermined value (for example, approximately 0.5m to 5.5m) so that depth images of the required range can be captured. The specific configuration of the shooting studio is not limited. For example, any shooting equipment capable of performing the necessary shooting and sensing to generate performer model 4 may be used as appropriate.

[0050] Returning to Figure 5, the imaging processing unit 31 is a data processing unit that integrates the output of the imaging unit 30 and generates the data necessary to generate a volumetric model. For example, a computer such as a PC (Personal Computer) can be used as the imaging processing unit 31. Alternatively, the imaging processing unit 31 may be configured by the distribution server 10, which will be described later. The shooting processing unit 31 includes, as functional blocks, a camera image generation unit 36, a multi-view video streaming processing unit 37, a body position information generation unit 38, and a 3D model position information generation unit 39.

[0051] The camera image generation unit 36 ​​reads the output of the multi-view video shooting unit 32 and generates camera images 26 captured by each camera. The camera images 26 are images of performer 1 taken from multiple viewpoints at the same time. Figure 5 schematically illustrates the camera images 26 captured by each camera (Cam(1) to Cam(N)). The multi-view video streaming processing unit 37 generates multi-view video 27 of performer 1 based on multiple camera images 26 and performs streaming processing on the multi-view video 27. For example, the multi-view video 27 is generated by synchronizing the shooting timing of the camera images 26 and arranging them along the time axis. Streaming processing such as compression and conversion is performed on this multi-view video 27. Figure 5 schematically illustrates an example of multi-view video 27 using a series of images along the time axis.

[0052] The body position information generation unit 38 reads the output of the body position sensing unit 33 and generates body position information representing the positions of various parts of the performer's body. For example, the body position information generated includes the position information of the part to which the wearable position sensor 33a is attached, the depth image 28 of the performer 1 taken by the depth camera 33b, and the infrared image of the performer 1 (infrared marker 34) taken by the infrared camera 33c. Figure 5 schematically illustrates the depth image 28 of the performer 1 as an example of body position information. The 3D model position information generation unit 39 generates model position information representing the position of each part of the performer model 4 (in this case, a volumetric model of performer 1) based on the body position information described above. Specifically, bone estimation processing is performed based on the body position information, and bone data 29 is calculated, which estimates the position and posture of the performer 1's skeleton. This bone data 29 is used as the model position information of the performer model 4. An example of bone data 29 is schematically illustrated in Figure 5.

[0053] Furthermore, methods other than the bone estimation described above may be used to sense the body position of performer 1. For example, position estimation of various body parts using image recognition or 3D position estimation using machine learning may be used. Alternatively, motion capture technology using infrared detection with an infrared camera and infrared markers may be used.

[0054] [Distribution Server Configuration] The distribution server 10 includes a network transmission unit 40, a storage unit 41, and a server control unit 42. The network transmission unit 40 is a communication module that performs network communication with other devices via the internet 13. The network transmission unit 40 has a data transmission function that transmits data (such as live content) generated by the distribution server 10, and a data reception function that receives data transmitted from the client terminal 12 via the internet 13. The specific configuration of the network transmission unit 40 is not limited, and various communication modules compatible with wired LAN, wireless LAN, optical communication, etc., may be used.

[0055] The storage unit 41 is a non-volatile storage device. For example, the storage unit 41 may be a recording medium using solid elements such as an SSD (Solid State Drive) or a magnetic recording medium such as an HDD (Hard Disk Drive). Furthermore, the type of recording medium used as the storage unit 41 is not limited; for example, any recording medium that records data non-temporarily may be used. The memory unit 41 stores the control program according to this embodiment. The control program is, for example, a program that controls the operation of the entire distribution server 10. The information stored in the memory unit 41 is not limited to this.

[0056] The server control unit 42 controls the operation of the distribution server 10. The server control unit 42 has the necessary hardware configuration for a computer, such as a CPU and memory (RAM, ROM). Various processes are executed by the CPU loading the control program stored in the memory unit 41 into the RAM and executing it. The server control unit 42 corresponds to the information processing device according to this embodiment.

[0057] For the server control unit 42, a device such as an FPGA (Field Programmable Gate Array) or other PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit) may be used. Alternatively, a processor such as a GPU (Graphics Processing Unit) may be used as the server control unit 42.

[0058] In this embodiment, the CPU of the server control unit 42 executes the program (control program) according to this embodiment, thereby realizing the content data generation unit 43 and the virtual camera control unit 44 as functional blocks. These functional blocks then execute the information processing method according to this embodiment. Dedicated hardware such as ICs (integrated circuits) may be used as appropriate to realize each functional block.

[0059] The content data generation unit 43 generates content data here. Content data is, for example, data necessary to construct the virtual space 3. The content data includes data related to virtual objects placed in the virtual space 3 (performer model 4, viewer model 5, stage equipment, etc.), audio data in the virtual space 3, and comments from performers and viewers. As shown in Figure 5, the content data generation unit 43 includes a performer model generation unit 45 and a viewer model generation unit 46.

[0060] The performer model generation unit 45 generates a performer model 4, which will be a virtual model of performer 1. Specifically, it generates data for a volumetric model (performer model 4) of performer 1 based on the multi-view video 27 and bone data 29 of performer 1 output from the shooting system 11. For example, shape data and texture data for performer model 4 are generated from multi-view video 27. Also, for example, motion data for performer model 4 is generated from bone data 29. Furthermore, the performer model generation unit 45 calculates the placement position of the performer model 4 in the virtual space 3 according to the actions of performer 1 and the staging of the live performance.

[0061] The viewer model generation unit 46 generates a viewer model 5, which will be a virtual model of viewer 2. Specifically, it obtains data representing the positions of various parts of viewer 2's body (head, hands, etc.) from each client terminal 12, and generates data for viewer model 5 (virtual avatar, etc.) based on this data. The design of viewer model 5 may be specified by viewer 2, for example, or a default design may be used. Furthermore, the viewer model generation unit 46 calculates the placement position of each viewer model 5 in the virtual space 3 according to the actions of viewer 2, etc.

[0062] In addition, the content data generation unit 43 generates content data such as data for other virtual objects placed in the virtual space 3, audio data for performer 1 and viewer 2, and music performance data. The specific content of the content data is not limited. The content data generated by the content data generation unit 43 is distributed to each client terminal 12 as live content.

[0063] The virtual camera control unit 44 controls the virtual camera 20 to generate virtual camera images 22 (see Figure 4) of the virtual space 3 captured by the virtual camera 20. For example, performer models 4 and viewer models 5 generated by the content data generation unit 43 are placed on the virtual space 3, recreating a virtual live venue. The virtual camera control unit 44 controls the behavior of the virtual camera 20 in the virtual space 3 configured in this way.

[0064] In this embodiment, the virtual camera control unit 44 performs collision avoidance processing to avoid collisions between the performer model 4, which operates in the virtual space 3, and the virtual camera 20, which moves within the virtual space 3 to photograph the performer model 4, depending on the positional relationship between the performer model 4 and the virtual camera 20 that moves within the virtual space 3 to photograph the performer model 4. In this embodiment, the virtual camera control unit 44 corresponds to the photography control unit.

[0065] The positional relationship between the performer model 4 and the virtual camera 20 is typically expressed using the distance (relative distance) between the performer model 4 and the virtual camera 20. This distance may be a current value or a predicted value. For example, the distance between performer model 4 and virtual camera 20 is monitored, and collision avoidance processing is executed if that value meets a predetermined condition. Alternatively, as collision avoidance processing, a process is executed that sequentially controls the behavior of virtual camera 20 so that the distance between performer model 4 and virtual camera 20 meets a predetermined condition.

[0066] The virtual camera control unit 44 also generates virtual camera images 22. For example, images from a virtual camera 20 moving within the virtual space 3 during a live broadcast are generated at a predetermined frame rate. Based on these images, virtual camera images 22 capturing the virtual space 3 during the live broadcast are generated. The virtual camera video 22 is distributed to each client terminal 12 as live content. In this embodiment, the image captured by the virtual camera 20 (virtual camera video 22) is distributed in real time.

[0067] [Client terminal configuration] Figure 8 is a block diagram showing an example of the functional configuration of a client terminal 12. The client terminal 12 includes a network transmission unit 50, a storage unit 51, and a terminal control unit 52. The client terminal shown in Figure 8 also includes an HMD 14 and a hand controller 15. In this embodiment, the client terminal 12 functions as a display device. Here, viewer 2 using client terminal 12 will be referred to as the participant, and viewer 2 other than the participant will be referred to as other viewers.

[0068] The HMD14 is a display device worn on the head by the user. The HMD14 includes a display 53, an audio output unit 54, an audio input unit 55, and an HMD motion sensor 56. The display 53 is positioned to cover the user's field of vision. For example, a liquid crystal display or an organic EL display can be used as the display 53. The audio output unit 54 is an element that reproduces sound, such as a speaker or headphones. The audio input unit 55 is an element that detects sound, such as a microphone. The HMD motion sensor 56 is a sensor that detects the position and orientation of the HMD 14 body, and includes, for example, an accelerometer, a gyroscope, a compass, etc. In this embodiment, the display 53 corresponds to a display unit that displays images captured by the virtual camera.

[0069] The hand controller (HC) 15 is an operating device that accepts input operations in accordance with the user's hand movements. The hand controller 15 can be a gripping type device that the user holds and operates, or a wearable type device that is worn on the user's hand. The hand controller 15 has a vibration generating unit 57 and an HC motion sensor 58. The vibration generation unit 57 is a device that generates vibrations, and for example, a voice coil motor or an eccentric motor is used. The HC motion sensor 58 is a sensor that detects the position and orientation of the hand controller 15 body.

[0070] The network transmission unit 50 is a communication module that performs network communication with other devices via the internet 13. The network transmission unit 50 has, for example, a data receiving function that receives data generated by the distribution server 10 and a data transmission function that sends data generated by the client terminal 12. The memory unit 51 is a non-volatile memory device, and recording media such as SSDs or HDDs are used. The memory unit 51 stores a control program that controls the operation of the entire client terminal 12. In addition, the information stored in the memory unit 51 is not limited to this.

[0071] The terminal control unit 52 controls the operation of the client terminal 12. The terminal control unit 52 has the necessary hardware configuration for a computer, such as a CPU and memory (RAM, ROM). Various processes are executed by the CPU loading the control program stored in the memory unit 51 into the RAM and executing it. The specific configuration of the terminal control unit 52 is not limited. In this embodiment, various functional blocks are realized by the CPU of the terminal control unit 52 executing a control program stored in the memory unit 51.

[0072] As shown in Figure 8, the terminal control unit 52 has the following functional blocks: a video acquisition unit 60, a performer data analysis unit 61, a other viewer data analysis unit 62, timing synchronization units 63a to 63c, a participant data output unit 64, a 3D object control unit 65, a 3D object collision detection unit 66, and a display control unit 67. Furthermore, the area enclosed by the dotted line in Figure 8 schematically illustrates the data (network communication data) transmitted and received via the Internet 13.

[0073] The video acquisition unit 60 acquires the virtual camera video 22 captured by the virtual camera 20 via the network transmission unit 50 and outputs it to the display control unit 67. Figure 8 schematically illustrates the virtual camera video 22 output to the display control unit 67. As described above, in this embodiment, the distribution server 10 (virtual camera control unit 44) performs collision avoidance processing to avoid collisions between the virtual camera 20 and the performer model 4, depending on the positional relationship between the performer model 4 operating in the virtual space 3 and the virtual camera 20 moving around the virtual space 3 to photograph the performer model 4. Accordingly, the video acquisition unit 60 acquires images (virtual camera images 22) captured by the virtual camera 20 operating in accordance with this collision avoidance processing. In this embodiment, the video acquisition unit 60 corresponds to the image acquisition unit.

[0074] The performer data analysis unit 61 acquires data about performer 1 via the network transmission unit 50, analyzes the acquired data, and generates performer data (performer video data, performer audio stream, performer metadata). Performer video data is data representing the appearance (shape and texture) of performer model 4, for example. Performer audio stream is streaming data of performer 1's voice and music, for example. Performer metadata is data representing the body position of performer model 4 (bone data 29) and text data such as performer 1's comments. Body position data is data representing the position and posture of the head and both hands, for example, and data representing each part with 6DoF (Degree of Freedom) or 3DoF is used. Performer data generated by performer data analysis unit 61 is output to timing synchronization unit 63a.

[0075] The other viewer data analysis unit 62 acquires data about other viewers other than the participant via the network transmission unit 50, analyzes the acquired data to generate other viewer data (other viewer audio stream, other viewer metadata). The other viewer audio stream is streaming data of other viewers' audio. The other viewer metadata includes, for example, data (bone data, etc.) of the virtual avatar (viewer model 5) used by the other viewer, and text data such as comments from the other viewer. The other viewer data generated by the other viewer data analysis unit 62 is output to the timing synchronization unit 63b.

[0076] The timing synchronization unit 63a synchronizes the timing of the performer data (performer video data, performer audio stream, performer metadata) generated by the performer data analysis unit 61. Figure 8 schematically illustrates the performer data with synchronized timing. Of the synchronized performer data, the graphics-related data (performer video data, bone data 29, comments, etc.) is output to the 3D object control unit 65. The audio-related data (performer audio stream, etc.) is output to the audio output unit 54 of the HMD 14.

[0077] The timing synchronization unit 63b synchronizes the timing of the other viewer data (other viewer audio stream, other viewer metadata) generated by the other viewer data analysis unit 62. Figure 8 schematically illustrates the other viewer data with synchronized timing. Of the synchronized data from other viewers, graphic data (bone data, comments, etc.) is output to the 3D object control unit 65. Audio data (audio streams from other viewers, etc.) is output to the audio output unit 54 of the HMD 14.

[0078] The timing synchronization unit 63c generates participant data (participant audio stream, participant metadata) related to the participant using the client terminal 12 so that the timing is synchronized. The participant audio stream is streaming data of the participant's voice and is generated based on the output of the audio input unit 55 of the HMD 14. Participant metadata is, for example, data (bone data, etc.) of the virtual avatar (viewer model 5) used by the participant. The virtual avatar data is generated based on the output of the HMD motion sensor 56 of the HMD 14 and the HC motion sensor 58 of the hand controller 15. In addition, text data such as the participant's comments may be generated as participant metadata based on text input or speech recognition.

[0079] The participant data output unit 64 compresses and converts the participant data to generate participant data for transmission. This data is transmitted to the distribution server 10 via the network transmission unit 50. On the distribution server 10, for example, a viewer model 5 corresponding to each viewer 2 is generated based on the participant data transmitted from each client terminal 12.

[0080] The 3D object control unit 65 generates virtual objects (3D objects) such as the performer model 4 and the viewer model 5, and constructs a virtual space 3 (see Figure 2) in which the virtual objects are placed. For example, performer model 4 (a volumetric model of performer 1) is generated based on the output of the timing synchronization unit 63a. Viewer model 5 used by other viewers is generated based on the output of the timing synchronization unit 63b. Viewer model 5 used by the participant (the participant's virtual avatar) is generated based on the outputs of the HMD motion sensor 56 and the HC motion sensor 58. Icons representing comments from performer 1, other viewers, and the participant are placed in the virtual space 3. Data relating to the virtual space 3, configured by the 3D object control unit 65, is output to the 3D object collision detection unit 66 and the display control unit 67.

[0081] The 3D object collision detection unit 66 detects collisions (contacts) between virtual models in the virtual space 3. Specifically, it detects collisions between the viewer model 5 used by the user and other virtual objects (for example, the performer model 4 or other viewer models 5). Furthermore, when a collision between models is detected in the 3D object collision detection unit 66, a vibration signal corresponding to that collision is generated. The generated vibration signal is output to the vibration generation unit 57 of the hand controller 15.

[0082] The display control unit 67 controls the display of the virtual space 3 on the client terminal 12 (in this case, the display 53 of the HMD 14). Specifically, it generates the video output to the display 53 (hereinafter referred to as the output video) as appropriate. The display control unit 67 generates a field of view video representing the view of the viewer model 5 used by the user, based on the data of the virtual space 3. The display control unit 67 also receives virtual camera video 22 captured by the virtual camera 20 as input. In this embodiment, the output video is a combination of the field of view video and the virtual camera video 22. For example, the field of view video and the virtual camera video 22 are switched in response to a predetermined switching operation performed by the user.

[0083] Furthermore, the method for generating the output video is not limited. For example, if the client terminal 12 has multiple displays, the field of view video and the virtual camera video 22 may be output separately. Also, if the viewer model 5, etc., is not used, a configuration that outputs only the virtual camera video 22 may be used. Furthermore, the virtual camera video 22 is not limited to the case where the distribution server 10 generates it; for example, the virtual camera video 22 may be generated by the client terminal 12. In this case, for example, the virtual space 3 configured on the client terminal 12 is captured by the virtual camera 20. In this case, the collision avoidance process for the virtual camera 20, as described below, is executed on the client terminal 12.

[0084] [Operation of the virtual camera control unit] The operation of the virtual camera control unit 44 of the distribution server 10 will be explained in detail. The virtual camera control unit 44 monitors the positional relationship between the performer model 4 and the virtual camera 20 in the virtual space 3. Based on this monitoring, conditions that are likely to cause a collision between the performer model 4 and the virtual camera 20 are detected in advance. Specifically, the virtual camera control unit 44 detects the proximity state between the performer model 4 and the virtual camera 20 based on the positional relationship between the performer model 4 and the virtual camera 20. When a proximity state is detected, collision avoidance processing is executed. Here, a proximity state refers to a state where, for example, the performer model 4 and the virtual camera 20 are in close proximity, making a collision between the performer model 4 and the virtual camera 20 likely. Therefore, when a proximity state is detected, collision avoidance processing is executed to prevent a collision between the performer model 4 and the virtual camera 20 in advance.

[0085] Figure 9 is a schematic diagram showing an example of a proximity state. In this embodiment, the virtual camera control unit 44 detects a state in which the relative distance between the performer model 4 and the virtual camera 20 is less than or equal to a predetermined threshold R as a proximity state. Figure 9 schematically illustrates the state in which the relative distance = R.

[0086] Here, the relative distance between performer model 4 and virtual camera 20 is, for example, the distance between the reference point of performer model 4 and the reference point of virtual camera 20. The reference point of virtual camera 20 is typically the viewpoint (shooting position) of virtual camera 20. The reference point for performer model 4 is set, for example, to the centroid of performer model 4 (the center position of the model). In this case, the relative distance can be easily calculated, and the computational resources required for monitoring the relative distance (positional relationship) can be reduced. Alternatively, a reference point may be set on the surface of the performer model 4. In this case, for example, the distance from the viewpoint of the virtual camera 20 to the nearest surface of the performer model 4 is calculated as the relative distance. This makes it possible to reliably detect when the virtual camera 20 is in close proximity to the performer model 4, regardless of its shape or size.

[0087] Furthermore, the relative distance between the performer model 4 and the virtual camera 20 may be the current value or a predicted value. The current relative distance is calculated based, for example, on the current positions of the reference points of the performer model 4 and the virtual camera 20. Furthermore, the predicted relative distance is calculated based, for example, on the predicted positions of the reference points of the performer model 4 and the virtual camera 20. The predicted positions of the reference points of the performer model 4 are estimated based, for example, on movement predictions using the current movement direction and movement speed of the performer model 4. The predicted positions of the reference points of the virtual camera 20 are estimated based on the shooting path of the virtual camera 20. By using these predicted values, it becomes possible to reliably avoid collisions between the performer model 4 and the virtual camera 20.

[0088] As explained below, when the relative distance between performer model 4 and virtual camera 20 falls below threshold R and a proximity state is detected, the path of virtual camera 20 is controlled so that the relative distance becomes greater than threshold R. Therefore, threshold R can be said to be the minimum distance (close-up shooting tolerance) at which virtual camera 20 can approach performer model 4 to take a picture. The threshold R is set to approximately 2m on a life-size scale, for example. However, it is not limited to this, and the threshold R may be set appropriately depending on the performer model 4, the type of performance, etc.

[0089] [Collision avoidance process] This section describes the collision avoidance process that is executed when a proximity condition is detected. In the following, it is assumed that the shooting path of the virtual camera 20 in virtual space 3 is predetermined. That is, the virtual camera 20 will shoot the performer model 4 along the predetermined shooting path. In this embodiment, when the virtual camera control unit 44 detects a proximity condition, the shooting path is changed as a collision avoidance process.

[0090] Figure 10 is a schematic diagram illustrating an example of collision avoidance processing that alters the shooting path. Figure 10 schematically illustrates the shooting path 21 of the virtual camera 20, which is set up to photograph the performer model 4. The shooting path 21 includes multiple relay points. The virtual camera control unit 44 moves the virtual camera 20 so that it passes through these relay points in a predetermined order, thereby photographing the performer model 4. Thus, the shooting path 21 is a path set up so that the virtual camera 20 passes through multiple relay points in sequence.

[0091] In the example shown in Figure 10, the shooting path 21 is set to pass through relay points P1, P2, P3, and P4 in that order. The virtual camera 20 moves, for example, along a straight line connecting each relay point. Note that the path between relay points may be set in a curved shape. At this time, the shooting direction of the virtual camera 20 is adjusted as appropriate so that the performer model 4 fits within the shooting range (angle of view).

[0092] Furthermore, each relay point is assigned a time at which the virtual camera 20 should pass through that point. For example, relay points P1, P2, P3, and P4 are assigned passage times T1, T2, T3, and T4, respectively. Therefore, the virtual camera 20 is moved so that it passes through relay point P1 at time T1, through relay point P2 at time T2, through relay point P3 at time T3, and through relay point P4 at time T4. Thus, the shooting path 21 is a path in which the passage time of the virtual camera 20 is set at each of the multiple relay points.

[0093] The shooting path 21 is set appropriately, taking into account the position of the performer model 4, so as not to cause collisions between the performer model 4 and the virtual camera 20. By using the shooting path 21, it is possible to set various camera works in detail according to the content of the performance that performer model 4 (performer 1) will perform. For example, when a song is being played, the shooting path 21 is set according to the schedule of camera work, such as the time to shoot close-ups of performer model 4's facial expressions and the time to shoot a full view of performer model 4.

[0094] Furthermore, when designing the shooting route 21, at least one important relay point is set as one of several relay points. Here, an important relay point is, for example, a relay point where it is important to ensure that filming takes place during the live performance, and may be set as appropriate by the designer of the shooting route 21 (performer, creator, director, etc.). For example, a relay point for filming a specific pose of performer model 4, or the appearance of performer model 4 during the chorus of a song, may be set as an important relay point.

[0095] Furthermore, the method for setting the shooting path 21 is not limited. For example, the shooting path 21 may be set automatically. For instance, a designer sets a rough movement route for the virtual camera 20. Based on this rough movement route, the final shooting path of the virtual camera 20 (intermediate points and passing times) is automatically set according to the length of the content and the movements of performer 1.

[0096] Furthermore, even while the virtual camera 20 is moving along the shooting path 21, the position of the performer model 4 changes sequentially in accordance with the actions of performer 1 and the live performance's staging. Therefore, when live streaming, the position of the performer model 4 may move significantly more than the expected value when the shooting path 21 was set.

[0097] In the collision avoidance process shown in Figure 10, the virtual camera control unit 44 changes the relay point that serves as the movement target for the virtual camera 20 when a proximity condition is detected, so as to avoid a collision between the performer model 4 and the virtual camera 20. In other words, the relay point that was previously the movement target for the virtual camera 20 is changed to another relay point when a proximity condition is detected. As a result, the shooting path 21 of the virtual camera 20 is changed to a path that leads to the modified relay point. In this way, the process of changing the relay point becomes a collision avoidance process that changes the shooting path 21. This makes it possible to prevent the virtual camera 20 from colliding with the performer model 4.

[0098] Specifically, in a shooting path 21 with predetermined relay points, if proximity (close-range state) between the performer model 4 and the virtual camera 20 is detected, the shooting path 21 is changed to the nearest relay point where a collision will not occur. In other words, when a close-range state is detected, a shortcut is taken to the relay point that is the shortest distance away on the shooting path 21 where a collision can be avoided.

[0099] This collision avoidance process includes the following steps: • Monitoring process that detects proximity by monitoring the trajectory of the virtual camera 20 in real time according to the positional relationship between the performer model 4 and the virtual camera 20. - Route modification process that, when proximity is detected, selects the nearest relay point in sequence where a collision will not occur, and modifies the shooting route 21 so that the vehicle travels towards that relay point. • Speed ​​adjustment process that adjusts the movement speed of the virtual camera 20 by calculating backward from the increase or decrease in the movement distance of the virtual camera 20 due to the change in the shooting path 21. Please refer to Figure 10 below for a detailed explanation of each process.

[0100] For example, if virtual camera 20 passes relay point P1 at time T1, its next target is set to relay point P2. Virtual camera 20 then moves from relay point P1 to relay point P2 so that it reaches relay point P2 at time T2. During the monitoring process, the relative distance between the performer model 4 and the moving virtual camera 20 is constantly calculated, and the presence or absence of a proximity state is monitored based on the relative distance. Specifically, it is determined whether or not the relative distance is below the threshold R. For example, suppose a proximity state is detected because performer model 4 has moved. In Figure 10, when the virtual camera 20 heading towards relay point P2 reaches point X at time Tx, the relative distance between performer model 4 and virtual camera 20 becomes less than or equal to the threshold R, and a proximity state is detected.

[0101] When proximity is detected, a route rerouting process is executed. Here, the nearest staging point that will not cause a collision is selected. For example, relay points that the virtual camera 20 has not yet passed through are selected in order of proximity, and it is determined whether a collision would occur if the route were changed towards each relay point. The relay point that is determined not to cause a collision is then selected as the next target for the virtual camera 20. The determination of whether or not a collision will occur is made based on, for example, the current position and predicted movement of performer model 4, or the planned movement path (planned actions) of performer model 4.

[0102] In Figure 10, at time Tx, relay points P2, P3, and P4 are relay points that the virtual camera 20 has not yet passed through. Of these, if the virtual camera 20 is moved toward relay point P2, there is a risk of collision with the performer model 4. If the virtual camera 20 is moved toward relay point P3, the distance between the performer model 4 and the virtual camera 20 increases, so it is determined that no collision will occur. Therefore, relay point P3 is set as the next target for the virtual camera 20 as the nearest relay point where no collision will occur.

[0103] Thus, in this embodiment, the virtual camera control unit 44 sets the moving target of the virtual camera 20 to the nearest relay point that allows for collision avoidance between the performer model 4 and the virtual camera 20. This makes it possible to achieve the planned camera work without significantly altering the original shooting path 21. Furthermore, the method for changing the relay points is not limited; for example, instead of selecting relay points in the order of the shooting path 21, relay points that are closer in distance in the virtual space 3 may be selected. This makes it possible to quickly return to the original shooting path 21.

[0104] Furthermore, there are no limitations on how the relay points can be changed. For example, a new relay point may be added so that the virtual camera 20 does not collide with the performer model 4, and the virtual camera 20 may be moved toward the newly added relay point. In other words, by increasing the number of relay points, the performer model 4 can be bypassed, thus avoiding a collision between the performer model 4 and the virtual camera 20.

[0105] When the shooting path 21 is changed, a speed adjustment process is executed. In this embodiment, the virtual camera control unit 44 adjusts the movement speed of the virtual camera 20 moving along the changed shooting path 21 based on the passage time set at the relay point. For example, changing the relay point alters the distance traveled by the virtual camera 20. Therefore, if the previous travel speed were maintained, the shooting (camera work) schedule might be disrupted. To prevent such schedule disruptions, the virtual camera 20's travel speed is changed based on the time it passes the relay point. This prevents inconsistencies in the length of the content video.

[0106] In this embodiment, the virtual camera control unit 44 adjusts the movement speed of the virtual camera in accordance with the passing times of important relay points included in the modified shooting path 21. Specifically, the movement speed of the virtual camera 20 is increased or decreased so that it can pass through a key relay point at the designated time.

[0107] For example, suppose that the relay point P3 shown in Figure 10 is an important relay point. In this case, the movement speed of the virtual camera 20 is adjusted so that the virtual camera 20 passes through the relay point P3 at the set passing time T3. For example, if the distance traveled decreases by taking a shortcut to the relay point P3, the movement speed is set to be lower, and if the distance traveled increases, the movement speed is set to be higher. Furthermore, if, for example, relay point P4 is set as an important relay point, the movement speed of the virtual camera 20 is adjusted so that the virtual camera 20 passes through relay point P4 at the time T4 set for relay point P4. In this case, the timing of passing through relay point P3 does not necessarily have to be time T3.

[0108] In this way, by adjusting the movement speed of the virtual camera 20 based on the time of passage through important relay points, it becomes possible to reliably capture images at important relay points at the appropriate time. This makes it possible to capture important scenes, such as when performer model 4 (performer 1) performs a specific act, without missing any opportunities.

[0109] The method for adjusting the movement speed of the virtual camera 20 is not limited. For example, the movement speed may be adjusted to match the passing time of the changed relay point (relay point P3 in Figure 10) without referring to important relay points. This will enable shooting that is closer to the original schedule. Alternatively, the passing times of the remaining relay points may be adjusted as appropriate to match the overall performance time. In this case, the movement speed of the virtual camera 20 will be set to match the adjusted passing times. This will make it possible to suppress, for example, unnatural increases or decreases in movement speed.

[0110] Figure 11 is a flowchart of the collision avoidance process shown in Figure 10. First, while monitoring the performer model 4, the virtual camera 20 is moved so that it passes the next relay point at a predetermined time (step 101). For example, the position of the performer model 4 is monitored, and the relative distance to the virtual camera 20 is calculated based on the monitoring results. At this time, the movement speed of the virtual camera 20 is set so that it reaches the relay point set as the movement target at the time it passes that point.

[0111] It is determined whether the positional relationship between the performer model 4 and the virtual camera 20 is in a close-proximity state (step 102). The determination of the close-proximity state is performed according to the method described with reference to Figure 9, for example. If no proximity condition is detected (No. in step 102), step 105, described later, is executed.

[0112] If proximity is detected (Yes in step 102), the shooting path of the virtual camera 20 is changed to the nearest relay point where a collision will not occur (step 103). For example, if the route is changed to a relay point that the virtual camera 20 has not passed through, it is determined whether a collision can be avoided, and the nearest relay point where a collision can be avoided is selected. Then, a new shooting route 21 is set that moves towards the selected relay point.

[0113] When the shooting path 21 is changed, the movement speed of the virtual camera 20 is adjusted so that it passes through important relay points at specified times (step 104). For example, if the distance to a key relay point decreases, the movement speed is set to a lower value, and if the distance increases, the movement speed is set to a higher value. In other words, the movement speed of the virtual camera 20 is calculated backward from the changed distance.

[0114] Once the movement speed is adjusted, it is determined whether or not the virtual camera 20 has arrived at the final relay point (step 105). The final relay point is the last relay point set in the shooting path 21. If the virtual camera 20 has not reached the final relay point (No. in step 105), the process from step 101 onwards is repeated, assuming that there are still relay points that the virtual camera 20 has not passed through. If the shooting path 21 is changed, the relay point that is the target of movement in step 101 is changed to the newly set relay point, and the virtual camera 20 moves at the movement speed set in step 104.

[0115] If the virtual camera 20 arrives at the final relay point (Yes in step 105), it is determined whether the recording end time has been reached (step 106). The recording end time is the time when recording by the virtual camera 20 ends. The recording end time is set appropriately, for example, according to the performance schedule of performer model 4 (performer). If the recording end time has not been reached (No in step 106), recording will continue from the final relay point. If the recording end time has been reached (Yes in step 106), recording by virtual camera 20 will end, and the live broadcast will be completed.

[0116] Figure 12 is a schematic diagram illustrating another example of collision avoidance processing that alters the shooting path. In the collision avoidance process shown in Figure 12, the virtual camera control unit 44, when a proximity condition is detected, modifies at least a portion of the path to the relay point that serves as the movement target for the virtual camera 20 so as to avoid a collision between the performer model 4 and the virtual camera 20. In other words, the virtual camera 20 moves along a new path to reach the relay point, which was the target of the virtual camera 20's movement. As a result, the shooting path 21 of the virtual camera 20 is changed from the previous path to the next relay point (for example, a path that goes straight to the next relay point) to the new path.

[0117] This new route may be one that returns to a point along the previous route to the next relay point, or it may be a route that goes directly to the next relay point without passing through the previous route. As shown in Figure 12, collision avoidance processing is performed to change the path between relay points. This makes it possible to prevent the virtual camera 20 from colliding with the performer model 4.

[0118] Specifically, in a shooting path 21 with predetermined relay points, if proximity (close-range state) between the performer model 4 and the virtual camera 20 is detected, the virtual camera 20 moves along a bypass path 23 that bypasses the performer model 4 while monitoring the performer model 4. In other words, the virtual camera control unit 44 moves the virtual camera 20 along a bypass path 23 that bypasses the performer model 4 from the point where proximity is detected. Note that the detour route 23 will change depending on the movement of the performer model 4, etc.

[0119] This collision avoidance process includes the following steps: • Monitoring process that detects proximity by monitoring the trajectory of the virtual camera 20 in real time according to the positional relationship between the performer model 4 and the virtual camera 20. - Route modification process that, when proximity is detected, changes the shooting route 21 to detour while monitoring performer model 4, so that the route is returned to the original relay point. • Speed ​​adjustment process that adjusts the movement speed of the virtual camera 20 by calculating backward from the increase or decrease in the movement distance of the virtual camera 20 due to the change in the shooting path 21. Please refer to Figure 12 below for a detailed explanation of each process.

[0120] The monitoring process is performed in the same manner as described with reference to Figure 10, for example. In Figure 12, when the virtual camera 20 traveling from relay point P1 to relay point P2 reaches point X at time Tx, the relative distance between the performer model 4 and the virtual camera 20 becomes less than or equal to the threshold R, and a proximity state is detected.

[0121] When proximity is detected, a route change process is executed. Here, the movement of the virtual camera 20 is controlled to detour back to the original relay point (relay point P2 in Figure 12). The trajectory of the detour becomes the detour route 23. For example, as shown in Figure 12, the virtual camera 20 moves from detection point X, where proximity is detected, to the original path to relay point P2, bypassing the performer model 4. At this time, the behavior of the virtual camera 20 is controlled so that the relative distance between the performer model 4 and the virtual camera 20 remains constant. Therefore, the bypass path 23 can be said to be a path that maintains a constant relative distance between the performer model 4 and the virtual camera 20.

[0122] Figure 12 schematically illustrates the detour route of the virtual camera 20 when the performer model 4 remains in a nearly constant position. In this case, the detour route 23 is an arc-shaped path that returns to the original path from the proximity detection point X, centered on the performer model 4. Furthermore, if performer model 4 moves further after time Tx, a detour will be taken according to the movement of performer model 4, so the detour route 23 will not necessarily be in the shape of an arc.

[0123] In any case, the virtual camera 20, while taking a detour, will move at a certain distance away from the performer model 4, thus avoiding a collision between the performer model 4 and the virtual camera 20. Furthermore, the virtual camera 20 is moved back to its original path. This makes it possible to sufficiently suppress any deviation from the initially set camera work.

[0124] Furthermore, the method for changing the route to the relay point that serves as the target of movement is not limited. For example, any route that does not cause a collision, connecting the point where proximity is detected and the relay point that serves as the target of movement, may be calculated and used as the detour route 23. In this case, a route that does not cause a collision is appropriately calculated based on, for example, the current position of the performer model 4, movement prediction, and planned actions.

[0125] When the shooting path 21 is changed, a speed adjustment process is executed. For example, the method described with reference to Figure 10 can be applied as the speed adjustment process. For example, the movement speed of the virtual camera 20 traveling along the modified shooting route 21 (detour route 23) is adjusted based on the passage time set at the relay point. For instance, the movement speed of the virtual camera 20 is increased so that it reaches the relay point P2 by the passage time T2. This makes it possible to quickly return to the original shooting schedule.

[0126] Furthermore, if the shooting path 21 includes important relay points, the movement speed of the virtual camera 20 is adjusted to match the time of passage through the important relay points. For example, if relay point P3 is an important relay point, the movement speed of the virtual camera 20 is increased so that it passes through relay point P3 at time T3. In this case, the change in movement speed is smaller compared to when the speed is increased to match the time T2 of passing through relay point P2. This makes it possible to reliably capture important scenes while avoiding unnatural speed increases.

[0127] Figure 13 is a flowchart of the collision avoidance process shown in Figure 12. First, while monitoring the performer model 4, the virtual camera 20 is moved so that it passes the next relay point at a predetermined time (step 201). Next, it is determined whether the relative positions of the performer model 4 and the virtual camera 20 are in close proximity (step 202). If no proximity condition is detected (No. in step 202), step 205, described later, is executed.

[0128] If proximity is detected (Yes in step 202), performer model 4 is monitored, and based on the monitoring results, the virtual camera 20 is rerouted towards the original relay point (step 203). During detours, the virtual camera 20's position is adjusted so that, for example, the relative distance between the performer model 4 and the virtual camera 20 remains constant. The direction of movement of the virtual camera 20 is also controlled to return to the original shooting path 21. In addition, any other method of bypassing the performer model 4 may be used.

[0129] The movement speed of the virtual camera 20 is adjusted so that it passes through important relay points at specified times (step 204). For example, the increase or decrease in the distance traveled by the virtual camera 20 is estimated according to the method of detour. Here, for example, assuming that the performer model 4 does not move, the increase or decrease in distance traveled when the virtual camera 20 moves while maintaining a constant relative distance to the performer model 4 is calculated. Based on these estimation results, the travel speed required to pass the target relay point at a specified time is calculated in reverse. In addition, if the distance traveled changes in accordance with the movement of the performer model 4, the travel speed may be adjusted accordingly.

[0130] Once the movement speed is adjusted, it is determined whether or not the virtual camera 20 has arrived at the final relay point (step 205). If the virtual camera 20 has not arrived at the final relay point (No in step 205), the process from step 201 onwards is executed again. If the virtual camera 20 has arrived at the final relay point (Yes in step 205), it is determined whether or not the recording end time has been reached (step 206). If the recording end time has not been reached (No in step 206), recording from the final relay point continues, and if the recording end time has been reached (Yes in step 206), recording by the virtual camera 20 ends, and the live broadcast is completed.

[0131] The above describes collision avoidance processing mainly used when the virtual camera 20 moves along a pre-defined shooting path 21. Even when such a shooting path 21 is not defined, it is possible to avoid collisions with the performer model 4 by appropriately controlling the behavior of the virtual camera 20. The following describes how to control the virtual camera 20 that tracks and captures the performer model 4.

[0132] Figure 14 is a schematic diagram illustrating another example of collision avoidance processing for a virtual camera. Figure 14 schematically shows how a virtual camera 20 tracks the performer model 4 as it moves in the virtual space 3. Virtual space 3 contains a performer model 4 and multiple viewer models 5. The performer model 4 can move freely within virtual space 3. The viewer models 5 are positioned to surround the stage (the rectangular area in the diagram) set up in virtual space 3.

[0133] In the example shown in Figure 14, the virtual camera control unit 44 moves the virtual camera 20 so that the relative distance between the performer model 4 and the virtual camera 20 remains constant as a collision avoidance process. That is, the movement of the virtual camera 20 is controlled so that the relative distance between the virtual camera 20 and the performer model 4 is maintained at a constant interval L (for example, 2m).

[0134] This collision avoidance process includes the following steps: • Monitoring process that calculates the relative distance by monitoring the trajectory of the virtual camera 20 in real time, based on the positional relationship between the performer model 4 and the virtual camera 20. Tracking process involves moving the virtual camera 20 so that the relative distance is a constant interval L (shooting distance), and tracking and shooting the performer model 4.

[0135] During the monitoring process, the relative distance between performer model 4 and virtual camera 20 is constantly calculated. For example, the position of performer model 4, which changes according to the movement of performer 1 and the performance, and the position of virtual camera 20 are read, and the relative distance is calculated.

[0136] In the tracking process, the position and direction of the virtual camera 20 are calculated so that the relative distance calculated in the monitoring process becomes the interval L, and the virtual camera 20 is moved according to the calculation result. For example, the movement of performer model 4 may temporarily cause the relative distance to become greater than the interval L. In this case, the virtual camera 20 moves closer to performer model 4 until the relative distance becomes the interval L. Similarly, if the relative distance temporarily becomes less than the interval L, the virtual camera 20 moves away from performer model 4 until the relative distance becomes the interval L. In other words, the virtual camera control unit 44 performs feedback control to maintain the relative distance between the performer model 4 and the virtual camera 20 at interval L, based on the position of the performer model 4.

[0137] Thus, in the collision avoidance process shown in Figure 14, the relay points are not fixed to a predetermined shooting path 21. Instead, the virtual camera 20 is moved to track the performer model 4 while maintaining a certain distance L from it, and the performer model 4 is photographed. This avoids collisions between the performer model 4 and the virtual camera 20, making it possible to continue stable close-up shooting. The direction in which the virtual camera 20 photographs the performer model 4 is not limited. For example, the virtual camera 20 may move according to the posture of the performer model 4 so as to photograph the performer model 4 from the front. Alternatively, the virtual camera 20 may be controlled to move freely around the performer model 4 within a range where the relative distance is interval L.

[0138] Additionally, for avatars other than performer model 4 (viewer model 5), "clipping" and "passing through" of the virtual camera 20 are permitted. For example, for viewer model 5 that the virtual camera 20 comes into contact with, the data that makes up the model is removed so that it does not appear within the shooting range of the virtual camera 20. In other words, viewer model 5 that the virtual camera 20 comes into contact with is treated as if it does not appear in the virtual camera 20's view. This makes it possible to properly film the performer model 4. Furthermore, since the movement range of the virtual camera 20 is not restricted, it becomes easy to perform shots and other actions that would be difficult in real space. Furthermore, it is possible to intentionally represent "penetration" or "passing through" of the viewer model 5 by using silhouettes or other methods.

[0139] In the example shown in Figure 14, performer model 4, who was on the stage at time T1, leaves the stage, and at times T2 and T3, moves to the area where viewer model 5 is positioned. Virtual camera 20 moves in accordance with performer model 4 and tracks and photographs performer model 4 from a position a certain distance L away. At this time, viewer model 5 that is in contact with virtual camera 20 is treated as not being captured by virtual camera 20. This makes it possible to properly capture close-up shots of performances that take place away from the stage (such as fan service).

[0140] Figure 15 is a flowchart of the collision avoidance process shown in Figure 14. As the performer model 4 moves, the virtual camera 20 moves to maintain a constant relative distance L (step 301). For example, the current positions of the performer model 4 and the virtual camera 20 are read, and the relative distance is calculated (monitoring process). The position and direction in which the virtual camera 20 will move are calculated so that this relative distance becomes the interval L, and the virtual camera 20 moves based on the calculation result (tracking process).

[0141] Next, it is determined whether the recording end time has been reached (step 302). If the recording end time has not been reached (No in step 302), step 301 is executed again. If the recording end time has been reached (Yes in step 302), recording by the virtual camera 20 ends, and the live broadcast is completed. In this way, the tracking process (monitoring process) continues until the recording end time is reached.

[0142] As part of the tracking process, control may be performed such as slowing down and stopping the virtual camera 20 to avoid collisions, and then resuming tracking once it has moved a certain distance away. For example, if the performer model 4, which was moving, suddenly stops, the virtual camera 20 is gradually decelerated and then brought to a stationary position where no collision will occur. At this time, the relative distance between the performer model 4 and the virtual camera 20 may be smaller than the interval L. For example, when the performer model 4, which was stationary, starts moving again, the virtual camera 20 waits until the relative distance between it and the performer model 4 increases by a certain distance (e.g., interval L) before starting to track it.

[0143] This can be described as a process of moving the virtual camera 20 by shifting the timing of the change in the virtual camera's speed from the timing of the change in the performer model 4's speed. In other words, the virtual camera control unit 44 delays the timing of the change in the virtual camera 20's speed relative to the timing of the change in the performer model 4's speed so that the relative distance falls within a predetermined range. This makes it possible to achieve camera work where the shooting distance dynamically changes when performer model 4 starts moving or stops. As a result, it becomes possible to convey the dynamic movements of performer model 4 (performer 1). Furthermore, the method of tracking and filming the performer model 4 is not limited, and any tracking method that can avoid collisions between the performer model 4 and the virtual camera 20 may be used.

[0144] In this embodiment, the server control unit 42 performs collision avoidance processing according to the positional relationship between the performer model 4 in the virtual space 3 and the virtual camera 20 that photographs the performer model 4 while moving around the virtual space 3. As a result, collisions between the performer model 4 and the virtual camera 20 are avoided in advance, making it possible to properly photograph the performer model 4 in the virtual space 3.

[0145] In live streaming using virtual spaces, when a virtual camera is brought close to the subject to be filmed for close-up shots, there is a possibility of collision between the subject and the virtual camera. Figure 16 is a schematic diagram illustrating an example of a collision between a subject being filmed and a virtual camera. In Figure 16, a filming path 21 (P1~P4) is set in which the virtual camera 20 approaches the subject 70 to perform close-up filming. If the subject 70 moves and overlaps with the filming path 21, the virtual camera 20 may collide with the subject 70, resulting in embedded or penetrating footage being filmed, potentially causing a broadcast accident.

[0146] In this embodiment, collision avoidance processing is performed according to the positional relationship between the performer model 4 and the virtual camera 20 that are to be filmed. The collision avoidance processing is, for example, a process that monitors the position of the performer model 4 and controls the movement of the virtual camera 20 so that it does not get too close to the performer model 4. For example, as explained with reference to Figures 10 and 12, if the shooting path 21 of the virtual camera 20 is pre-set, the proximity state between the performer model 4 and the virtual camera 20 is detected and the shooting path 21 is changed. Also, as explained with reference to Figure 14, if the shooting path 21 is not set, the virtual camera 20 moves to track the performer model 4 at regular intervals L.

[0147] By performing this process, it becomes possible to prevent situations such as the virtual camera 20 colliding with performer model 4, regardless of the performer model 4's behavior. This makes it possible to live stream the performance of performer model 4 (performer 1) without causing broadcast accidents such as clipping or penetrating footage. In the future, it is expected that services that deliver interactive, real-time live content using a volumetric model of performer 1 in real space will become widespread. Even in such cases, this technology makes it possible to deliver high-quality live images while avoiding collisions between the virtual camera 20 and the performer model 4.

[0148] <Other Embodiments> This technology is not limited to the embodiments described above, and various other embodiments can be realized.

[0149] Figures 10 and 12 primarily illustrate collision avoidance processing, which involves changing the virtual camera's shooting path to avoid collisions with the performer model. However, the process is not limited to this; for example, collision avoidance processing may be performed by controlling the virtual camera's movement speed. For example, if proximity is detected, the virtual camera can be slowed down and brought to a stop without changing its path. If the performer model moves and the proximity condition is removed, the virtual camera can be accelerated to match the time of passing through a relay point. Furthermore, if the performer model remains stationary and proximity persists for a certain period, the camera's shooting path can be changed.

[0150] The above describes the case where a performer model is captured using a single virtual camera. For example, the performer model may be captured using multiple virtual cameras. In this case, each virtual camera is controlled to move along a different trajectory (shooting path). Furthermore, a display image for displaying the performer model is selected from the images (videos) captured by each virtual camera. Here, the display image is, for example, the image that is ultimately displayed on each client terminal.

[0151] In this configuration, where multiple virtual cameras are used, if proximity is detected, a collision avoidance process is executed to switch the display image used to show the performer model from the image captured by one virtual camera to the image captured by another virtual camera. For example, suppose a proximity condition is detected for a virtual camera that is capturing an image for display. In this case, the display image is switched to an image from another virtual camera that is not in proximity. This process of switching virtual cameras to prevent the image of the moment of collision from being streamed is also included in the collision avoidance process in this disclosure. This makes it possible to deliver natural live video without causing any broadcasting errors.

[0152] The above explanation described collision avoidance processing for virtual cameras, using the example of a case where the subject of the virtual camera's shooting is a live-action 3D model (volumetric model) of the performer. This technology is applicable regardless of the type of subject being shot. For example, the subject of the virtual camera's shooting may be a CG3D model. For instance, a 3D CG model (virtual avatar) that moves in conjunction with the performer's movements may be captured by the virtual camera. Even in such cases, the method described above can be used to sufficiently avoid collisions between the virtual camera and the CG model.

[0153] Furthermore, the above section explained the case of filming live performances, such as music concerts, in a virtual space. This technology is not limited to this example; it can also be applied to the distribution of entertainment content such as "live sports broadcasts" and "variety shows." For example, a sports venue can be recreated in a virtual space from data captured of a sporting event taking place in the real world. This enables free-viewpoint photography of the sports venue using a virtual camera. In such cases, the collision avoidance process described above is applied to prevent collisions between the player and the virtual camera.

[0154] Furthermore, this technology may be applied when distributing content in fields such as "distance education," "distance lessons," and "remote work support." For example, a work site can be recreated in a virtual space from data captured by photographing a work site where remote work is performed using robots, etc. In this case, the virtual camera is moved to avoid collisions with the robot. This makes it possible to monitor the work site without interruption and fully support remote work.

[0155] The above primarily described the case of live streaming images captured by a virtual camera. However, this technology is not limited to this case; for example, it can also be applied to generating content based on data recorded from performance in a virtual space. For example, when generating live video after a live performance (for instance, when the content production company and the company that shoots and edits the content are different), the previously performed music live performance is recreated based on data from the virtual space, and filmed using virtual cameras. In this case, this technology makes it easy to generate virtual camera routes that avoid collisions between performer models and virtual cameras. Thus, this technology functions as a virtual camera path generation tool or a design support tool for shooting routes.

[0156] It is also possible to combine at least two of the feature features of the present technology described above. In other words, the various feature features described in each embodiment may be combined arbitrarily, regardless of the specific embodiment. Furthermore, the various effects described above are merely examples and not limiting, and other effects may also be exhibited.

[0157] In this disclosure, "same," "equal," "orthogonal," etc., are concepts that include "substantially the same," "substantially equal," "substantially orthogonal," etc. For example, states that fall within a predetermined range (e.g., a range of ±10%) based on "exactly the same," "exactly equal," "exactly orthogonal," etc.

[0158] Furthermore, this technology can also be configured as follows. (1) A shooting control unit that performs collision avoidance processing to avoid collisions of the virtual camera with the target object, depending on the positional relationship between the target object operating in the virtual space and the virtual camera that moves in the virtual space and photographs the target object. An information processing device equipped with the following. (2) An information processing device as described in (1), The shooting control unit detects the proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and executes the collision avoidance process when the proximity state is detected. Information processing device. (3)(2) Information processing device described above, The shooting control unit detects a state in which the relative distance between the target object and the virtual camera is below a predetermined threshold as the proximity state. Information processing device. (4)(3) The information processing device described above, The aforementioned relative distance includes the current value or the predicted value. Information processing device. (5) An information processing device described in any one of (2) to (4), The virtual camera photographs the target object along a predetermined shooting path, When the proximity condition is detected, the shooting control unit changes the shooting path as part of the collision avoidance process. Information processing device. (6)(5) Information processing device, The aforementioned shooting path is a path configured such that the virtual camera passes through multiple relay points in sequence. Information processing device. (7)(6) The information processing device described above, When the proximity condition is detected, the shooting control unit changes the relay point that serves as the virtual camera's movement target to avoid a collision between the target object and the virtual camera. Information processing device. (8)(7) Information processing device, The shooting control unit sets the virtual camera's target to the nearest intermediate point in the sequence that allows for collision avoidance between the target object and the virtual camera. Information processing device. An information processing device described in any one of (9)(6) to (8), When the proximity state is detected, the shooting control unit modifies at least a portion of the path to the relay point that serves as the virtual camera's target so as to avoid a collision between the target object and the virtual camera. Information processing device. (10)(9) Information processing device, The aforementioned shooting control unit moves the virtual camera along a detour path that bypasses the target object, starting from the point where the proximity state is detected. Information processing device. (11)(10) Information processing device, The aforementioned detour route is a route that maintains a constant relative distance between the target object and the virtual camera. Information processing device. An information processing device described in any one of (12)(6) to (11), The aforementioned shooting path is a path in which the passage time of the virtual camera is set for each of the multiple relay points. The shooting control unit adjusts the movement speed of the virtual camera moving along the modified shooting path based on the passage time set at the relay point. Information processing device. (13)(12) Information processing device, The aforementioned plurality of relay points include at least one important relay point, The shooting control unit adjusts the movement speed of the virtual camera to match the passing time of the important relay points included in the modified shooting path. Information processing device. An information processing device described in any one of (14)(2) to (13), When the proximity condition is detected, the shooting control unit, as part of the collision avoidance process, switches the display image for showing the target object from an image captured by the virtual camera to an image captured by another virtual camera. Information processing device. (15) An information processing device described in any one of (1) to (14), The aforementioned shooting control unit, as part of the collision avoidance process, moves the virtual camera so that the relative distance between the target object and the virtual camera remains constant. Information processing device. (16)(15) Information processing device, The shooting control unit delays the timing of the virtual camera's velocity change relative to the timing of the target object's velocity change so that the relative distance falls within a predetermined range. Information processing device. (17) An information processing device described in any one of (1) to (16), The aforementioned target object includes a 3D live-action model of the performer. Information processing device. (18) An information processing device described in any one of (1) to (17), The aforementioned shooting control unit distributes the images captured by the virtual camera in real time. Information processing device. (19) Depending on the positional relationship between the target object operating in the virtual space and the virtual camera that moves in the virtual space and photographs the target object, a collision avoidance process is executed to avoid collisions of the virtual camera with the target object. A method of information processing performed by a computer system. (20) An image acquisition unit that acquires an image taken by a virtual camera which operates in accordance with a collision avoidance process that is performed according to the positional relationship between a target object operating in a virtual space and a virtual camera that moves in the virtual space and takes pictures of the target object, in order to avoid collisions of the virtual camera with the target object, A display unit that displays the image captured by the virtual camera. A display device equipped with the following. [Explanation of symbols]

[0159] 1…Performer 2…Viewers 3…Virtual space 4…Performer Model 5… Viewer Model 10…Distribution Server 12…Client terminals 14…HMD 20…Virtual Camera 21…Shooting route 22…Virtual camera footage 23... Detour route 40…Network transmission section 41...Storage section 42…Server Control Unit 43…Content Data Generation Unit 46... Viewer Model Generation Unit 100…Distribution system

Claims

1. A shooting control unit executes collision avoidance processing to avoid collisions between the virtual camera and the target object, depending on the positional relationship between the target object operating in the virtual space and the virtual camera that moves through the virtual space and automatically photographs the target object. It is equipped with, The shooting control unit detects the proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and when the proximity state is detected, it executes the collision avoidance process. The virtual camera photographs the target object along a predetermined shooting path, When the proximity condition is detected, the shooting control unit changes the shooting path as part of the collision avoidance process. Information processing device.

2. An information processing apparatus according to claim 1, The shooting control unit detects a state in which the relative distance between the target object and the virtual camera is below a predetermined threshold as the proximity state. Information processing device.

3. An information processing apparatus according to claim 2, The aforementioned relative distance includes the current value or the predicted value. Information processing device.

4. An information processing apparatus according to claim 1, The aforementioned shooting path is a path configured such that the virtual camera passes through multiple relay points in sequence. Information processing device.

5. An information processing apparatus according to claim 4, When the proximity condition is detected, the shooting control unit changes the relay point that serves as the virtual camera's movement target to avoid a collision between the target object and the virtual camera. Information processing device.

6. An information processing device according to claim 5, The shooting control unit sets the virtual camera's target to the nearest intermediate point in the sequence that allows for collision avoidance between the target object and the virtual camera. Information processing device.

7. An information processing apparatus according to claim 4, When the proximity state is detected, the shooting control unit modifies at least a portion of the path to the relay point that serves as the virtual camera's target so as to avoid a collision between the target object and the virtual camera. Information processing device.

8. An information processing apparatus according to claim 7, The aforementioned shooting control unit moves the virtual camera along a detour path that bypasses the target object, starting from the point where the proximity state is detected. Information processing device.

9. An information processing apparatus according to claim 8, The aforementioned detour route is a route that maintains a constant relative distance between the target object and the virtual camera. Information processing device.

10. An information processing apparatus according to claim 4, The aforementioned shooting path is a path in which the passage time of the virtual camera is set for each of the multiple relay points. The shooting control unit adjusts the movement speed of the virtual camera moving along the modified shooting path based on the passage time set at the relay point. Information processing device.

11. An information processing apparatus according to claim 10, The aforementioned plurality of relay points include at least one important relay point, The shooting control unit adjusts the movement speed of the virtual camera to match the passing time of the important relay points included in the modified shooting path. Information processing device.

12. An information processing apparatus according to claim 1, The aforementioned target object includes a 3D live-action model of the performer. Information processing device.

13. An information processing apparatus according to claim 1, The aforementioned shooting control unit distributes the images captured by the virtual camera in real time. Information processing device.

14. Depending on the positional relationship between the target object operating in the virtual space and the virtual camera that moves through the virtual space and automatically photographs the target object, collision avoidance processing is executed to prevent the virtual camera from colliding with the target object. This is an information processing method performed by a computer system, The process of executing the collision avoidance process according to the positional relationship between the target object and the virtual camera involves detecting a proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and executing the collision avoidance process when the proximity state is detected. The virtual camera photographs the target object along a predetermined shooting path, The process of executing the collision avoidance process according to the positional relationship between the target object and the virtual camera, when the proximity state is detected, changes the shooting path as part of the collision avoidance process. Information processing methods.

15. An image acquisition unit that acquires an image captured by a virtual camera that operates according to a collision avoidance process to avoid collisions between the virtual camera and the target object, which is executed according to the positional relationship between the target object operating in the virtual space and the virtual camera that moves in the virtual space and automatically photographs the target object. A display unit that displays the image captured by the virtual camera. It is equipped with, The process of executing the collision avoidance process according to the positional relationship between the target object and the virtual camera involves detecting a proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and executing the collision avoidance process when the proximity state is detected. The virtual camera photographs the target object along a predetermined shooting path, The process of executing the collision avoidance process according to the positional relationship between the target object and the virtual camera, when the proximity state is detected, changes the shooting path as part of the collision avoidance process. Display device.

16. A shooting control unit executes collision avoidance processing to avoid collisions between the virtual camera and the target object, depending on the positional relationship between the target object operating in the virtual space and the virtual camera that moves through the virtual space and automatically photographs the target object. It is equipped with, The shooting control unit detects the proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and when the proximity state is detected, it executes the collision avoidance process. When the proximity condition is detected, the shooting control unit, as part of the collision avoidance process, switches the display image for showing the target object from an image captured by the virtual camera to an image captured by another virtual camera. Information processing device.

17. Depending on the positional relationship between the target object operating in the virtual space and the virtual camera that moves through the virtual space and automatically photographs the target object, collision avoidance processing is executed to prevent the virtual camera from colliding with the target object. This is an information processing method performed by a computer system, The process of executing the collision avoidance process according to the positional relationship between the target object and the virtual camera involves detecting a proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and executing the collision avoidance process when the proximity state is detected. The collision avoidance process, which is performed according to the positional relationship between the target object and the virtual camera, when the proximity state is detected, switches the display image for displaying the target object from an image captured by the virtual camera to an image captured by another virtual camera as part of the collision avoidance process. Information processing methods.

18. An image acquisition unit that acquires an image captured by a virtual camera that operates according to a collision avoidance process to avoid collisions between the virtual camera and the target object, which is executed according to the positional relationship between the target object operating in the virtual space and the virtual camera that moves in the virtual space and automatically photographs the target object. A display unit that displays the image captured by the virtual camera. It is equipped with, The process of executing the collision avoidance process according to the positional relationship between the target object and the virtual camera involves detecting a proximity state between the target object and the virtual camera based on the positional relationship between the target object and the virtual camera, and executing the collision avoidance process when the proximity state is detected. The collision avoidance process, which is performed according to the positional relationship between the target object and the virtual camera, when the proximity state is detected, switches the display image for displaying the target object from an image captured by the virtual camera to an image captured by another virtual camera as part of the collision avoidance process. Display device.

Images