Information processing device, information processing method, and information processing system

Abstract

[Problem] To provide an information processing device, an information processing method and an information processing system, with which it is possible to make photographing in a virtual space a more entertaining experience. [Solution] An information processing device comprising a control unit that, when an imaging trigger that images a subject in a virtual space is detected, generates an image in which the subject has been imaged and performs control to position the generated imaged image as a virtual object in the virtual space.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing system. [Background technology]

[0002] Virtual reality (VR) applications, which have become popular in recent years, allow users to view virtual spaces in which 3D models are placed from any viewpoint. Such VR worlds are primarily provided using non-transparent head-mounted displays (HMDs), which cover the user's field of vision with a display. One way to enhance the experience in virtual spaces is to take photographs within the virtual space (save the images displayed as virtual space). Current games and video streaming content have a screenshot function that captures still images of the video rendered across the entire display device.

[0003] Furthermore, with regard to the technology for providing virtual spaces, the following documents can be cited, for example:

[0004] The following Patent Document 1 discloses a technology in which, when a user operates a camera object (virtual object) placed in a virtual space to take a picture of the virtual space (a panoramic image unfolded in the virtual space), the photographic image generated by the shooting is displayed on a monitor object. Then, in response to a slide operation on the monitor object by the user, the photographic object is placed in the virtual space.

[0005] Furthermore, Patent Document 2 below discloses a technique for outputting a photographic image taken by operating an avatar (a character object representing the user) in a virtual space from a printer in the real world. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-021122 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-176025 Summary of the Invention [Problem to be solved by the invention]

[0007] However, when placing a photographic image captured in a virtual space in the virtual space, the method described in Patent Document 1 requires two operations, a photographing operation and a sliding operation, which makes the operations complicated.

[0008] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing system that can further enhance the entertainment value of the photography experience in a virtual space. [Means for solving the problem]

[0009] According to the present disclosure, an information processing device is proposed that includes a control unit that, when an imaging trigger for capturing an image of a subject in a virtual space is detected, generates an image of the subject and controls the placement of the generated image in the virtual space as a virtual object.

[0010] According to the present disclosure, an information processing method is proposed, which includes, when a processor detects an imaging trigger for capturing an image of a subject in a virtual space, generating an image of the subject and controlling the placement of the generated image in the virtual space as a virtual object.

[0011] According to the present disclosure, an information processing system is proposed that includes a display device, a controller, and an information processing device having a control unit that, when an imaging trigger for imaging a subject in a virtual space displayed on the display device is detected by the controller, generates an image of the subject and controls the generated image to be placed in the virtual space as a virtual object. [Brief explanation of the drawings]

[0012] [Figure 1]FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a diagram showing an example of a virtual space image from a user's viewpoint according to this embodiment. [Figure 3] 1A and 1B are diagrams illustrating the arrangement of captured images in a virtual space according to the present embodiment. [Figure 4] 1 is a block diagram showing an example of the configuration of a display device according to an embodiment of the present invention. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of a controller according to the present embodiment. [Figure 6] 1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment of the present invention; [Figure 7] 10 is a flowchart showing an example of the flow of an imaging process according to the present embodiment. [Figure 8] 10A to 10C are diagrams illustrating image acquisition in response to a photographing operation according to the present embodiment. [Figure 9] 10A and 10B are diagrams illustrating continuous image acquisition performed as insurance against photographing failure according to the present embodiment. [Figure 10] 10 is a flowchart showing an example of the flow of a captured image arrangement process according to the present embodiment. [Figure 11] 10A and 10B are diagrams illustrating arrangement positions of captured images according to the present embodiment. [Figure 12] 10A and 10B are diagrams illustrating a case where a movement trajectory to an arrangement position of a captured image is drawn according to the present embodiment. [Figure 13] 10 is a flowchart showing an example of the flow of tag processing of captured images according to the present embodiment. [Figure 14] 10A to 10C are diagrams illustrating a filtering process of a captured image according to the present embodiment. [Figure 15] 10 is a flowchart showing an example of the flow of filtering processing during image capture according to the present embodiment. [Figure 16] 10A and 10B are diagrams illustrating an example of manual rearrangement of captured images according to the present embodiment. [Figure 17]10A and 10B are diagrams illustrating an example of automatic rearrangement (alignment) according to the present embodiment. [Figure 18] 10 is a flowchart showing an example of the flow of a captured image rearrangement process according to the present embodiment. [Figure 19] FIG. 10 is a diagram illustrating an example of a sharing operation according to the present embodiment. [Figure 20] 10 is a flowchart showing an example of the flow of a captured image sharing process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0014] The explanation will be given in the following order: 1. Overview of information processing system according to one embodiment of the present disclosure 2.Configuration example 3. Operation processing 3-1. Image processing 3-2. Arrangement of captured images 3-3. Tag processing of captured images 3-4. Filtering process during imaging 3-5.Rearrangement of captured images 3-6. Sharing captured images 4.Other 5. Summary

[0015] <<1. Overview of Information Processing System According to One Embodiment of the Present Disclosure>> An overview of an information processing system according to an embodiment of the present disclosure will be described with reference to Figures 1 to 3. Figure 1 is a diagram showing an example configuration of an information processing system according to an embodiment of the present disclosure. As shown in Figure 1, the information processing system according to this embodiment includes a display device 10, a controller 20, and an information processing device 30.

[0016] In this embodiment, a more entertaining shooting experience is provided in a virtual space (VR: Virtual Reality). The virtual space is presented using, for example, a non-transparent HMD (Head Mounted Display) that covers the user's field of vision. In this embodiment, it is assumed, as an example, that the display device 10 is realized by an HMD.

[0017] The HMD that realizes the display device 10 is worn on the user's head and may have an image display unit for each eye, and may be configured to control the user's vision and hearing by using headphones in combination. By blocking the outside world, the sense of virtual reality (the sense of immersion in a virtual space) during viewing can be enhanced. The HMD can also project different images to each eye, and can present 3D images by displaying images with parallax to the left and right eyes.

[0018] In addition, virtual objects, which are generated three-dimensional images (so-called 3D models), and stereoscopic images generated based on information obtained by capturing images of the real space are arranged (displayed) in the virtual space. The real space may be captured by an omnidirectional camera. Alternatively, the real space may be captured by multiple cameras.

[0019] Such virtual space images (also referred to herein as VR content) may be spherical content or free-viewpoint content. Free-viewpoint content generates images as if a virtual camera were placed at any position, allowing the image to be viewed from any viewpoint. As used herein, the term "virtual space" (or "VR") refers to a representation of a real or imaginary environment with rules of interaction simulated by one or more processors that a real user can perceive through one or more display devices and / or interact with through one or more user interfaces. The term "user interface" refers to a real device through which a user can send input to or receive output from a virtual world.

[0020] In the virtual space, the user may be represented by an avatar, or the world of the virtual space may be displayed from the viewpoint of the avatar without displaying the avatar on the display. In this specification, the viewpoint of the user (or avatar) in the virtual space (also referred to as the "user viewpoint in the virtual space") can be considered as the field of view of a virtual camera. The "virtual camera" refers to a viewpoint in the virtual space, and is used in calculations to render the three-dimensional virtual space as a two-dimensional image on a display (display device).

[0021] Furthermore, the VR content according to this embodiment may be, for example, live distribution (real-time distribution) of a music concert, a stage performance, various events, lectures, classes, programs, etc., or may be a recording of a past distribution, or may be recorded or generated for distribution. Furthermore, the VR content may be a game, a movie, a drama, an anime, etc. The content of the VR content is not particularly limited.

[0022] The information processing system according to this embodiment also includes a controller 20 for transmitting the user's intentions to the system. There may be multiple controllers 20. The controller 20 may be held in the user's hand, or may be attached to a part of the body, such as the elbow, arm, knee, ankle, or thigh. The user can control photography in the virtual space using the controller 20.

[0023] Furthermore, the information on the virtual space (VR content) provided by the display device 10 may be acquired from an information processing device 30. The information processing device 30 may be a server provided on a network, or may be realized by a dedicated terminal located in the same space as the user, a smartphone, a tablet terminal, a PC, or the like. The information processing device 30 may also be realized by a plurality of devices. For example, the information processing device 30 may be realized by a server provided on a network and a dedicated terminal located in the same space as the user.

[0024] The information processing device 30 has a function of storing VR content and providing a virtual space constructed based on the VR content to the display device 10. Specifically, the information processing device 30 generates a free viewpoint image from the VR content in accordance with the user's movements, controller operations, etc., and outputs the image to the display device 10. The user's movements refer to changes in the user's position and posture (including the direction of the head). The display device 10 detects the user's position and posture and outputs the detected position and posture to the information processing device 30. The information processing device 30 recognizes the user's viewpoint position (three-dimensional position) in the real world and calculates the user's viewpoint position in the virtual space based on the recognized user's viewpoint position.

[0025] The display device 10 may perform the process of generating a free viewpoint image from the VR content in accordance with the user's movements, controller operations, and the like.

[0026] (background) Here, one way to further enjoy the virtual space experience is to take photos in the virtual space, such as to commemorate a particular scene in the virtual space or share it with friends.

[0027] Currently, when watching video streaming on a smartphone or other device, there is a screenshot function that saves the video displayed on the display screen as a still image as a way to save the scene. However, since this is simply a still image of the video displayed on the display screen, it does not have the same sense of realism as a photograph taken on location.

[0028] On the other hand, if it were possible to take photos from the user's point of view in a virtual space, it would be possible to obtain realistic site photos. Furthermore, if the photos taken in this way could be shared with friends, the VR content would be even more enjoyable, increasing its value and potentially contributing to business expansion.

[0029] Furthermore, if photos taken in a virtual space could be viewed at hand within the virtual space, users could enjoy the photos they took while maintaining their immersive feeling in the virtual space. Furthermore, when watching a music concert as VR content, it is expected that users will want to focus on the object of their viewing while taking photos of their favorite scenes, so it is desirable to reduce the hassle of taking and viewing photos as much as possible.

[0030] Therefore, in the information processing system disclosed herein, the process from capturing images to arranging the captured images is performed in accordance with user operations to photograph a subject in a virtual space, reducing the hassle of operations, and the captured images are arranged in the virtual space as virtual objects, thereby making it possible to further enhance the entertainment value of the photography experience in the virtual space.

[0031] Specifically, as shown in FIG. 2, for example, when a virtual space image 400 is displayed on the display unit 140 of the display device 10, it is assumed that the user uses the controller 20 to capture an image of a subject 401 appearing in the image 400. The user can perform a so-called shutter release by performing a capture operation, such as pressing a predetermined button on the controller 20. In this specification, a "subject in a virtual space" is an image displayed on the display unit 140 as information about the virtual world. The "subject in a virtual space" may be, for example, a person or object included in a stereoscopic image generated based on real-life video, or may be a 3D model (virtual object) generated by CG or the like. The "capture operation" is an example of an image capture trigger for capturing an image of a subject in a virtual space.

[0032] When a photographing operation, such as pressing a predetermined button on the controller 20, is performed, the information processing device 30 regards the shutter as having been released, and acquires (generates) an image with a predetermined angle of view 600 (frame size) based on the user's viewpoint in the virtual space. The generation of such an image corresponds to so-called "photographing." The image capturing processing unit 302 may display the angle of view 600 in the virtual space. This allows the user to intuitively grasp the angle of view (photographing range) for photographing. For example, the image capturing processing unit 302 may cause the angle of view 600 (frame) to appear when the user half-presses a predetermined button on the controller 20, and then execute photographing when the button is pressed down (fully pressed). The information processing device 30 may also be capable of operating zoom-in and zoom-out for photographing.

[0033] The information processing device 30 renders (generates) a 2D texture (two-dimensional image) with a field of view of 600, and draws the 2D texture on a photo object (a virtual object used as a photographic image). The photo object on which the 2D texture is drawn corresponds to a "captured image" (a photograph obtained by a photographing operation) in the virtual space.

[0034] The information processing device 30 then places the captured image in the virtual space. The location of the placement may be, for example, beside the user, in a position that does not obstruct the user's field of view (at least a position that does not overlap with the stage). For example, as shown in FIG. 2, when the user photographs a subject 401 while looking toward the stage, the captured image (image with a field of view of 600) may be placed to the right of the user (outside the field of view). In this case, when the user turns to the right (turns their face to the right), the user's viewpoint in the virtual space changes, and as shown in FIG. 3, an image 410 of the virtual space to the right is displayed on the display unit 140. Then, the captured image 510 (virtual object) placed to the right of the user can be viewed.

[0035] The information processing system according to an embodiment of the present disclosure has been outlined above. Next, the specific configuration of each device included in the information processing system according to the present embodiment will be described with reference to the drawings.

[0036] <<2. Configuration Example>> <2-1. Configuration example of display device 10> 4 is a block diagram showing an example of the configuration of the display device 10 according to this embodiment. As shown in FIG. 4, the display device 10 includes a control unit 100, a communication unit 110, an operation input unit 120, a sensor unit 130, a display unit 140, an audio output unit 150, and a storage unit 160.

[0037] The control unit 100 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the display device 10 in accordance with various programs. The control unit 100 is realized by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 100 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0038] The control unit 100 according to this embodiment controls the display unit 140 and the audio output unit 150 based on information received from the information processing device 30 via the communication unit 110, thereby controlling the presentation of virtual space information to the user's visual and auditory senses. The control unit 100 also controls the transmission of information about user operations input from the operation input unit 120 and changes in the user's position, posture, etc. based on data sensed by the sensor unit 130 from the communication unit 110 to the information processing device 30. The control unit 100 also controls the transmission of information about user operations received from the controller 20 to the information processing device 30. The control unit 100 may also control the display unit 140 and the audio output unit 150 in accordance with the information about the user operations and changes in the position, posture, etc., thereby changing the virtual space information presented to the user's visual and auditory senses.

[0039] The following describes the calculation of position, orientation, and the like by the control unit 100 based on sensing data sensed by the sensor unit 130. Based on various sensing data, the control unit 100 tracks the user's head movement (head tracking), gaze movement (eye tracking), and position and orientation (position tracking). More specifically, for example, head tracking can be performed by calculating head orientation information based on nine-axis detection results detected by the sensor unit 130 provided in the display device 10 worn on the user's head. Eye tracking can also be performed by calculating the user's gaze (gaze direction) based on an image of the user's eye (e.g., an infrared image captured by emitting infrared light to the eye and capturing the reflection) detected by the sensor unit 130 provided in the display device 10 worn on the user's head. Position tracking can be classified into an outside-in method using an externally installed sensor, an inside-out method using a sensor mounted on the object to be measured (the display device 10 (HMD)) itself, and a hybrid method that combines these.

[0040] (Communication unit 110) The communication unit 110 is connected to and communicates with the information processing device 30 and the controller 20 via wired or wireless means to transmit and receive data. The communication unit 110 can perform communication using, for example, wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or a mobile communication network (LTE (Long Term Evolution), 3G (third generation mobile communication system), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), etc.

[0041] (Operation input unit 120) The operation input unit 120 receives operation instructions from a user and outputs the operation content to the control unit 100. The operation input unit 120 may be, for example, a touch sensor, a pressure sensor, or a proximity sensor. Alternatively, the operation input unit 120 may be a physical component such as a button, a switch, or a lever.

[0042] (Sensor unit 130) The sensor unit 130 has a function of sensing real space, such as the user or surrounding conditions. Specifically, the sensor unit 130 includes a position information acquisition unit, a camera (inward-facing / outward-facing camera), a microphone, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, and a biosensor (for detecting pulse, heart rate, sweat, blood pressure, body temperature, breathing, electromyography, brain waves, etc.). Specific examples of the sensor unit 130 are not limited to these. For example, the sensor unit 130 may have a sensor capable of detecting a total of nine axes, such as a three-axis gyro sensor, a three-axis acceleration sensor, and a three-axis geomagnetic sensor. Furthermore, the sensor unit 130 may have, as a gaze detection sensor for detecting the user's gaze, an inward-facing camera such as an infrared sensor (infrared light emitting unit and infrared camera) provided around the display unit 140 located in front of the user's eyes, an electromyography sensor for detecting muscle movements around the user's eyes, or an electroencephalogram sensor.

[0043] (Display section 140) For example, when the display device 10 is configured as an HMD, the display unit 140 has left and right screens fixed to the user's left and right eyes, respectively, and displays an image for the left eye and an image for the right eye. The screen of the display unit 140 is configured, for example, by a display panel such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display, or a laser scanning display such as a direct retinal imaging display. The display unit 140 may also include an imaging optical system that enlarges and projects the display screen and forms an enlarged virtual image with a predetermined angle of view on the user's pupil.

[0044] (Audio output unit 150) For example, when the display device 10 is configured as an HMD, the audio output unit 150 is configured as headphones worn on the user's head and plays back audio signals. Note that the audio output unit 150 is not limited to a headphone type, and may be configured as earphones or a bone conduction speaker.

[0045] (Storage unit 160) The storage unit 160 is realized by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 100, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0046] Although the configuration of the display device 10 has been specifically described above, the configuration of the display device 10 according to the present disclosure is not limited to the example shown in FIG. 4. For example, the display device 10 may be realized by a plurality of devices. Specifically, the display device 10 may be configured to include a display device (corresponding to at least the display unit 140) realized by an HMD or the like, and an information processing terminal (corresponding to at least the control unit 100) realized by a smartphone, tablet terminal, PC or the like. Furthermore, the tracking process by the control unit 100 described above may be performed by an external device.

[0047] Furthermore, each function of the control unit 100 may be realized by a server (e.g., information processing device 30) installed on the network, or may be realized by a dedicated terminal located in the same space as the user, a smartphone, a tablet terminal, a PC, or the like.

[0048] The display device 10 may also be a non-wearable device such as a smartphone or a tablet terminal.

[0049] <2-2. Example of the configuration of the controller 20> 5 is a block diagram showing an example of the configuration of the controller 20 according to this embodiment. As shown in FIG. 5, the controller 20 has a control unit 200, a communication unit 210, an operation input unit 220, a sensor unit 230, and a storage unit 240.

[0050] The control unit 200 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the controller 20 in accordance with various programs. The control unit 200 is realized by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 200 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0051] The control unit 200 according to this embodiment controls the transmission of information on user operations input from the operation input unit 220 and changes in user movement based on data sensed by the sensor unit 230 from the communication unit 210 to the controller 20. For example, the control unit 200 detects that a button provided on the controller 20 has been pressed (an example of a photographing operation), and outputs the information to the display device 10.

[0052] (Communication unit 210) The communication unit 210 is connected to the controller 20 via wired or wireless communication to transmit and receive data. The communication unit 210 can perform communication using, for example, wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or a mobile communication network (LTE (Long Term Evolution), 3G (third generation mobile communication system), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), etc.

[0053] (Operation input unit 220) The operation input unit 220 receives operation instructions from the user and outputs the operation content to the control unit 200. The operation input unit 220 may be, for example, a touch sensor, a pressure sensor, or a proximity sensor. Alternatively, the operation input unit 220 may be a physical component such as a button, a switch, or a lever.

[0054] (sensor unit 230) The sensor unit 230 has a function of sensing real space, such as the user or surrounding conditions. Specifically, the sensor unit 230 may be a sensor capable of detecting a total of nine axes, including a three-axis gyro sensor, a three-axis acceleration sensor, and a three-axis geomagnetic sensor. The sensor unit 230 may further include a camera, a microphone, and a biosensor (for detecting pulse, sweat, blood pressure, body temperature, etc.).

[0055] (Storage unit 240) The storage unit 240 is realized by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 200, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0056] Although the configuration of the controller 20 has been specifically described above, the configuration of the controller 20 according to the present disclosure is not limited to the example shown in Fig. 5. For example, the controller 20 may further include a display unit, a vibration unit, a light-emitting unit, etc. The controller 20 may provide feedback to the user from the display unit, vibration unit, light-emitting unit, etc. in accordance with a control signal received from the display device 10.

[0057] Furthermore, the controller 20 may be a controller dedicated to operating the VR content, or may be a general-purpose device such as a smartphone, a mobile phone terminal, or a wearable device such as a smart watch.

[0058] In the present embodiment, the system configuration includes the controller 20 as an example, but the present disclosure is not limited to this. The user can also communicate their intention to the system without using the controller 20, for example, by gestures with their hands or their entire body, or by voice. Gesture input and voice input may be detected by an outward-facing camera or microphone provided in the display device 10 (HMD). Furthermore, gesture input may be detected by a camera (not shown) provided around the user that captures the user's entire body, and transmitted to the display device 10 or the information processing device 30.

[0059] <2-3. Configuration example of information processing device 30> Next, a configuration example of the information processing device 30 will be described with reference to Fig. 6. The information processing device 30 may be a server provided on a network, or may be a dedicated terminal, a smartphone, a tablet terminal, a PC, or the like, located in the same space as the user.

[0060] 6 is a block diagram showing an example of the configuration of the information processing device 30 according to this embodiment. As shown in FIG. 6, the information processing device 30 includes a control unit 300, a communication unit 310, and a storage unit 320.

[0061] The control unit 300 functions as an arithmetic processing unit and a control device, and controls the overall operation of the information processing device 30 in accordance with various programs. The control unit 300 is realized by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 300 may also include a ROM (Read Only Memory) that stores the programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0062] The control unit 300 according to this embodiment also functions as a content viewing control unit 301 , an imaging processing unit 302 , a placement control unit 303 , a tagging unit 304 , and a sharing control unit 305 .

[0063] The content viewing control unit 301 has a function of controlling the viewing of VR content. Specifically, based on information about the VR content stored in the storage unit 320, the content viewing control unit 301 generates an image of a virtual space (a free viewpoint image) corresponding to a user's viewpoint (three-dimensional position coordinates) in a virtual space (a three-dimensional virtual space) constructed (defined) by the VR content. The content viewing control unit 301 can recognize the user's viewpoint in the virtual space from the user's position and posture (including the direction of the head) calculated based on, for example, sensing data output from the display device 10. The content viewing control unit 301 also transmits the generated image of the virtual space from the communication unit 310 to the display device 10, and causes the display unit 140 to display the image. The content viewing control unit 301 also generates audio information of the virtual space corresponding to the user's viewpoint based on the information about the VR content, and transmits the audio information to the display device 10.

[0064] By continuously performing the above processing, when the user moves forward, backward, left, or right in the real world (or inputs an operation command to move forward, backward, left, or right using the controller 20) or changes the viewpoint (for example, the direction of the head), the content viewing control unit 301 can move forward, backward, left, or right or change the viewpoint in the virtual space by the same amount of movement (or a corresponding amount of movement at a predetermined ratio). This increases the sense of immersion in the virtual space and provides a more realistic VR experience.

[0065] The image capturing processing unit 302 performs processing to capture an image of a subject in a virtual space. Specifically, when a user performs a shooting operation (for example, pressing a predetermined button on the controller 20), the image capturing processing unit 302 acquires (generates) an image of a predetermined angle of view (within the user's field of view) (a frame size specified in vertical and horizontal directions) based on the user's viewpoint in the virtual space. The acquired image may be a two-dimensional image or a three-dimensional image. The image capturing processing unit 302 then renders the image of the acquired angle of view 600 (for example, a 2D texture) on a photo object to generate a "captured image" (a virtual object that can be handled in the virtual space) in the virtual space. The image capturing processing unit 302 may also perform appropriate filtering processing during image capturing, such as excluding additional virtual objects that overlap with the subject.

[0066] The placement control unit 303 controls the placement of captured images generated by the image capture processing unit 302 in the virtual space. The placement may be performed according to a preset placement rule. For example, the placement control unit 303 places the captured images at a predetermined location relative to the position of the user viewing the virtual space. For example, the placement control unit 303 may place the captured images outside the user's field of view. Alternatively, the placement control unit 303 may place the captured images around the user, avoiding the main viewing target (which may be preset) of the content being viewed, such as the virtual space stage, the broadcaster (performer, etc.), or the user's direction of travel. In addition, the placement control unit 303 re-places the captured images placed in the virtual space to any position (manual placement position) specified by the user in response to a user operation. The placement control unit 303 can also automatically align manually placed captured images in a predetermined space in the virtual space.

[0067] The tagging unit 304 performs processing to add accompanying information (referred to as "tags" in this specification) related to the captured image to the captured image generated by the imaging processing unit 302. For example, the tagging unit 304 acquires information about the subject appearing in the captured image, the date and time of shooting, information about the VR content that captured the image, information about the time of shooting on the playback time axis of the VR content (playback start position), etc., and adds these to the captured image as tags. Such tags can be used when searching for captured images, when grouping and arranging a large number of captured images, when starting playback, etc.

[0068] The sharing control unit 305 controls the sharing of the captured image with others (display devices of other users) or externally with other systems, etc. Specifically, the sharing control unit 305 controls the transmission of a copy of the captured image selected by the user to a sharing destination (for example, a friend's display device, or a system that provides a service of printing on real objects such as T-shirts, mugs, or paper and delivering them to the user's home). The sharing control unit 305 also controls the reception of captured images shared by others and the storage unit 320 to store the images, and the placement of the received captured images in the virtual space being viewed by the user.

[0069] The above has described the various functions of the control unit 300. Note that the various functions described above are merely examples, and the present embodiment is not limited to these. For example, the control unit 300 can provide the user with various notifications and feedback through displays and sounds in the virtual space, vibrations of the display device 10 or the controller 20, etc.

[0070] (Communication unit 310) The communication unit 310 transmits and receives data to and from the display device 10 via a wired or wireless connection. The communication unit 310 is communicatively connected to the display device 10 via, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 3G (third generation mobile communication system), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), or the like.

[0071] (Storage unit 320) The storage unit 320 is realized by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 300, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0072] According to this embodiment, the storage unit 320 stores VR content. The storage unit 320 may also store user information (user name, ID, address, etc.).

[0073] Although the configuration of the information processing device 30 has been specifically described above, the configuration of the information processing device 30 according to the present disclosure is not limited to the example shown in FIG. 6 . For example, the information processing device 30 may be realized by a plurality of devices. Specifically, the control unit 300 may be provided in an information processing terminal realized by a smartphone, a tablet terminal, a PC, etc., or in a dedicated terminal located in the same space as the user, and the storage unit 320 may be provided in a server on a network. Furthermore, at least some of the functions of the control unit 300 may be executed by the display device 10. Furthermore, the information processing device 30 may be realized by the display device 10, an information processing terminal (a smartphone, a tablet terminal, a PC, etc.) or a dedicated terminal, and a server.

[0074] <<3. Operation Processing>> Next, the operation and processing of the information processing system according to this embodiment will be specifically described with reference to the drawings.

[0075] <3-1. Image Processing> 7 is a flowchart showing an example of the flow of imaging processing according to this embodiment. As shown in FIG. 7, first, the display device 10 starts viewing of VR content based on information acquired from the information processing device 30 (step S103). Specifically, the content viewing control unit 301 of the information processing device 30 controls viewing of the VR content, and images and sounds from the user's viewpoint in the virtual space are generated and transmitted to the display device 10.

[0076] Next, the imaging processing unit 302 of the information processing device 30 determines whether or not an imaging operation (operation input intending to perform photography) has been performed based on user operation information acquired from the display device 10 or the controller 20 (step S106). The imaging processing unit 302 may recognize, for example, pressing a predetermined button provided on the controller 20 as an imaging operation.

[0077] Next, if an image capturing operation is performed (step S106 / Yes), the image capturing processing unit 302 generates a 2D texture (image) of a specified vertical and horizontal frame size (i.e., the size of the angle of view 610) with a target point T at which the central axis S of the virtual camera 60 (i.e., the user's viewpoint) intersects orthogonally with an object in the virtual space (here, the subject 401C) as shown in FIG. 8 (step S109). The frame size (size of the angle of view 610) of the acquired image may be set in advance or may be arbitrarily set by the user. In this manner, the image capturing processing unit 302 captures an image of a part of the field of view of the user viewing the virtual space (part of the display range of the display unit 140). In FIG. 8, the target point T is set so that the three-dimensional position of the subject 401 to be captured is used in the tagging process described later. However, the present embodiment is not limited to this. The angle of view 610 is a frame of the specified vertical and horizontal size with a point on a plane perpendicular to the central axis S of the virtual camera 60 at which the plane intersects orthogonally with the central axis.

[0078] Next, the imaging processing unit 302 acquires a plurality of images in succession at a fixed frame cycle following acquisition of the above image, and stores them in the storage unit 320 (step S112). That is, as shown in FIG. 9, images (with the same angle of view) for n frames are acquired in succession at a fixed frame cycle from the shooting start frame (frame 0). These multiple images are acquired as insurance in case of shooting failure. That is, when saving a video as a still image, a delay in the shooting timing can cause a shooting error (the subject has their eyes closed, the subject's clothes are turned up, camera shake, etc.), so it is necessary to save multiple images. image By acquiring the number of images, it is possible to keep a successful image even if a photograph is not taken. The number of images to be acquired continuously can be set in advance.

[0079] Specifically, the image capturing processing unit 302 determines whether the image capturing has been successful or not, starting from the first acquired image (step S115). For example, the image capturing processing unit 302 detects whether the subject has closed eyes, the direction of the face, the facial expression, the clothes are turned up, and determines whether the image capturing has failed as set in advance.

[0080] Next, if it is not determined that the photographing has failed (step S118), the image can be said to be an image for which it has been determined that the photographing has been successful, and therefore the image capture processing unit 302 generates a photo object (a virtual object that can be handled in a virtual space) using the image for which it has been determined that the photographing has been successful as a final image, and the placement control unit 303 places the photo object in the virtual space (step S121). The photo object may be placed in the virtual space by the image capture processing unit 302. Alternatively, the photo object may be placed in a predetermined location in the virtual space. The placement process will be described next with reference to FIG. 10.

[0081] The above-described processes shown in steps S106 to S121 can be repeated until viewing of the VR content is completed (step S124). As described above, in this embodiment, in response to the shooting operation by the user, the process from obtaining the captured image to arranging it in the virtual space is performed in a series, which reduces the hassle of the operation and makes it possible to further enhance the entertainment value of the shooting experience in the virtual space.

[0082] The imaging process may be performed only during a predetermined time period during which imaging is permitted within the playback time of the VR content. Also, the imaging processing unit 302 may be configured to capture images of only predetermined places and objects (only permitted places and objects) in the virtual space.

[0083] The imaging process according to this embodiment has been described above. Note that the operational process shown in FIG. 7 is an example, and the present disclosure is not limited to the example shown in FIG. 7. For example, the present disclosure is not limited to the order of steps shown in FIG. 7. At least some of the steps may be processed in parallel, or may be processed in the reverse order. Also, all of the processes shown in FIG. 7 do not necessarily need to be executed. For example, images may be continuously acquired while determining whether or not the image capture is successful, and images may be continuously acquired until the image capture is successful.

[0084] 7 does not necessarily have to be performed by a single device. For example, the above description assumes that all of the processes shown in Fig. 7 are performed by the information processing device 30, but the present disclosure is not limited to this. For example, all of the processes shown in Fig. 7 may be performed by the display device 10, or may be performed by both the display device 10 and the information processing device 30.

[0085] <3-2. Arrangement of captured images> Next, the placement of the captured image (photo object) in the virtual space shown in step S121 above will be described. The placement in the virtual space may be randomly placed around the user's viewpoint, or may be placed in a predetermined location. The predetermined location is assumed to be, for example, a location that does not interfere with the viewing of the VR content, that is, a location outside the user's field of view, or a location that does not overlap with the main viewing target in the VR content (for example, the stage, the direction of travel, etc.). A specific description will be given below with reference to Figs. 10 to 12.

[0086] Fig. 10 is a flowchart showing an example of the flow of the arrangement process of the captured image according to this embodiment. Fig. 11 is a diagram for explaining the arrangement position of the captured image according to this embodiment.

[0087] As shown in FIGS. 10 and 11 , first, the placement control unit 303 sets a final placement position Q outside a horizontal viewing angle of approximately 110° from a viewing position P (user's viewpoint) in the virtual space to a viewing target of the VR content (for example, toward the stage, or may be a front direction defined within the VR content) (step S203). Here, "a horizontal viewing angle of approximately 110°" is an example of a range of "field of view" that takes into account a human's discriminative viewing angle, but the range (angle) of "field of view" in this embodiment is not limited to this. Furthermore, the definition of "field of view" is not limited to one that takes into account a discriminative viewing angle. Furthermore, the placement control unit 303 may set the final placement position Q outside the user's field of view but within the user's reach.

[0088] Next, the placement control unit 303 draws a movement trajectory of the captured image up to the final placement position Q (step S206). Here, as an example, a case where a movement trajectory of the captured image up to the final placement position is drawn will be described, but the present disclosure is not limited to this, and the captured image may be placed at the final placement position without drawing a movement trajectory.

[0089] 12 is a diagram illustrating a case where a movement trajectory of a captured image to an arrangement position is drawn according to this embodiment. As shown in FIG. 12, when a user performs a shooting operation on a virtual space image 430 displayed on the display unit 140 of the display device 10, an area with a field of view 600 appears from the image 430, and the captured image 510 moves toward a final arrangement position Q set outside the user's field of view (outside the virtual space image 430 displayed on the display unit 140). For example, if the final arrangement position Q is to the right of the user, an animation may be presented in which the captured image 510 gradually becomes smaller and fades out to the right of the image 430. This allows the user to intuitively grasp the direction in which the captured image is arranged.

[0090] The movement trajectory in the captured image may be given speed effects such as easy curves (for example, acceleration or deceleration at the start of movement, during movement, around curves, just before arrival, etc.) or a curved trajectory design.

[0091] Next, when the captured image reaches the final placement position Q (step S209 / Yes), the placement control unit 303 fixes the captured image at the final placement position Q and notifies the user of the direction of the placement position with a sound effect (step S212). For example, if the final placement position Q is to the right of the user, the sound effect coming from the right allows the user to intuitively understand that the captured image is placed on the right side, even though it is not visible in the user's field of view. Note that the placement control unit 303 may perform control to immediately place the captured image 510 (photo object) at the final placement position Q in response to the shooting operation, without displaying an animation of the movement process, and to output a sound effect (such as a shutter sound) so that it can be heard from the placement direction. While a sound notification is given here as an example, the present disclosure is not limited thereto. Notification that the captured image has been placed outside the field of view, the placement direction, the placement position, etc. may also be notified by a display, vibration, or the like. An example of a display is, for example, the display of an arrow indicating the placement direction.

[0092] Furthermore, "fixing the captured image" may mean fixing a relative position relative to the user's position in the virtual space, or fixing an absolute position in the virtual space. In the case of fixing a relative position, when the user moves in the virtual space, the captured image positioned next to the user also moves to follow the user's movement.

[0093] The user can also pick up the captured image 510 (see FIG. 3 ) placed at the final placement position Q in the virtual space and move it to another location or zoom in and out. Interaction with the captured image 510 (photo object) can be performed by operating or using gestures on the controller 20. Specifically, for example, the content viewing control unit 301 may display a virtual hand (a hand-shaped virtual object) in the virtual space that reflects the user's operation, hand shape, hand position, hand movement, and the like, and enable the user to grasp the captured image 510 with the virtual hand. In this case, the user can zoom in and out of the captured image 510 by, for example, pinching out or pinching in on the captured image 510 with the virtual hand, grasping both ends of the captured image 510 with the virtual hand to stretch or shrink the captured image 510, or fisting or unfisting the virtual hand on the captured image 510. The user can also rearrange (manually arrange) each captured image to a desired position to create an original viewing space. The rearrangement of captured images will be described later.

[0094] The arrangement control unit 303 may also display multiple captured images obtained by multiple shooting operations in an overlapping manner at the final arrangement position Q. For example, in the example shown in Fig. 3, the captured image 510 captured most recently may be considered to be arranged at the forefront. The arrangement control unit 303 may also display a certain number of captured images slightly shifted behind the captured image 510 so that the user can intuitively understand that multiple captured images are present at the final arrangement position Q.

[0095] The above describes the placement process according to this embodiment. Note that the operation process shown in Fig. 10 is an example, and the present disclosure is not limited to the example shown in Fig. 10. For example, the present disclosure is not limited to the order of steps shown in Fig. 10. At least some of the steps may be processed in parallel, or may be processed in the reverse order. Also, all of the processes shown in Fig. 10 do not necessarily have to be executed.

[0096] Furthermore, all of the processes shown in Fig. 10 do not necessarily have to be performed by a single device. For example, although the above description assumes that all of the processes shown in Fig. 10 are performed by the information processing device 30, the present disclosure is not limited to this. For example, all of the processes shown in Fig. 10 may be performed by the display device 10, or may be performed by the display device 10 and the information processing device 30.

[0097] <3-3. Tag processing of captured images> Next, a description will be given of how to assign tags to the captured images (photo objects) generated in step S121 and how to use the tags. Fig. 13 is a flowchart showing an example of the flow of tag processing for captured images according to this embodiment.

[0098] As shown in FIG. 13, first, the tagging unit 304 estimates information about the subject based on the position information or face recognition results of the subject appearing in the captured image and assigns the information to the captured image as a tag (accompanying information) (step S223). The subject's position information is, for example, three-dimensional position coordinates in a virtual space. For example, when an image is captured with the angle of view 610 shown in FIG. 8, the tagging unit 304 acquires the standing position Pc (Xc, Yc, Zc) of the subject 401C, which is close to the target point T, as the subject's position information. In the case of a concert or stage performance, the standing positions of performers may be predetermined. The tagging unit 304 can acquire information (subject information) about the subject appearing in the captured image, such as the subject's name, based on the previously generated information about the standing positions, the image capture time, and the subject's standing position. The tagging unit 304 may also analyze the subject's facial image and estimate the subject's identity based on facial recognition. Facial recognition information for performers in VR content may be prepared in advance.

[0099] Note that the tagging unit 304 basically tags information about a subject that appears at the center of a captured image, but the present disclosure is not limited to this. For example, the tagging unit 304 may tag information about all subjects that appear in a captured image. Furthermore, the tagging unit 304 may tag information about a subject that is in focus (in focus) among the subjects that appear in the captured image. Furthermore, the tagging unit 304 may tag information about the subject that appears largest among the subjects that appear in the captured image.

[0100] Next, the tagging unit 304 assigns information about the captured VR content and shooting time information (playback start position information) to the captured image as tags (step S226). The shooting time information is information about the time when the image was captured on the playback time axis of the captured VR content, and is treated as a playback start position.

[0101] The process of adding tags to captured images has been described above. Next, how to use the tags added to captured images will be described.

[0102] For example, when the user performs an operation to group and arrange the captured images (step S229 / Yes), the arrangement control unit 303 performs control to arrange the captured images for each subject (for example, for each subject 401A, subject 401C, and subject 401D) based on, for example, the subject information assigned to each captured image (step S232). The arrangement control unit 303 may arrange the multiple captured images for each group vertically or horizontally, or may arrange them overlapping each other. Furthermore, the location of the grouping arrangement in the virtual space is not particularly limited, and examples include a location that does not overlap with the main viewing target of the VR content (such as a stage or a direction of travel). More specifically, the location may be a location slightly away from the user, behind the user, near the user's hands, above the user (such as the ceiling or sky), etc.

[0103] Here, the case of grouping by subject has been described as an example, but the present embodiment is not limited to this. The placement control unit 303 may group and arrange the captured images by other factors based on the tags assigned to each captured image. For example, the placement control unit 303 may group and arrange the captured images by scene (stage change), by type of subject, by type of costume, by group of standing positions, etc. In other words, the scene when the image was captured, the type of subject, the standing position, etc. may also be assigned as tags to each captured image.

[0104] Furthermore, when a playback operation of the VR content from which the captured image was acquired is performed for the captured image (step S235 / Yes), the content viewing control unit 301 controls to start playback of the VR content based on the tag (playback start position information) of the selected captured image (step S238). This allows the user to play the VR content in the virtual space from the point in time when the image was captured.

[0105] The tag processing according to this embodiment has been described above. Note that the operational processing shown in Fig. 13 is an example, and the present disclosure is not limited to the example shown in Fig. 13. For example, the present disclosure is not limited to the order of steps shown in Fig. 13. At least some of the steps may be processed in parallel, or may be processed in the reverse order. Also, all of the processing shown in Fig. 13 does not necessarily have to be executed.

[0106] Furthermore, all of the processes shown in Fig. 13 do not necessarily have to be performed by a single device. For example, although the above description has been given assuming that all of the processes shown in Fig. 13 are performed by the information processing device 30, the present disclosure is not limited to this. For example, all of the processes shown in Fig. 13 may be performed by the display device 10, or may be performed by the display device 10 and the information processing device 30.

[0107] <3-4. Filtering process during imaging> The imaging processing unit 302 according to this embodiment may perform filtering processing to remove distracting effects and subjects other than the performers (for example, avatars of other users) when generating the captured image in step S121.

[0108] 14 is a diagram illustrating the filtering process of captured images according to this embodiment. As shown in FIG. 14, for example, in a captured image 520, at least a part of a target subject 401C (close to a target point T) or another subject 401B may be hidden by an effect image 403 (403A, 403B) or another user's avatar 405 (405A to 405C). Therefore, in this embodiment, it is possible to capture only subjects (performers, game characters, etc.) that are the main viewing targets in VR content.

[0109] 15 is a flowchart showing an example of the flow of filtering processing during image capture according to this embodiment. As shown in FIG. 15, when it is set to capture images of only the performers (step S243 / Yes), the image capture processing unit 302 generates a 2D texture (acquires a captured image) (within the range of the angle of view 600) while excluding CG objects other than the performers (effects, avatars of other users, etc.) (step S24 6 ). Here, each virtual object in the virtual space is assigned information such as whether it is a performer, an effect, or another user's avatar. The imaging processing unit 302 recognizes each virtual object reflected within the angle of view 600 and acquires an image after removing virtual objects other than the performer. Even if the subject is part of a stereoscopic image generated from live-action video, it is possible to remove distracting reflections by performing a process to remove virtual objects consisting of effects or other users' avatars. Note that although the term "performer" has been used here, this embodiment makes it possible to reliably capture the main viewing target in the VR content without being obstructed by additional virtual objects used for presentation purposes.

[0110] Furthermore, the image capture processing unit 302 may be configured to remove additional virtual objects for the purpose of effect only when they are located in front of the subject (performer, etc.), or may be configured to remove additional virtual objects only when they overlap the subject (performer, etc.).

[0111] The filtering process according to this embodiment has been described above. Note that the operational process shown in Fig. 15 is an example, and the present disclosure is not limited to the example shown in Fig. 15.

[0112] Furthermore, all of the processes shown in Fig. 15 do not necessarily have to be performed by a single device. For example, although the above description has been given assuming that all of the processes shown in Fig. 15 are performed by the information processing device 30, the present disclosure is not limited to this. For example, all of the processes shown in Fig. 15 may be performed by the display device 10, or may be performed by the display device 10 and the information processing device 30.

[0113] <3-5. Rearrangement of captured images> Furthermore, in this embodiment, the user can rearrange (manually rearrange) each captured image to a desired position to create an original viewing space.

[0114] More specifically, as shown in FIG. 3, the user can select a captured image 510 arranged at a predetermined position in response to a photographing operation using a virtual hand or the like, and move it to another location that is easier to view, such as in front of the user (toward the stage, etc.), or to a location of their choice. FIG. 16 shows an example in which the user manually rearranges multiple captured images. As shown in FIG. 16, the user can arrange the captured images 510a to 510g in a location that is easier for the user to view (for example, toward the front (stage)) in a layout (desired position and posture) as desired. That is, the arrangement control unit 303 of the information processing device 30 performs control to arrange the captured image selected by the user in an arbitrary position and posture designated by the user, based on the user operation detected by the controller 20, etc.

[0115] While watching the stage, users can experience creating their own space surrounded by their favorite photos. This experience will be different for each viewer and each broadcast, and it can be a new VR experience value that stimulates the desire to experience VR content repeatedly. In addition, by arranging multiple photos and capturing the original space they created (for example, saving the entire field of view as a still image or video) and showing it to other users, it can stimulate the motivation for competition among fans, and is expected to contribute to expanding the scale of the business.

[0116] Note that manually arranged captured images may be fixed at absolute positions in the virtual space, or may be fixed at positions relative to the user's position. In the case of relative arrangement, when the user moves in the virtual space, the captured images 510a to 510g arranged around the user also move to follow the user's movement.

[0117] Furthermore, in this embodiment, when the number of manually rearranged captured images around the user exceeds an upper limit or when the area occupied by the captured images exceeds a predetermined value, the manually rearranged captured images can be automatically aligned (automatically rearranged) at a location a short distance away from the user. The alignment may be vertically or horizontally aligned in the virtual space, or may be aligned in the depth direction. Such an alignment group is hereinafter referred to as an "aligned shelf" in this specification. Automatic rearrangement (alignment) may be performed automatically when a predetermined condition is met, such as when the number of manually rearranged captured images exceeds a predetermined number, or the user may instruct the automatic rearrangement (alignment) to be performed.

[0118] Fig. 17 is a diagram illustrating an example of automatic rearrangement (alignment) according to this embodiment. As shown in Fig. 17, the arrangement control unit 303 vertically arranges (aligns) a plurality of captured images that have been manually arranged around the user, for example, at a position to the left of the user at a certain distance in the direction of the stage, to create an alignment shelf 710a. The alignment shelf 710a is formed by vertically aligning the captured images 510a to 510f that have been manually arranged around the user.

[0119] A plurality of arranging shelves 710 may be created. For example, the arrangement control unit 303 may generate a new arranging shelf 710 each time automatic rearrangement (arrangement) is performed. In the example shown in FIG. 17, n arranging shelves 710 are created. Also, an arranging shelf 710 may be prepared for each sorting group. A sorting group is, for example, a group based on the above-mentioned tag. For example, the arrangement control unit 303 may store captured images on each arranging shelf 710 for each subject shown in the captured image (such as an arranging shelf for subject A, an arranging shelf for subject B, etc.).

[0120] Furthermore, the area (size) of the sorting shelf 710 may be set in advance, and the arrangement control unit 303 may reduce the size of each captured image when the captured images do not all fit on the sorting shelf, and arrange (store) the captured images. Alternatively, when the captured images do not all fit on the sorting shelf, the arrangement control unit 303 may newly generate the next sorting shelf and arrange (store) the captured images on it.

[0121] Furthermore, the placement location of the sorting shelf 710 is not particularly limited, but it is preferable that the sorting shelf 710 is placed, for example, so as not to overlap the stage and at a distance that is somewhat far from the user but still ensures the visibility of each captured image. The user can also manually pick up any captured image from the sorting shelf 710 and arrange it there. If the sorting shelf 710 is placed out of the user's reach, the user can use an operation item such as a laser pointer in the virtual space to pick up any captured image from the sorting shelf 710, move the sorting shelf 710, and perform other operations.

[0122] Furthermore, when one or more manually arranged captured images are automatically arranged on the arrangement shelf, the arrangement control unit 303 may display a trajectory (movement animation) of each captured image moving to the arrangement shelf. Such a trajectory may be given a speed effect such as an easy curve or a curved trajectory design.

[0123] The arrangement control unit 303 can also store the position and orientation of each captured image at the time of manual arrangement, and return the multiple captured images arranged on the arrangement shelf to the manual arrangement state again.

[0124] The operational processing relating to each rearrangement of the captured image explained above will be explained below with reference to FIG.

[0125] Fig. 18 is a flowchart showing an example of the flow of the captured image rearrangement process according to this embodiment. As shown in Fig. 18, first, when a manual arrangement operation is performed by the user (step S303 / Yes), the arrangement control unit 303 moves the selected captured image to a manual paste position designated by the user (step S306). The user, for example, uses the controller 20 to perform an operation of grabbing (selecting) the captured image 510 (see Fig. 3 ) that was arranged at a predetermined position immediately after capture, and moves the hand holding the controller 20 in the real world so as to move the captured image 510 to an arbitrary position in the virtual space while holding it. The arrangement control unit 303 controls the position and orientation of the captured image 510 in accordance with the user operation detected by the controller 20 (sensing data such as a button operation, acceleration, etc.).

[0126] The processes shown in steps S303 to S306 above can be repeated until the upper limit of manual placement is exceeded (step S309). That is, the user can perform an operation to paste (move) a plurality of captured images in order to any position.

[0127] Next, if the upper limit of manual placement is exceeded (step S309 / Yes), the placement control unit 303 stores the manually placed coordinates of all manually placed captured images in the storage unit 320 (step S312). Each captured image is assigned an image ID, and the placement control unit 303 stores the manually placed coordinates in association with the image ID. The upper limit of manual placement may be the upper limit number of images or the upper limit occupancy rate (e.g., the ratio of manually placed captured images to the user's surrounding area). The manually placed coordinates include not only the three-dimensional position in the virtual space but also the placement orientation (tilt, angle) of each captured image. The placement control unit 303 may also store the zoom status of each captured image. In this way, the placement control unit 303 can store the manual placement status of multiple captured images in association with the image ID. The placement control unit 303 may store a group of image IDs consisting of the image IDs of each captured image, along with the storage date and time (the date and time when the image was manually placed) and a call number.

[0128] Next, the arrangement control unit 303 determines the automatic arrangement coordinates of each image and executes automatic arrangement (step S315). The automatic arrangement coordinates of each image are information indicating the arrangement destination of the image. Specifically, for example, they correspond to the address of the arrangement shelf. The arrangement control unit 303 assigns an address of the arrangement shelf to each captured image. For example, the arrangement control unit 303 assigns addresses of the arrangement shelf to each captured image in order, starting from the first address of arrangement shelf A (the topmost arrangement location in the case of an arrangement shelf where images are arranged vertically). When all addresses are filled, the arrangement control unit 303 similarly assigns addresses from the first address of the next arrangement shelf B in order. Note that if multiple arrangement shelves are prepared for each sorting group, the arrangement control unit 303 assigns an empty address of the corresponding arrangement shelf based on the tag of each captured image. The arrangement control unit 303 moves each captured image to the assigned address (executes automatic arrangement). Furthermore, each automatic arrangement coordinate is stored in association with each image ID.

[0129] The above describes the automatic alignment of manually arranged photographed images on the alignment shelf. The photographed images that have been automatically aligned in this manner can be returned to their manual alignment positions at any time based on the stored information.

[0130] Specifically, when the user performs a manual placement call operation (step S318 / Yes), the placement control unit 303 controls the movement of each captured image from the sorting shelf to a manual pasting position based on the manual placement coordinates of the target captured image group (step S321). The manual placement can be called by specifying the call number or the date and time of manual placement. The user may specify the call number from, for example, a list of manual placement call numbers.

[0131] The above has described the captured image rearrangement process according to this embodiment. Note that the operational process shown in Fig. 18 is an example, and the present disclosure is not limited to the example shown in Fig. 18.

[0132] Furthermore, all of the processes shown in Fig. 18 do not necessarily have to be performed by a single device. For example, although the above description has been given assuming that all of the processes shown in Fig. 18 are performed by the information processing device 30, the present disclosure is not limited to this. For example, all of the processes shown in Fig. 18 may be performed by the display device 10, or may be performed by the display device 10 and the information processing device 30.

[0133] <3-6. Sharing captured images> In addition, in this embodiment, it is possible to share captured images with the outside. For example, a user can select any captured image and perform a predetermined sharing operation to share the captured image (send a copy of the captured image) with other users who are watching the VR content together. The user can arbitrarily select the sharing destination. As an example, the sharing destination may be other users who are watching the VR content together, or other users who are registered as friends. When the VR content is distributed in real time, it is also possible to participate and watch it together with friends. In this case, the friend's avatar is placed in the virtual space, and the user can converse (by voice or chat) with the friend, share captured images, etc.

[0134] The sharing operation may be, for example, as shown in FIG. 19 , an operation in which a captured image 510g to be shared is grasped with a virtual hand 820 and slid, touched, or inserted into a notch provided in a predetermined shared object 810 (virtual object). When such an operation is performed, the sharing control unit 305 may display a screen for selecting a sharing destination and allow the user to select a sharing destination. Alternatively, the sharing control unit 305 may automatically determine friends participating together (other users registered as friends who are simultaneously viewing the same VR content) as sharing destinations. The sharing control unit 305 places and presents the shared captured image at a predetermined position in the virtual space of the sharing destination user. The sharing destination user may be notified of the presentation of the shared captured image by sound or display. Note that friends participating together are not limited to friends viewing the same VR content on an HMD, but may also be friends viewing the same VR content on a smartphone, tablet, or the like.

[0135] Furthermore, the captured images may be shared with a system (another server, not shown) that provides users with various services using the captured images. For example, there may be a service that creates software or hardware products using the captured images, such as creating an album from the captured images or creating mugs, T-shirts, paper, etc. with the captured images printed on them. Such services may be available for a fee.

[0136] Furthermore, "external" in sharing captured images externally means outside the display device 10, and can include all devices other than the display device 10. Therefore, in this system, it is also possible to share (transfer) captured images to, for example, a user's smartphone, tablet terminal, etc. Furthermore, as external sharing, it is also possible to acquire images (place them in a virtual space) from the user's smartphone, tablet terminal, or cloud used by the user.

[0137] It will provide next-generation experiences that extend the experience in virtual space to the real world, such as the ability to share (transfer) images you have taken in virtual space with others, and create new value for VR experiences.

[0138] The above-described captured image sharing process will be described below with reference to FIG.

[0139] FIG. 20 is a flowchart showing an example of the flow of a captured image sharing process according to this embodiment. As shown in FIG. 20, first, when an external sharing operation is performed (step S403), the sharing control unit 305 transmits the captured image selected by the user in the sharing operation to a predetermined sharing destination (step S406). More specifically, the content viewing control unit 301 arranges the captured image to be shared in the virtual space of the shared user, so that the captured image can be essentially transmitted to the display device of the shared user. Alternatively, the sharing control unit 305 transmits the captured image to the display device of the shared user, and controls the display device of the shared user to arrange the captured image in the virtual space. The sharing destination may be a server of a system that provides various services using the captured image. The predetermined sharing destination may be a sharing destination arbitrarily selected by the user or a sharing destination set in advance. Alternatively, a plurality of shared objects 810 (virtual objects) to be used in the sharing operation may be prepared, and a sharing destination previously associated with the used shared object 810 may be set as the predetermined sharing destination.

[0140] On the other hand, if a captured image is shared from outside (step S409 / Yes), the sharing control unit 305 places the shared captured image at a predetermined position in the user's virtual space (step S412). The display device 10 may notify the user that a captured image has been presented by sound or display. The predetermined position of the shared captured image may be, for example, outside the user's field of view in the virtual space. That is, for example, the captured image captured by the user may be placed on the right side outside the user's field of view, and the shared captured image may be placed on the left side outside the user's field of view. The sharing control unit 305 may also indicate by an icon or the like who (or where) the captured image has been shared.

[0141] The above has described the captured image sharing process according to this embodiment. Note that the operational process shown in Fig. 20 is an example, and the present disclosure is not limited to the example shown in Fig. 20.

[0142] Furthermore, all of the processes shown in Fig. 20 do not necessarily have to be performed by a single device. For example, although the above description assumes that all of the processes shown in Fig. 20 are performed by the information processing device 30, the present disclosure is not limited to this. For example, all of the processes shown in Fig. 20 may be performed by the display device 10, or may be performed by both the display device 10 and the information processing device 30.

[0143] <<4. Other>> The information processing system according to this embodiment will be further explained below.

[0144] The imaging is not limited to capturing still images, but may also capture moving images. Specifically, for example, the imaging processing unit 302 can capture more realistic images, such as photos that move for only a few seconds. The duration of the movement (number of seconds, etc.) may be set in advance by the user. Furthermore, such moving photos (moving images) may also be accompanied by sound.

[0145] 7 and 8 may be performed for each frame when capturing moving images in a virtual space. The image capturing processing unit 302 may cut or darken frames determined to be failures, such as when the subject's eyes are closed or their clothes are turned up. The content distributor may be able to set in advance which frames are determined to be failures.

[0146] Furthermore, the information processing device 30 may set an upper limit on the size of the subject's face when capturing an image of the subject. That is, by limiting the zoom-in process, it becomes possible to prohibit taking a photo that is too close to the subject's face. Such a setting may be set appropriately by the content distributor.

[0147] 16, the content viewing control unit 301 may control the captured image pasted in the space to flutter (move) in conjunction with the movement of the content distributor (e.g., a performer on stage, a game character, or the like, who is the main viewing target of the content) in the virtual space. Furthermore, the content viewing control unit 301 may acquire bone information of the performer, etc., and may cause the captured image to vibrate or otherwise behave when the performer collides with the captured image pasted in the space (e.g., when a performer on stage reaches out and approaches the user). In this way, by moving the captured image pasted in the space in accordance with the movement of the performer, etc., the user can be given a sense of realism in the virtual space.

[0148] The placement control unit 303 may also have a function of recognizing the positions of performers and the like in the virtual space and supporting placement so that the captured images do not overlap with the performers and the like.

[0149] Furthermore, the images to be placed in the virtual space may not only be captured images obtained by taking pictures within the virtual space, but may also be various images that have already been saved in the memory unit 320 or other terminals, and may be laid out and placed in any desired position within the virtual space.

[0150] Furthermore, the captured image to be acquired is not limited to a 2D image (2D texture), but may be a 3D image (3D still image or moving image).

[0151] The content viewing control unit 301 may also display, in the virtual space, avatars of other users who are simultaneously viewing the same content. The content viewing control unit 301 may also present to the user information indicating the positions of the avatars of the user's friends (other users who are registered as friends). 0 It is also possible to identify the location in the virtual space where a captured image shared by a user was taken, and present the identified location to the user.

[0152] Furthermore, while the above embodiment has been described assuming VR content as an example, the present disclosure can also be applied to AR (Augmented Reality) and MR (Mixed Reality). For example, in the case of AR, the information processing device 30 acquires as an image (referred to as "capturing" in this specification) a portion of the display range including a subject (an image displayed as information on the virtual world that blends with the real world, such as a virtual object) of the virtual space superimposed on the real world (real space), and places the captured image as a photographic object in the virtual space superimposed on the real space. Furthermore, the display device 10 may be a see-through HMD, a smartphone, a tablet terminal, or the like.

[0153] The information processing device 30 also performs the various functions described above (acquisition of captured images, manual arrangement of captured images, alignment of captured images, sharing of captured images, etc.). etc. ) may each be a billable feature.

[0154] In the above-described embodiment, the captured images manually arranged are automatically arranged on the arrangement shelf, but the present disclosure is not limited to this. The arrangement control unit 303 may perform control to automatically arrange the captured images arranged after image capture on the arrangement shelf, for example, as shown in Fig. 3. Such automatic arrangement control may be triggered by a user operation, or may be performed when the number of arranged captured images reaches an upper limit.

[0155] <<5. Summary>> As described above, the information processing system according to the embodiment of the present disclosure can further enhance the entertainment value of the photography experience in virtual space.

[0156] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0157] For example, a computer program can be created to cause hardware such as a CPU, ROM, and RAM built into the above-described display device 10, controller 20, or information processing device 30 to perform the functions of the display device 10, controller 20, or information processing device 30. Also provided is a computer-readable storage medium storing the computer program.

[0158] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0159] The present technology can also be configured as follows. (1) An information processing device comprising: a control unit that, when detecting an imaging trigger for imaging a subject in a virtual space, generates an image of the subject and controls the generated image to be placed in the virtual space as a virtual object. (2) The information processing device according to (1), wherein the control unit arranges the captured images in the virtual space according to a predetermined arrangement rule. (3) The information processing device according to (2), wherein the control unit is placed at a predetermined location relative to a position of a user viewing the virtual space. (4) The information processing device according to (3), wherein the control unit performs control to place the device outside the field of view of the user. (5) The information processing device according to (3) or (4), wherein the control unit performs control to place the virtual space in a location that avoids overlapping with a predetermined viewing target. (6) The information processing device according to any one of (1) to (5), wherein the control unit captures an image of a part of a field of view of a user viewing the virtual space. (7) The field of view of the user viewing the virtual space is a display range of a display unit on which the virtual space is displayed, The information processing device according to (6), wherein the control unit acquires a part of a display range of the display unit as a captured image. (8) The information processing device according to any one of (1) to (7), wherein the control unit, upon detecting the imaging trigger, continuously acquires the captured images and determines whether each captured image is successfully captured. (9) The information processing device according to (8), wherein the control unit places, in the virtual space, a captured image determined to have been successfully captured among the continuously acquired captured images, as the virtual object. (10) The information processing device according to any one of (1) to (9), wherein the control unit notifies a user that the captured image has been arranged. (11) The information processing device according to any one of (1) to (10), wherein the control unit adds accompanying information related to the captured image to the captured image. (12) The information processing device described in (11), wherein the accompanying information includes at least information about the subject appearing in the captured image, or information about the time at which the image was captured on the playback time axis of the content provided in the virtual space. (13) The information processing device according to any one of (1) to (12), wherein the control unit acquires a captured image from which at least an additional virtual object overlapping with the subject has been excluded. (14) The information processing device according to any one of (1) to (13), wherein the control unit rearranges the arranged captured image in the virtual space at an arbitrary position and orientation in response to a user operation. (15) The information processing device according to any one of (1) to (14), wherein the control unit moves and aligns a plurality of captured images arranged at predetermined positions to another location in the virtual space. (16) The information processing device described in (15), wherein the control unit stores the arrangement state of the plurality of captured images before they are aligned, and controls the plurality of captured images that have been moved to the different location and aligned to return to the state before they were aligned. (17) The information processing device according to any one of (1) to (16), wherein the control unit performs control to share the captured image arranged in the virtual space with an external device. (18) The information processing device according to any one of (1) to (17), wherein the control unit performs a display indicating the imaging range on a display unit that displays the virtual space. (19) The processor: An information processing method including, when an imaging trigger for capturing an image of a subject in a virtual space is detected, generating an image of the subject and controlling the placement of the generated image in the virtual space as a virtual object. (20) a display device; A controller; an information processing device having a control unit that, when an imaging trigger for imaging a subject in a virtual space displayed on the display device is detected by the controller, generates a captured image of the subject and controls the generated captured image to be placed in the virtual space as a virtual object; An information processing system comprising: [Explanation of symbols]

[0160] 10 Display device 140 Display section 20 Controller 30 Information processing equipment 300 control section 301 Content viewing control unit 302 Imaging processing section 303 Placement control unit 304 Tagging section 305 Shared control section

Claims

1. Displaying a field of view object indicating an imaging range in a virtual space, When an imaging trigger for imaging a subject in a virtual space is acquired, a captured image of the subject is generated in accordance with the range of the angle of view object; An information processing device comprising: a control unit that performs control to place the generated captured image as a virtual object in the virtual space.

2. The control unit setting a position at which the captured image is to be placed in accordance with a position of the user in the virtual space; placing the captured image at the set position; The information processing device according to claim 1 .

3. The information processing device according to claim 2 , wherein the control unit sets a placement position outside the field of view of the user in accordance with a position and orientation of the user, and performs control to place the captured image.

4. The information processing device according to claim 3 , wherein the control unit sets a placement position at a location that does not overlap with an object of viewing of the user based on a position and an orientation of the user, and controls placement of the captured image.

5. An information processing device as described in Claim 2, wherein after arranging the captured image, the control unit fixes the arranged position as a relative position with respect to the user's position, or fixes the arranged position as an absolute position in the virtual space.

6. The information processing device according to claim 1 , wherein the control unit captures an image of a part of a field of view of the user viewing the virtual space.

7. The field of view of the user viewing the virtual space is a display range of a display unit on which the virtual space is displayed, The information processing device according to claim 6 , wherein the control unit acquires a part of a display range of the display unit as a captured image.

8. The information processing apparatus according to claim 1 , wherein the control unit, upon receiving the image capturing trigger, continuously captures the captured images and determines whether each captured image is successfully captured.

9. The information processing device according to claim 8 , wherein the control unit arranges, in the virtual space, a captured image determined to be successfully captured among the continuously acquired captured images, as the virtual object.

10. The information processing device according to claim 1 , wherein the control unit notifies a user that the captured image has been arranged.

11. The information processing device according to claim 1 , wherein the control unit adds accompanying information related to the captured image to the captured image.

12. The information processing device according to claim 11 , wherein the accompanying information includes at least information about the subject appearing in the captured image, or information about the time at which the image was captured on a playback time axis of content provided in the virtual space.

13. The information processing device according to claim 1 , wherein the control unit acquires a captured image from which at least an additional virtual object overlapping the subject has been excluded.

14. The information processing device according to claim 1 , wherein the control unit rearranges the arranged captured image in the virtual space at an arbitrary position and orientation in response to a user operation.

15. The information processing device according to claim 1 , wherein the control unit moves and aligns the plurality of captured images arranged at predetermined positions to another location in the virtual space.

16. The information processing device according to claim 15 , wherein the control unit stores an arrangement state of the plurality of captured images before the alignment, and performs control to return the plurality of captured images that have been moved to the different location and aligned to the state before the alignment.

17. The information processing device according to claim 1 , wherein the control unit controls sharing of the captured image arranged in the virtual space with an external device.

18. The processor: A field of view object indicating the imaging range is displayed in the virtual space. When an imaging trigger for imaging a subject in a virtual space is acquired, a captured image of the subject is generated in accordance with the range of the angle of view object; An information processing method including controlling the placement of the generated captured image as a virtual object in the virtual space.

19. a display device; A controller; an information processing device having a control unit that displays a field of view object indicating an imaging range in a virtual space displayed on the display device, and when an imaging trigger for imaging a subject in the virtual space is acquired from the controller, generates a captured image of the subject in accordance with the range of the field of view object, and controls the generated captured image to be placed in the virtual space as a virtual object; An information processing system comprising:

Images