Information processing device, information processing system, and information processing method

Abstract

To provide an information processing device that can improve the expressiveness of a synthetic stereo image visually recognized by a user.SOLUTION: An information processing device 10 generates a composite stereo image IM and includes a control unit 15 that acquires a stereo image IM1 from an imaging device 20, performs image processing on the acquired stereo image IM1 to determine a first object OB1 to be included in the composite stereo image IM, and then executes a synthesis process in which the first object OB1 determined on the basis of the image processing and a reference image IM2 are synthesized together to generate the composite stereo image IM.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, and an information processing method.

Background Art

[0002] Conventionally, techniques for generating a synthetic stereo image to be visually recognized by a user are known. For example, Patent Document 1 discloses an image processing apparatus that accurately displays a three-dimensional model of an object on a display device regardless of a change in the position and orientation of the display device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology including the image processing apparatus described in Patent Document 1, there is room for improvement in the expressiveness of the synthetic stereo image visually recognized by the user.

[0005] An object of the present disclosure is to provide an information processing apparatus, an information processing system, and an information processing method capable of improving the expressiveness of a synthetic stereo image visually recognized by a user.

Means for Solving the Problems

[0006] An information processing apparatus according to a first aspect for solving the above problems is an information processing apparatus that generates a synthetic stereo image, An imaging device acquires a stereo image, and performs image processing on the acquired stereo image. After executing the image processing for determining a first object to be included in the composite stereo image, a control unit performs a composite process of generating the composite stereo image by compositing the first object determined based on the image processing and a reference image with each other. It includes.

[0007] An information processing system according to a second aspect The above information processing device, The imaging device that acquires the stereo image and provides it to the information processing device, A display device that displays the composite stereo image generated by the information processing device in a visible state to the user. It includes.

[0008] An information processing method according to a third aspect An information processing method executed by an information processing device that generates a composite stereo image, Acquiring a stereo image from an imaging device; Performing image processing on the acquired stereo image, the image processing for determining a first object to be included in the composite stereo image; Performing a composite process of generating the composite stereo image by compositing the first object determined based on the image processing and a reference image with each other. It includes.

Advantages of the Invention

[0009] According to the information processing device, information processing system, and information processing method according to an embodiment of the present disclosure, it is possible to improve the expressiveness of the composite stereo image visually recognized by the user.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0011] The background and problems of the prior art will be explained in more detail.

[0012] Conventionally, in an online meeting tool, a background replacement function has been utilized. This function accurately extracts only the person area from a 2D image obtained from a monocular camera using object recognition AI (Artificial Intelligence) technology such as instance segmentation, and replaces other areas with a background image. By this function, a 2D image is presented on a display of a terminal device such as a PC (Personal Computer).

[0013] However, attempts to present a 3D video to a display device such as VR (Virtual Reality) goggles having an HMD (Head Mounted Display) by a similar function are not known.

[0014] The present disclosure provides an information processing apparatus, an information processing system, and an information processing method capable of improving the expressiveness of a synthetic stereo image visually recognized by a user. Hereinafter, an embodiment of the present disclosure will be mainly described with reference to the accompanying drawings. The following description also applies to an information processing system 1 having an information processing apparatus 10 and an information processing method executed by the information processing apparatus 10 to which the present disclosure is applied.

[0015] FIG. 1 is a schematic diagram schematically showing a part of the configuration of an information processing system 1 according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing an example of the configuration of the information processing system 1 in FIG. 1. With reference to FIGS. 1 and 2, an example of the configuration and functions of the information processing system 1 including the information processing apparatus 10 according to an embodiment of the present disclosure will be mainly described. The information processing system 1 includes, in addition to the information processing apparatus 10, an imaging apparatus 20 and a display apparatus 30 that are integrally configured with each other.

[0016] In FIGS. 1 and 2, for simplicity of explanation, only one combination of the imaging apparatus 20 and the display apparatus 30 is shown, but the information processing system 1 may have two or more combinations of the imaging apparatus 20 and the display apparatus 30. The information processing system 1 may have the same number of imaging apparatuses 20 and display apparatuses 30 as each other, or may have different numbers of imaging apparatuses 20 and display apparatuses 30 from each other. In FIG. 2, for simplicity of explanation, only one information processing apparatus 10 is shown, but the number of information processing apparatuses 10 included in the information processing system 1 may be two or more. The information processing apparatus 10 and the imaging apparatus 20 are communicably connected to each other. The information processing apparatus 10 and the display apparatus 30 are communicably connected to each other.

[0017] As an outline of an embodiment, the information processing apparatus 10 generates a synthetic stereo image IM. The information processing apparatus 10 acquires a stereo image IM1 from the imaging apparatus 20. In the present disclosure, the “stereo image IM1” includes, for example, a three-dimensional image configured based on two left and right images, similar to the principle by which a human visually recognizes an object three-dimensionally.

[0018] The information processing apparatus 10 executes image processing on the acquired stereo image IM1. The image processing is for determining a first object OB1 to be included in the composite stereo image IM. In the present disclosure, the "first object OB1" is, for example, any object such as a person, an animal, a plant, and an object included in the stereo image IM1, and is also included in the composite stereo image IM in the same manner as the stereo image IM1. In the example shown in FIG. 1, the first object OB1 includes, for example, a piano, a chair arranged in a set with the piano, and a performer sitting on the chair and playing the piano.

[0019] The information processing apparatus 10 executes a composite process of generating a composite stereo image IM by compositing the first object OB1 determined based on the above-described image processing and the reference image IM2 with each other. In the present disclosure, the "reference image IM2" includes, for example, a three-dimensional VR image, but is not limited thereto, and may include a three-dimensional real-shot image. In the example shown in FIG. 1, the reference image IM2 includes, for example, a three-dimensional virtual background image. The reference image IM2 may be prepared in advance by a host who holds, for example, a live performance, a dinner show, and a concert in which a piano performance by a performer is presented, and provided to the information processing apparatus 10 as information. The information processing apparatus 10 causes the display device 30 to display the generated composite stereo image IM so that the user can visually recognize it. In the present disclosure, the "image" includes, for example, a moving image, but is not limited thereto, and may include a still image.

[0020] For example, the user wears a device in which the imaging device 20 and the display device 30 are integrally configured on his / her head. At this time, the imaging device 20 provides the information processing apparatus 10 with a stereo image IM1 obtained by imaging any object located on the user's line of sight. The display device 30 displays the composite stereo image IM generated by the information processing apparatus 10 in a state visible to the user.

[0021] The information processing device 10 includes any general-purpose electronic device such as a smartphone, a tablet PC, a PC, and a smartwatch, which is used by a user wearing a device in which the imaging device 20 and the display device 30 are integrally configured. Without being limited to these, the information processing device 10 may include one or a plurality of server devices capable of communicating with each other, or may include other electronic devices dedicated to the information processing system 1.

[0022] The imaging device 20 includes a device that acquires a stereo image IM1 and provides it to the information processing device 10. The imaging device 20 includes, for example, a stereo camera. The imaging device 20 is integrally configured with the display device 30 worn on the user's head and is disposed with respect to the user's head. The imaging device 20 is attached to, for example, the front surface of the display device 30 and faces the user's line of sight.

[0023] The display device 30 includes a device that displays the synthetic stereo image IM generated by the information processing device 10 in a visible state to the user. The display device 30 includes VR goggles having an HMD and smart glasses. The display device 30 is worn on the user's head. The display device 30 is located on the user's head, for example, such that the HMD of the display device 30 is disposed in front of the user's eyes.

[0024] In an example of the processing by the information processing device 10, for the stereo image IM1 from the stereo camera integrally mounted on the display device 30 having an HMD, only the regions to be left in the synthetic stereo image IM are extracted based on an object recognition technology such as instance segmentation for both the left and right images. The other background regions are replaced with a reference image IM2 such as a VR image and integrated. As described above, the user wearing the display device 30 on the head can perform operations such as replacing the background of a person as the first object OB1 via the HMD. The information processing device 10 can be used in a wide variety of applications, such as new video effects in a live event and 3D online meetings beyond online meetings via a PC display.

[0025] While referring to FIG. 2, an example of the schematic configuration of each of the information processing apparatus 10, the imaging apparatus 20, and the display apparatus 30 included in the information processing system 1 will be mainly described.

[0026] The information processing apparatus 10 included in the information processing system 1 has a communication unit 11, a storage unit 12, an input unit 13, an output unit 14, and a control unit 15.

[0027] The communication unit 11 includes one or more communication interfaces. The communication interface corresponds to any wireless communication standard or wired communication standard for performing wireless communication or wired communication with each of the imaging apparatus 20 and the display apparatus 30. The wireless communication standard includes Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), and any other short-range wireless communication standard. The wired communication standard includes a communication standard such as USB (Universal Serial Bus). In one embodiment, the information processing apparatus 10 is communicably connected to each of the imaging apparatus 20 and the display apparatus 30 by communication via the communication unit 11.

[0028] In addition, the communication unit 11 may include one or more communication interfaces for connecting to a network including a mobile communication network and the Internet. The communication interface may correspond to, for example, a mobile communication standard such as 4G (4th Generation) and 5G (5th Generation), a wired LAN (Local Area Network) standard, or a wireless LAN standard. In one embodiment, the information processing apparatus 10 may be communicably connected to a network via the communication unit 11. The communication unit 11 may transmit and receive various information via the network.

[0029] The storage unit 12 includes storage modules such as HDD (Hard Disk Drive), SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), and RAM (Random Access Memory). The storage unit 12 stores information necessary to realize the operation of the information processing apparatus 10. The storage unit 12 stores information obtained by the operation of the information processing apparatus 10. For example, the storage unit 12 stores system programs, application programs, and various data acquired by any means such as communication.

[0030] The storage unit 12 may function as a main storage module, an auxiliary storage module, or a cache memory. The storage unit 12 is not limited to being built into the information processing apparatus 10, and may include an external storage module connected by a digital input / output port such as a USB.

[0031] The input unit 13 includes one or more input interfaces that detect user input and acquire input information based on the user's operation. The input interface includes physical keys, capacitive keys, a touch screen provided integrally with the display of the output unit 14, an imaging module such as a camera, and a microphone that receives voice input.

[0032] The output unit 14 includes one or more output interfaces that output information and provide it to the user. The output interface includes a display that visually outputs information as an image, a speaker that auditorily outputs information as sound, and a vibrator that tactilely outputs information as vibration.

[0033] The control unit 15 includes one or more processors. In the present disclosure, the "processor" is a general-purpose processor or a dedicated processor specialized for specific processing, but is not limited thereto. The control unit 15 includes, for example, a CPU (Central Processing Unit). The control unit 15 is communicably connected to each component constituting the information processing apparatus 10 and controls the operation of the entire information processing apparatus 10.

[0034] The configuration of the imaging device 20 included in the information processing system 1 will be mainly described. The imaging device 20 includes a communication unit 21, a storage unit 22, an imaging unit 23, and a control unit 24.

[0035] The communication unit 21 includes one or more communication interfaces. The communication interface corresponds to any wireless communication standard or wired communication standard for performing wireless communication or wired communication with the information processing apparatus 10. The wireless communication standard includes Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), and any other short-range wireless communication standard. The wired communication standard includes communication standards such as USB. In one embodiment, the imaging device 20 is communicably connected to the information processing apparatus 10 by communication via the communication unit 21.

[0036] In addition, the communication unit 21 may include one or more communication interfaces for connecting to a network including a mobile communication network and the Internet. The communication interface may correspond to, for example, a mobile communication standard such as 4G and 5G, a wired LAN standard, or a wireless LAN standard. In one embodiment, the imaging device 20 may be communicably connected to a network via the communication unit 21. The communication unit 21 may transmit and receive various information via the network.

[0037] The storage unit 22 includes storage modules such as HDD, SSD, EEPROM, ROM, and RAM. The storage unit 22 stores information necessary to realize the operation of the imaging device 20. The storage unit 22 stores information obtained by the operation of the imaging device 20. For example, the storage unit 22 stores various data acquired by any means such as system programs, application programs, and communication.

[0038] The storage unit 22 may function as a main storage module, an auxiliary storage module, or a cache memory. The storage unit 22 is not limited to being built into the imaging device 20, and may include an external storage module connected by a digital input / output port such as USB.

[0039] The imaging unit 23 includes one or more imaging modules capable of imaging any object existing around the imaging device 20 at a predetermined viewing angle. The imaging module includes, for example, two left and right camera modules for a stereo camera. The plurality of camera modules included in the imaging unit 23 are arranged in an appropriate positional relationship in the imaging device 20 so as to be able to image the periphery of the imaging device 20 and acquire a stereo image IM1. The imaging unit 23 may be configured such that the imaging device 20 functions as a front camera capable of imaging a front subject, or may be configured such that the imaging device 20 functions as an omnidirectional camera capable of imaging the entire periphery of the imaging device 20.

[0040] The control unit 24 includes one or more processors. The control unit 24 includes, for example, a CPU. The control unit 24 is communicably connected to each component constituting the imaging device 20 and controls the operation of the entire imaging device 20.

[0041] The configuration of the display device 30 included in the information processing system 1 will be mainly described. The display device 30 has a communication unit 31, a storage unit 32, an output unit 33, and a control unit 34.

[0042] The communication unit 31 includes one or more communication interfaces. The communication interface corresponds to any wireless communication standard or wired communication standard for performing wireless communication or wired communication with the information processing apparatus 10. The wireless communication standards include Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), and any other short-range wireless communication standards. The wired communication standards include communication standards such as USB. In one embodiment, the display device 30 is communicably connected to the information processing apparatus 10 by communication via the communication unit 31.

[0043] In addition, the communication unit 31 may include one or more communication interfaces for connecting to networks including a mobile communication network and the Internet. The communication interface may correspond to, for example, mobile communication standards such as 4G and 5G, a wired LAN standard, or a wireless LAN standard. In one embodiment, the display device 30 may be communicably connected to a network via the communication unit 31. The communication unit 31 may transmit and receive various information via the network.

[0044] The storage unit 32 includes storage modules such as HDD, SSD, EEPROM, ROM, and RAM. The storage unit 32 stores information necessary to realize the operation of the display device 30. The storage unit 32 stores information obtained by the operation of the display device 30. For example, the storage unit 32 stores system programs, application programs, and various data acquired by any means such as communication.

[0045] The storage unit 32 may function as a main storage module, an auxiliary storage module, or a cache memory. The storage unit 32 is not limited to being built into the display device 30, and may include an external storage module connected by a digital input / output port such as USB.

[0046] The output unit 33 includes one or more output interfaces that output information and provide it to the user. The output interfaces include a display that visually outputs information as an image, a speaker that aurally outputs information as sound, a vibrator that tactually outputs information as vibration, and the like. The display of the output unit 33 includes those that can three-dimensionally display the synthesized stereo image IM generated by the information processing apparatus 10 to the user. The display includes, for example, an HMD.

[0047] The control unit 34 includes one or more processors. The control unit 34 includes, for example, a CPU. The control unit 34 is communicably connected to each component constituting the display device 30 and controls the operation of the entire display device 30.

[0048] FIG. 3 is a sequence diagram for explaining a first example of an information processing method executed by the information processing system 1 of FIG. 1. With reference to FIG. 3, a first example of the information processing method executed by the information processing system 1 of FIG. 1 will be mainly described.

[0049] In step S100, the control unit 24 of the imaging device 20 images the periphery of the imaging device 20 using the imaging unit 23 and acquires a stereo image IM1.

[0050] In step S101, the control unit 24 of the imaging device 20 transmits the stereo image IM1 acquired in step S100 to the information processing apparatus 10 via the communication unit 21. The control unit 15 of the information processing apparatus 10 receives the stereo image IM1 acquired by the imaging device 20 in step S100 from the imaging device 20 via the communication unit 11. Thereby, the information processing apparatus 10 acquires the stereo image IM1 from the imaging device 20.

[0051] In step S102, the control unit 15 of the information processing apparatus 10 executes an identification process of associating identification information with each object included in the stereo image IM1 acquired from the imaging device 20 in step S101.

[0052] In the present disclosure, an "object" is any target such as a person, an animal, a plant, and an object included in, for example, the stereo image IM1. "Identification information" includes information that can uniquely identify one object by distinguishing objects from each other. The identification information includes, for example, a listing of any length such as numbers, letters, and symbols.

[0053] In step S103, the control unit 15 of the information processing apparatus 10 acquires input information based on a user input using the input unit 13.

[0054] The input information includes, for example, first input information from the user that designates the first object OB1 to be extracted from the stereo image IM1 acquired in step S101. For example, while checking the two-dimensional image corresponding to the stereo image IM1 displayed on the output unit 14 of the information processing apparatus 10, the user taps and selects the first object OB1 that the user desires to include in the composite stereo image IM.

[0055] In step S104, the control unit 15 of the information processing apparatus 10 executes a first extraction process for extracting the first object OB1 from the stereo image IM1 acquired in step S101. The control unit 15 executes the first extraction process according to the first input information from the user that designates the first object OB1 to be extracted from the stereo image IM1. The control unit 15 directly determines the first object OB1 to be included in the composite stereo image IM by the first extraction process.

[0056] In the present disclosure, "image processing" is for determining a first object OB1 to be included in the composite stereo image IM. In the first example shown in FIG. 3, the image processing includes, for example, the identification process in step S102 and the first extraction process in step S104. That is, the image processing includes an identification process based on an object recognition technology such as instance segmentation and a first extraction process according to the first input information from the user. The control unit 15 executes image processing on the stereo image IM1 acquired in step S101 in steps S102 and S104.

[0057] In addition to this, the image processing may further include an authentication process for authenticating a specific person included in the stereo image IM1 acquired in step S101. The control unit 15 may determine a specific first object OB1 by combining the identification process in step S102 and the authentication process. For example, the control unit 15 may execute both the identification process and the authentication process in parallel and collate the results of the processes with each other to determine a specific first object OB1.

[0058] In step S105, the control unit 15 of the information processing apparatus 10 executes a composite process of mutually combining the first object OB1 determined based on the above-described image processing and the reference image IM2 to generate a composite stereo image IM.

[0059] In step S106, the control unit 15 of the information processing apparatus 10 transmits the composite stereo image IM generated in step S105 to the display device 30 via the communication unit 11. The control unit 34 of the display device 30 receives the composite stereo image IM generated by the information processing apparatus 10 in step S105 from the information processing apparatus 10 via the communication unit 31. Thereby, the display device 30 acquires the composite stereo image IM from the information processing apparatus 10.

[0060] In step S107, the control unit 34 of the display device 30 causes the display of the output unit 33 to display the composite stereo image IM acquired from the information processing device 10 in step S106 in a visible state for the user wearing the display device 30.

[0061] FIG. 4 is a sequence diagram for explaining a second example of the information processing method executed by the information processing system 1 of FIG. 1. With reference to FIG. 4, a second example of the information processing method executed by the information processing system 1 of FIG. 1 will be mainly described.

[0062] In step S200, the control unit 24 of the imaging device 20 uses the imaging unit 23 to image the surroundings of the imaging device 20 and acquires a stereo image IM1.

[0063] In step S201, the control unit 24 of the imaging device 20 transmits the stereo image IM1 acquired in step S200 to the information processing device 10 via the communication unit 21. The control unit 15 of the information processing device 10 receives the stereo image IM1 acquired by the imaging device 20 in step S200 from the imaging device 20 via the communication unit 11. Thereby, the information processing device 10 acquires the stereo image IM1 from the imaging device 20.

[0064] In step S202, the control unit 15 of the information processing device 10 executes an identification process of associating identification information with each object included in the stereo image IM1 acquired from the imaging device 20 in step S201.

[0065] In step S203, the control unit 15 of the information processing device 10 acquires input information based on user input using the input unit 13.

[0066] The input information includes, for example, second input information from a user that specifies a second object OB2 to be extracted from the stereo image IM1 acquired in step S201. In the present disclosure, the "second object OB2" is, for example, any object such as a person, an animal, a plant, and an object included in the stereo image IM1, and is not included in the composite stereo image IM unlike the stereo image IM1. For example, while checking the two-dimensional image corresponding to the stereo image IM1 displayed on the output unit 14 of the information processing apparatus 10, the user taps and selects the second object OB2 that the user desires to be made non-displayed in the composite stereo image IM.

[0067] In step S204, the control unit 15 of the information processing apparatus 10 executes a second extraction process of extracting the second object OB2 not to be included in the composite stereo image IM from the stereo image IM1 acquired in step S201 and leaving only the first object OB1. The control unit 15 executes the second extraction process according to the second input information from the user that specifies the second object OB2 to be extracted from the stereo image IM1. The control unit 15 indirectly determines the first object OB1 to be included in the composite stereo image IM by the second extraction process.

[0068] In the second example shown in FIG. 4, the image processing includes, for example, the identification process in step S202 and the second extraction process in step S204. That is, the image processing includes an identification process based on an object recognition technique such as instance segmentation and a second extraction process according to the second input information from the user. The control unit 15 executes image processing on the stereo image IM1 acquired in step S201 in steps S202 and S204.

[0069] In addition to this, the image processing may further include an authentication process for authenticating a specific person included in the stereo image IM1 acquired in step S201. The control unit 15 may define a specific first object OB1 by combining the identification process in step S202 and the authentication process. For example, the control unit 15 may execute both the identification process and the authentication process in parallel, and determine a specific first object OB1 by collating the results of the processes with each other.

[0070] In step S205, the control unit 15 of the information processing apparatus 10 executes a synthesis process of synthesizing the first object OB1 determined based on the above image processing and the reference image IM2 with each other to generate a synthesized stereo image IM.

[0071] In the second example shown in FIG. 4, as shown in steps S202 to S205, after executing the image processing, the control unit 15 executes a synthesis process of synthesizing the first object OB1 and the reference image IM2 with each other to generate a synthesized stereo image IM.

[0072] In step S206, the control unit 15 of the information processing apparatus 10 transmits the synthesized stereo image IM generated in step S205 to the display device 30 via the communication unit 11. The control unit 34 of the display device 30 receives the synthesized stereo image IM generated by the information processing apparatus 10 in step S205 from the information processing apparatus 10 via the communication unit 31. Thereby, the display device 30 acquires the synthesized stereo image IM from the information processing apparatus 10.

[0073] In step S207, the control unit 34 of the display device 30 displays the synthesized stereo image IM acquired from the information processing apparatus 10 in step S206 on the display of the output unit 33 in a visible state for the user wearing the display device 30.

[0074] For the second object OB2 not to be included in the composite stereo image IM, the control unit 15 of the information processing apparatus 10 may execute two patterns of processing: a process of replacing it with a reference image IM2 or the like, and a process of deleting it. The control unit 15 may execute the deletion process based on the generation AI.

[0075] FIG. 5 is a first diagram for explaining an example of the processing by the information processing apparatus 10 in FIG. 2. FIG. 5 conceptually shows a state when image processing is executed in response to first input information from a user according to the processing flow by the control unit 15 of the information processing apparatus 10 shown in FIG. 3. With reference to FIG. 5, a basic series of flows of the processing by the information processing apparatus 10 in FIG. 2 will be mainly described.

[0076] The control unit 15 of the information processing apparatus 10 acquires a stereo image IM1 from the imaging device 20. The control unit 15 extracts the first object OB1 from the stereo image IM1 according to the first input information from the user that designates the first object OB1 to be extracted from the stereo image IM1. At this time, since, for example, two still images, left and right, are included in each frame of the stereo image IM1, the control unit 15 executes the first extraction process of the first object OB1 for each of the two still images.

[0077] The control unit 15 similarly executes the first extraction process of the first object OB1 for each of the two left and right still images for each frame of the stereo image IM1. The control unit 15 synthesizes the first object OB1 extracted from the stereo image IM1 with the reference image IM2 as a virtual image. Thereby, the control unit 15 generates a composite stereo image IM by synthesizing the first object OB1 determined based on the image processing including the first extraction process and the reference image IM2 with each other.

[0078] In the example shown in FIG. 5, the stereo image IM1 is obtained by the imaging device 20 imaging the state where a performer is performing a piano performance at a live event, a dinner show, a concert, etc. The control unit 15 identifies, from the stereo image IM1, for example, a piano, a chair arranged in a set with the piano, and a performer sitting on the chair and playing the piano based on the first input information, and extracts them as the first object OB1. The control unit 15 refers to the reference image IM2 as a virtual background image arranged around the first object OB1, and synthesizes the first object OB1 and the reference image IM2 with each other to generate a synthesized stereo image IM.

[0079] The control unit 15 causes the synthesized stereo image IM generated as described above to be displayed on the display of the output unit 33 of the display device 30 worn by the user on his or her head, and provides it to the user. The user visually recognizes the state where the performer is actually performing the piano in a virtual background that replaces the actual background of the venue where the performer is performing the piano. The user visually recognizes the synthesized stereo image IM in which the realistic image centered on the performer and the virtual image serving as the background are synthesized with each other via the display of the output unit 33 of the display device 30.

[0080] FIG. 6 is a second diagram for explaining an example of the processing by the information processing apparatus 10 in FIG. 2. FIG. 6 conceptually shows a state when determining the first object OB1 to be included in the synthesized stereo image IM according to the first input information from the user in accordance with the image processing by the control unit 15 of the information processing apparatus 10 shown in FIG. 3. With reference to FIG. 6, an example of the image processing by the information processing apparatus 10 in FIG. 2 will be mainly described.

[0081] For example, in image processing, in addition to the first object OB1 centered on the performer as shown in FIG. 5, the control unit 15 of the information processing apparatus 10 may also extract other objects such as food and drink located at the user's hand as the first object OB1 from the stereo image IM1. For example, when the user uses the information processing system 1 at a dinner show or the like, it is normal for the user to want to enjoy the performance of the performer while visually recognizing the food and drink at hand even while wearing the display device 30. Therefore, the user designates the performer-centered one and the food and drink at hand as the first object OB1 to be extracted from the stereo image IM1 and displayed in the composite stereo image IM while operating the input unit 13 of his or her own information processing apparatus 10. The control unit 15 may determine the first object OB1 to be combined into the composite stereo image IM according to the first input information at this time, extract it from the stereo image IM1, and generate the composite stereo image IM.

[0082] FIG. 7 is a third diagram for explaining an example of the processing by the information processing apparatus 10 in FIG. 2. FIG. 7 conceptually shows a state when the image displayed on the display of the output unit 33 of the display device 30 is switched by the control unit 15 of the information processing apparatus 10. An example of the processing by the information processing apparatus 10 in FIG. 2 will be mainly described with reference to FIG. 7.

[0083] In FIG. 7, the lower left image shows the composite stereo image IM in which the first object OB1 determined based on the above-described image processing by the control unit 15 of the information processing apparatus 10 and the reference image IM2 are combined with each other. For example, the lower left image shows the composite stereo image IM obtained by extracting the first object OB1 designated according to the first input information from the user from the stereo image IM1 and combining it with the reference image IM2.

[0084] In FIG. 7, the upper image shows a composite stereo image IM in which a first object OB1 determined based on the above-described image processing by the control unit 15 of the information processing apparatus 10 and a reference image IM2 are combined with each other. For example, the upper image shows a composite stereo image IM obtained by extracting a second object OB2 specified according to second input information from the user from the stereo image IM1 and combining the remaining first object OB1 with the reference image IM2.

[0085] Comparing the lower left image with the upper image, the stereo image IM1 acquired by the imaging apparatus 20 is, as an example, the sum of the first object OB1 of the person included in the lower left image and the first object OB1 of the background included in the upper image. At this time, in the lower left image obtained by the image processing based on the first input information, as a result of the person in the stereo image IM1 being specified and extracted by the user as the first object OB1, an actual person is included as the first object OB1 in the virtual reference image IM2. On the other hand, in the upper image obtained by the image processing based on the second input information, as a result of the person in the stereo image IM1 being specified and extracted by the user as the second object OB2, the actual person in the remaining first object OB1 of the actual background is replaced with the reference image IM2 which is a virtual avatar.

[0086] In FIG. 7, the lower right image shows a three-dimensional virtual image IM0 that combines a three-dimensional virtual background image and a virtual avatar, which is different from the composite stereo images IM such as the lower left image and the upper image generated by the control unit 15 of the information processing apparatus 10. Different from the two composite stereo images IM in FIG. 7, the virtual image IM0 does not include actual images of both the background and the person, but only includes virtual images.

[0087] The control unit 15 of the information processing apparatus 10 may instantaneously switch between the three images of the two synthesized stereo images IM and the virtual image IM0 shown in FIG. 7 and display them on the display of the output unit 33 of the display device 30. Alternatively, the control unit 15 may instantaneously switch between the four images including the two synthesized stereo images IM and the virtual image IM0 shown in FIG. 7, and the original stereo image IM1, and display them on the display of the output unit 33 of the display device 30. The control unit 15 is not limited to the above configuration, and may instantaneously switch between some of the four images including the two synthesized stereo images IM and the virtual image IM0 shown in FIG. 7, and the original stereo image IM1, and display them on the display of the output unit 33 of the display device 30.

[0088] In addition to the various processes described with reference to FIGS. 3 to 7, the control unit 15 of the information processing apparatus 10 may execute the following processes.

[0089] For example, the control unit 15 of the information processing apparatus 10 may generate the synthesized stereo image IM only in an image area corresponding to the line-of-sight information of the user who views the synthesized stereo image IM. At this time, in a state where the imaging device 20 is integrally configured with the display device 30, the imaging unit 23 of the imaging device 20 may further include an in-camera module capable of acquiring the line-of-sight information of the user wearing the display device 30. The control unit 15 of the information processing apparatus 10 may generate the synthesized stereo image IM only in an image area corresponding to a part of the corresponding viewing angle of the HMD of the output unit 33 of the display device 30, according to the line-of-sight information of the user acquired based on the in-camera module of the imaging unit 23 of the imaging device 20.

[0090] For example, when the control unit 15 of the information processing apparatus 10 determines that the field of view of the user viewing the synthesized stereo image IM with respect to the first object OB1 is blocked, it may supplement the image information of the blocked part based on the stereo image IM1 before the determined time point. Such determination processing by the control unit 15 may be executed based on the stereo image IM1 or based on the synthesized stereo image IM.

[0091] For example, when the user's view is blocked by another spectator different from the user at the venue where the performer is giving a performance, the control unit 15 of the information processing apparatus 10 may complement the image information of the blocked portion based on the past stereo image IM1. The control unit 15 may generate a composite stereo image IM as if the user's view were not blocked at all.

[0092] For example, the control unit 15 may obtain, from the past information of the stereo image IM1 as a video, the image information of the moment when the view is not blocked, which is before the time when the user's view is blocked. The control unit 15 may extract other spectators blocking the user's view as the second object OB2 from the stereo image IM1, and replace the extracted portion with an image corresponding to the past information.

[0093] The image used for replacement may be based on the image information of the stereo image IM1 after the acquisition of the stereo image IM1 is started along with the performance by the performer, or may be based on the image information of the stereo image IM1 that is pre-shot and registered by the organizer in advance before a live event or the like is held. If an object fixed in the user's view is blocked, the control unit 15 of the information processing apparatus 10 can accurately complement the image information by replacing the image as described above. In a more complex situation where the object blocked in the user's view changes over time, the control unit 15 may execute such a complement processing by an estimation process using a learning model or the like in machine learning.

[0094] For example, the control unit 15 of the information processing apparatus 10 may automatically present a virtual background image adapted to the geometry of the venue where the performer is giving a performance as the reference image IM2 in the composite stereo image IM. The control unit 15 may, for example, read information on markers arranged in the venue via the imaging device 20 to define a coordinate system of the venue including the x, y, and z axes and the origin. The control unit 15 may identify, by means of a technology such as AI, a location serving as a reference for coordinates such as the corners of the stage in the venue based on the stereo image IM1. The control unit 15 may calculate and set the coordinate system of the reference image IM2 as a virtual background image so as to conform to the identified information.

[0095] According to the above-described embodiment, it is possible to improve the expressiveness of the composite stereo image IM visually recognized by the user. The information processing apparatus 10 executes image processing for determining the first object OB1 to be included in the composite stereo image IM, and then executes a composite process of synthesizing the first object OB1 determined based on the image processing and the reference image IM2 with each other to generate the composite stereo image IM. Thereby, the information processing apparatus 10 can improve the expressiveness of the image visually recognized by the user, for example, in a live event and an online meeting, by means of the composite stereo image IM.

[0096] The information processing apparatus 10 is not limited to a person as the target determined as the first object OB1, and various objects other than a person can also be targets. Therefore, more diverse applications can be expected. For example, the information processing apparatus 10 can also realize four expression patterns in total of 2×2 by changing the assignment of a realistic image and a virtual image to each of the object determined as the first object OB1 and the background other than the object. The information processing apparatus 10 can realize a more advanced expression by appropriately switching a plurality of expression patterns.

[0097] The information processing device 10 executes image processing and then synthesizes the first object OB1 and the reference image IM2 with each other to generate a synthesized stereo image IM. For example, the information processing device 10 can also execute image processing as parallel processing simultaneously for each of the left and right two images included in the stereo image IM1. The information processing device 10 can also directly generate the synthesized stereo image IM by acquiring it from the imaging device 20 without importing the stereo image IM1 into 3D CG creation software or the like. As described above, the information processing device 10 can also execute the generation process of the synthesized stereo image IM more quickly and with higher precision.

[0098] The image processing includes a first extraction process of extracting the first object OB1 from the stereo image IM1. Thereby, the information processing device 10 can identify the first object OB1 to be included in the synthesized stereo image IM and directly extract it from the stereo image IM1. The information processing device 10 can directly determine the first object OB1 to be included in the synthesized stereo image IM by the first extraction process for the stereo image IM1, and can easily generate the synthesized stereo image IM.

[0099] The information processing device 10 executes the first extraction process according to the input information from the user who designates the first object OB1 to be extracted. Thereby, the user can directly designate, on the two-dimensional image displayed on the output unit 14 of the information processing device 10, the first object OB1 that the user desires to include in the synthesized stereo image IM by an input operation using the input unit 13 of the information processing device 10. The user can freely select the first object OB1 to be displayed in the synthesized stereo image IM, and can easily improve the expressiveness of the synthesized stereo image IM finally viewed by the user by himself / herself. For example, the user can also extract only the first object OB1 centered on the performer and immerse himself / herself in the performance of the performer through a more imposing synthesized stereo image IM. The information processing device 10 can control the first object OB1 to be included in the synthesized stereo image IM in real time according to the needs of individual users.

[0100] The image processing includes a second extraction process of extracting a second object OB2 that is not included in the composite stereo image IM from the stereo image IM1 and leaving only the first object OB1. Thereby, the information processing apparatus 10 can identify the second object OB2 that is not included in the composite stereo image IM and directly extract it from the stereo image IM1. The information processing apparatus 10 can indirectly determine the remaining first object OB1 to be included in the composite stereo image IM by the second extraction process of the second object OB2 with respect to the stereo image IM1.

[0101] The information processing apparatus 10 executes the second extraction process according to input information from a user who designates the second object OB2 to be extracted. Thereby, the user can directly specify, on the two-dimensional image displayed on the output unit 14 of the information processing apparatus 10, the second object OB2 that the user desires to make non-displayed in the composite stereo image IM, by an input operation using the input unit 13 of the information processing apparatus 10. The user can freely select the second object OB2 to be made non-displayed in the composite stereo image IM, and can easily improve the expressiveness of the composite stereo image IM finally viewed by the user by himself / herself. For example, the user can make other viewers blocking the performer on the composite stereo image IM non-displayed and display only the first object OB1 centered on the performer, and can also immerse in the performance of the performer through a more imposing composite stereo image IM. The information processing apparatus 10 can control the second object OB2 that is not included in the composite stereo image IM in real time according to the needs of individual users.

[0102] The image processing includes an identification process of associating identification information with each object included in the stereo image IM1. Thereby, the information processing apparatus 10 can easily identify the objects included in the stereo image IM1. The information processing apparatus 10 can, for example, also display the identification information corresponding to be associated with the first object OB1 displayed on the display of the output unit 33 of the display device 30. When the first object OB1 with new identification information is newly projected on the composite stereo image IM, the user can easily recognize it as a new target. On the other hand, when the first object OB1 with the same identification information as the first object OB1 that once went out of the range of the composite stereo image IM is projected again on the composite stereo image IM, it can be easily recognized as the same target by combining with the above-described authentication process.

[0103] The image processing further includes an authentication process of authenticating a specific person included in the stereo image IM1. The information processing apparatus 10 defines a specific first object OB1 by combining the identification process and the authentication process. Thereby, the information processing apparatus 10 can more accurately define the first object OB1 as a specific person in the image processing.

[0104] The information processing apparatus 10 generates the composite stereo image IM only in the image area according to the line-of-sight information of the user who views the composite stereo image IM. Thereby, the information processing apparatus 10 only needs to focus on generating the composite stereo image IM only in the area viewed by the user. Therefore, the information processing apparatus 10 can reduce the processing load in the generation process of the composite stereo image IM.

[0105] When the information processing apparatus 10 determines that the field of view of the user with respect to the first object OB1 that visually recognizes the synthesized stereo image IM is blocked, it complements the image information of the blocked portion based on the stereo image IM1 before the determined time point. As a result, the information processing apparatus 10 can reduce the adverse effects on the user, such as when the first object OB1 such as a performer on the synthesized stereo image IM is blocked by other audience members and cannot be seen. Therefore, the user can continuously visually recognize the first object OB1 such as a performer on the more impactful synthesized stereo image IM and can also immerse themselves in the performance of the performer.

[0106] In the information processing system 1, the imaging device 20 and the display device 30 are integrally configured. As a result, unlike the prior art that requires prior preparation in dedicated facilities such as a studio, the information processing system 1 can provide the synthesized stereo image IM to the user in real time without depending on the location and without prior preparation. Therefore, the convenience of the information processing system 1 for the user is improved.

[0107] As described above, the present disclosure relates to a technique of extracting only a specific region, for example, from a real-time stereo image IM1 and replacing the remaining portion with a virtual image, and its application. The present disclosure is new in that it displays a composite image of a real image and a virtual image, which utilizes object region extraction such as instance segmentation, on a display such as an HMD.

[0108] The present disclosure can also be used for video production in accordance with the performance of music in a concert or the like. The present disclosure can be applied to something like a 3D VR idol concert by using the background region as a real image as it is and replacing the person region with a virtual image. The present disclosure can be widely used in accordance with the needs of users who want to switch between real images and virtual images between people, specific objects, and other regions.

[0109] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, functions included in each component or each step, etc. can be rearranged so as not to be logically inconsistent, and a plurality of components or steps, etc. can be combined into one or divided.

[0110] For example, it is also possible to configure a general-purpose electronic device such as a smartphone or a computer to function as the information processing apparatus 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the information processing apparatus 10 according to the embodiment is stored in the memory of the electronic device, and the program is read and executed by the processor of the electronic device. Therefore, the present disclosure can also be realized as a program executable by a processor.

[0111] Alternatively, the present disclosure can also be realized as a non-transitory computer-readable medium storing a program executable by one or more processors for causing each function to be executed in the information processing apparatus 10 according to the embodiment. It should be understood that these are also included in the scope of the present disclosure.

[0112] For example, at least some of the processing operations executed in the information processing apparatus 10 in the above-described embodiment may be executed in at least one of the imaging apparatus 20 and the display apparatus 30. For example, instead of the information processing apparatus 10, the display apparatus 30 itself may execute the above-described series of processing operations related to the information processing apparatus 10. At least some of the processing operations executed in each of the imaging apparatus 20 and the display apparatus 30 may be executed in the information processing apparatus 10.

[0113] In the above embodiment, it has been described that the information processing apparatus 10 executes the first extraction process according to the input information from the user who designates the first object OB1 to be extracted, but the present invention is not limited to this. Instead of, or in addition to, the configuration in which the information processing apparatus 10 executes the first extraction process according to the input information from the user, the information processing apparatus 10 may automatically designate the first object OB1 to be extracted by itself and execute the first extraction process.

[0114] In the above embodiment, it has been described that the user who designates the first object OB1 to be extracted includes the first user wearing a device in which the imaging device 20 and the display device 30 are integrally configured, but the present invention is not limited to this. The user who designates the first object OB1 to be extracted may include, instead of, or in addition to, the first user, a second user such as a sponsor who holds a live performance, dinner show, concert, etc. in which a performance by a performer is presented, for the audience as the first user. The second user may, for example, designate the first object OB1 to be provided to the first user using his / her own terminal device or the like. Thereby, the information processing apparatus 10 may generate the composite stereo image IM including the first object OB1 designated by the second user. The display device 30 may display the composite stereo image IM generated by the information processing apparatus 10 for the first user.

[0115] For example, the first object OB1 to be included in the composite stereo image IM may be further individually designated in real time on the first user side using the information processing apparatus 10 in a state that is preliminarily set on the second user side in advance using his / her own terminal device or the like.

[0116] In the above embodiment, it has been described that the information processing apparatus 10 executes the second extraction process according to the input information from the user who designates the second object OB2 to be extracted, but the present invention is not limited to this. Instead of, or in addition to, the configuration in which the information processing apparatus 10 executes the second extraction process according to the input information from the user, the information processing apparatus 10 may automatically designate the second object OB2 to be extracted by itself and execute image processing.

[0117] In the above-described embodiment, the user who designates the second object OB2 to be extracted was described as including the first user wearing a device in which the imaging device 20 and the display device 30 are integrally configured, but the present invention is not limited thereto. The user who designates the second object OB2 to be extracted may include, instead of or in addition to the first user, a second user such as a promoter who holds a live performance, dinner show, concert, etc., in which a performance by a performer is presented, for the audience as the first user. The second user may, for example, designate the second object OB2 that he or she wishes to refrain from providing to the first user using his or her own terminal device or the like. Thereby, the information processing apparatus 10 may generate the composite stereo image IM in which the second object OB2 designated by the second user is not displayed. The display device 30 may display the composite stereo image IM generated by the information processing apparatus 10 for the first user.

[0118] For example, the second object OB2 to be made non-displayed in the composite stereo image IM may be further individually designated in real time on the first user side using the information processing apparatus 10 in a state that is preliminarily set on the second user side in advance using his or her own terminal device or the like.

[0119] In the above-described embodiment, the description mainly focused on the case where the information processing apparatus 10 executes only one of the first extraction process and the second extraction process, but the present invention is not limited thereto. The information processing apparatus 10 may execute a combination of the first extraction process and the second extraction process. For example, the information processing apparatus 10 may extract the first object OB1 from the stereo image IM1 and execute a second extraction process of further extracting the second object OB2 from the extracted first objects OB1. Conversely, the information processing apparatus 10 may extract the second object OB2 from the stereo image IM1 and execute a first extraction process of further extracting the first object OB1 from the extracted second objects OB2.

[0120] In the above embodiment, the image processing was described as including identification processing for associating identification information with each object included in the stereo image IM1, but it is not limited thereto. The information processing apparatus 10 does not have to execute such identification processing.

[0121] In the above embodiment, the image processing was described as further including authentication processing for authenticating a specific person included in the stereo image IM1, but it is not limited thereto. The information processing apparatus 10 does not have to execute such authentication processing. The information processing apparatus 10 may determine the first object OB1 based only on the identification processing without combining the identification processing and the authentication processing.

[0122] In the above embodiment, the information processing apparatus 10 was described as generating the composite stereo image IM only in an image area corresponding to the line-of-sight information of the user who views the composite stereo image IM, but it is not limited thereto. The information processing apparatus 10 does not have to execute the generation processing of the composite stereo image IM based on such line-of-sight information.

[0123] In the above embodiment, when the information processing apparatus 10 determines that the field of view of the user viewing the composite stereo image IM with respect to the first object OB1 is blocked, the information processing apparatus 10 was described as complementing the image information of the blocked portion based on the stereo image IM1 before the determined time point, but it is not limited thereto. The information processing apparatus 10 does not have to execute such complementing processing.

[0124] In the above embodiment, the information processing system 1 was described as including the information processing apparatus 10, the imaging apparatus 20, and the display apparatus 30, but it is not limited thereto. As described above, when the display apparatus 30 itself executes a series of processing operations related to the information processing apparatus 10 instead of the information processing apparatus 10, the information processing system 1 may include only the display apparatus 30 as the information processing apparatus 10 and the imaging apparatus 20. Or, the information processing system 1 may be realized as one integrated display apparatus in which the information processing apparatus 10, the imaging apparatus 20, and the display apparatus 30 are all integrated. The display apparatus as the information processing apparatus 10 may include a VR goggles having an HMD, smart glasses, and the like.

[0125] That is, the display device 30 may be configured as a device that integrally includes a dedicated digital circuit capable of realizing functions corresponding to the information processing device 10 and the imaging device 20. The processing corresponding to the various processes described above by the information processing device 10 may be completed inside the display device 30. In this way, by circuitizing specific processing, high speed is realized, and the real-time performance required in the information processing system 1 can be obtained more easily.

[0126] Conversely, when the information processing device 10 includes one or a plurality of server devices capable of communicating with each other instead of a general-purpose electronic device such as a smartphone, the information processing system 1 may further include a terminal device having a configuration corresponding to the input unit 13 and the output unit 14 described above. The terminal device may include any general-purpose electronic device such as a smartphone, a tablet PC, a PC, and a smartwatch used by a user wearing a device in which the imaging device 20 and the display device 30 are integrally configured. Without being limited thereto, the terminal device may include other electronic devices dedicated to the information processing system 1.

[0127] In the above embodiment, it has been described that the imaging device 20 and the display device 30 are integrally configured, but it is not limited thereto. The imaging device 20 and the display device 30 may be configured as separate devices separated from each other. At this time, the imaging device 20 may include, for example, a fixed installation stereo camera built in another device such as a PC. The information processing system 1 may display a composite stereo image IM including an arbitrary object on the other side of the meeting as the first object OB1 on the display device 30 in, for example, an online meeting using a PC.

[0128] In the above embodiment, it has been described that the information processing device 10 generates the composite stereo image IM, but it is not limited thereto. The information processing device 10 may apply a series of processing contents as shown in FIGS. 3 and 4 to generate a composite image not from the stereo image IM1 but from a single image acquired using a monocular camera.

[0129] Some embodiments of the present disclosure will be exemplified below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Appendix 1] An information processing apparatus for generating a synthesized stereo image, acquires a stereo image from an imaging device, and performs image processing on the acquired stereo image, and after executing the image processing for determining a first object to be included in the synthesized stereo image, performs a synthesis process of synthesizing the first object determined based on the image processing and a reference image with each other to generate the synthesized stereo image; a control unit, comprising an information processing apparatus. [Appendix 2] The information processing apparatus according to Appendix 1, wherein the image processing includes a first extraction process of extracting the first object from the stereo image. an information processing apparatus. [Appendix 3] The information processing apparatus according to Appendix 2, wherein the control unit executes the first extraction process in response to input information from a user specifying the first object to be extracted. an information processing apparatus. [Appendix 4] The information processing apparatus according to any one of Appendices 1 to 3, wherein the image processing includes a second extraction process of extracting a second object not to be included in the synthesized stereo image from the stereo image and leaving only the first object. an information processing apparatus. [Appendix 5] The information processing apparatus according to Appendix 4, wherein the control unit executes the second extraction process in response to input information from a user specifying the second object to be extracted. an information processing apparatus. [Appendix 6] The information processing apparatus according to any one of Appendices 1 to 5, The image processing includes identification processing for associating identification information with each object included in the stereo image. An information processing apparatus. [Appendix 7] The information processing apparatus according to Appendix 6, wherein the image processing further includes authentication processing for authenticating a specific person included in the stereo image, and the control unit determines a specific first object by combining the identification processing and the authentication processing. An information processing apparatus. [Appendix 8] The information processing apparatus according to any one of Appendices 1 to 7, wherein the control unit generates the synthesized stereo image only in an image area corresponding to the line-of-sight information of a user who visually recognizes the synthesized stereo image. An information processing apparatus. [Appendix 9] The information processing apparatus according to any one of Appendices 1 to 8, wherein when the control unit determines that the field of view of the user who visually recognizes the synthesized stereo image with respect to the first object is blocked, the control unit complements image information of the blocked portion based on the stereo image before the determined time point. An information processing apparatus. [Appendix 10] The information processing apparatus according to any one of Appendices 1 to 9, an imaging device that acquires the stereo image and provides it to the information processing apparatus, and a display device that displays the synthesized stereo image generated by the information processing apparatus in a visible state to the user. A configuration including: An information processing system. [Appendix 11] The information processing system according to Appendix 10, wherein the imaging device and the display device are integrally configured. An information processing system. [Appendix 12] An information processing method executed by an information processing apparatus that generates a synthesized stereo image, Obtaining a stereo image from an imaging device, Performing image processing on the obtained stereo image, and executing the image processing for determining a first object to be included in the composite stereo image, Performing a composite process of generating the composite stereo image by combining the first object determined based on the image processing with a reference image, Including An information processing method.

Explanation of Signs

[0130] 1 Information processing system 10 Information processing device 11 Communication unit 12 Storage unit 13 Input unit 14 Output unit 15 Control unit 20 Imaging device 21 Communication unit 22 Storage unit 23 Imaging unit 24 Control unit 30 Display device 31 Communication unit 32 Storage unit 33 Output unit 34 Control unit IM Composite stereo image IM0 Virtual image IM1 Stereo image IM2 Reference image OB1 First object OB2 Second object

Claims

1. An information processing apparatus for generating a synthesized stereo image, which acquires a stereo image from an imaging device, and performs image processing on the acquired stereo image, including image processing for determining a first object to be included in the synthesized stereo image, and then performs a synthesis process of synthesizing the first object determined based on the image processing and a reference image with each other to generate the synthesized stereo image; a control unit, comprising, an information processing apparatus.

2. The information processing apparatus according to claim 1, wherein the image processing includes a first extraction process of extracting the first object from the stereo image, an information processing apparatus.

3. The information processing apparatus according to claim 2, wherein the control unit executes the first extraction process according to input information from a user who designates the first object to be extracted, an information processing apparatus.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the image processing includes a second extraction process of extracting a second object not to be included in the synthesized stereo image from the stereo image and leaving only the first object, an information processing apparatus.

5. The information processing apparatus according to claim 4, wherein the control unit executes the second extraction process according to input information from a user who designates the second object to be extracted, an information processing apparatus.

6. The information processing apparatus according to any one of claims 1 to 3, wherein the image processing includes an identification process of associating identification information with each object included in the stereo image, an information processing apparatus.

7. The information processing apparatus according to claim 6, wherein the image processing further includes an authentication process of authenticating a specific person included in the stereo image, and the control unit determines a specific first object by combining the identification process and the authentication process, an information processing apparatus.

8. The information processing apparatus according to any one of claims 1 to 3, wherein the control unit generates the synthesized stereo image only in an image area according to the line-of-sight information of a user who visually recognizes the synthesized stereo image, an information processing apparatus.

9. The information processing apparatus according to any one of claims 1 to 3, When the control unit determines that the field of view of the user viewing the synthesized stereo image with respect to the first object is blocked, it complements the image information of the blocked portion based on the stereo image before the determined time point. Information processing apparatus.

10. The information processing apparatus according to any one of Claims 1 to 3, The imaging device that acquires the stereo image and provides it to the information processing apparatus, The display device that displays the synthesized stereo image generated by the information processing apparatus in a visible state to the user, Comprising, Information processing system.

11. The information processing system according to Claim 10, The imaging device and the display device are integrally configured, Information processing system.

12. An information processing method executed by an information processing apparatus that generates a synthesized stereo image, Acquiring a stereo image from an imaging device, Performing image processing on the acquired stereo image, and performing the image processing for determining a first object to be included in the synthesized stereo image, Performing a synthesis process of synthesizing the first object determined based on the image processing and a reference image with each other to generate the synthesized stereo image, Including, Information processing method.

Images