Information processing device, information communication system, information processing method, and program

Abstract

Facilitate interactive sessions. [Solution] An information processing device includes a session management unit that, after an interactive session is initiated between a first device that does not have a camera or microphone function and a second device that can send and receive video data and audio data, accepts a connection request from a third device that has a camera and microphone function and initiates a connection to the interactive session, and a video / audio transfer unit that, while the session management unit is connecting the third device to the interactive session, transmits video data and audio data acquired by the third device to the second device and transmits the video data to the first device.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information communication system, an information processing method, and a program. [Background technology]

[0002] Conventionally, as an interactive service using video and audio, facilities such as stores have provided remote customer service using dedicated terminals equipped with monitors, cameras, microphones, etc. (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-26728 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, STBs (set-top boxes) that connect to televisions to watch cable television and the like are known. STBs do not have camera or microphone functions. If an attempt is made to provide a similar interactive service, such as an interactive session, to a home that has an STB installed, it is conceivable to use the camera and microphone functions of a user device such as a smartphone. However, devices such as STBs cannot be directly connected to user devices, making it difficult to provide an interactive session using video and audio.

[0005] The present invention has been made in consideration of these points, and aims to provide an information processing device, an information communication system, an information processing method, and a program that can easily realize an interactive session even with a device that does not have a camera function or a microphone function. [Means for solving the problem]

[0006] (1) One aspect of the present invention is an information processing device that includes: a session management unit that, after an interactive session is initiated between a first device that does not have a camera function or a microphone function and a second device that can send and receive video data and audio data, accepts a connection request from a third device that has a camera function and a microphone function and initiates a connection to the interactive session; and a video / audio transfer unit that, while the session management unit is connecting the third device to the interactive session, transmits video data and audio data acquired by the third device to the second device and transmits the video data to the first device.

[0007] (2) Furthermore, one aspect of the present invention is that, in the information processing device of (1) described above, the information processing device further comprises an identification data generation unit that generates identifier data that can be displayed as video based on setting information including a session ID that identifies the interactive session, and the video / audio transfer unit transmits the identifier data to the first device.

[0008] (3) Also, one aspect of the present invention is that in the information processing device described above in (2), the video / audio transfer unit superimposes the identifier data on video data acquired by the second device and transmits it to the first device.

[0009] (4) Also, one aspect of the present invention is that in the information processing device described above in (3), the video / audio transfer unit stops transmitting the identifier data when the session management unit confirms that the third device has started connecting to the interactive session.

[0010] (5) Furthermore, one aspect of the present invention is that in the information processing device described above in (1), the video / audio transfer unit superimposes video data acquired by the third device on video data acquired by the second device and transmits the superimposed data to the first device.

[0011] (6) Furthermore, one aspect of the present invention is that in the information processing device described above in (5), the video / audio transfer unit does not transmit audio data acquired from the third device to the first device, but transmits it only to the second device.

[0012] (7) Furthermore, one aspect of the present invention is that, in the information processing device described above in (5), the information processing device further includes a setting storage unit that receives and stores setting information for the camera function and the microphone function from the third device, and the video / audio transfer unit transmits the setting information to the first device while the third device is connected to the interactive session.

[0013] (8) Also, one aspect of the present invention is that in the information processing device described above in (7), when the session management unit receives an update request from the first device to change the setting information of the camera function and the microphone function of the third device, it sends a setting change request for the camera function and the microphone function to the third device.

[0014] (9) Also, one aspect of the present invention is that in the information processing device described above in (8), when the setting storage unit receives updated setting information in response to the update request from the third device, it updates the stored setting information with the changed setting information, and the session management unit transmits the changed setting information to the first device.

[0015] (10) Furthermore, one aspect of the present invention is that in the information processing device described above in (7), the setting storage unit stores setting information regarding the camera function and microphone function of the third device, which are pre-set for each service used, each of which is associated with a unique session ID.

[0016] (11) In addition, according to one aspect of the present invention, in the information processing device of (10) above, the setting information further includes setting details relating to a background image for the video data of the third device.

[0017] (12) Another aspect of the present invention is an information and communication system including the information processing device according to any one of (1) to (10) above, the first device, the second device, and the third device.

[0018] (13) Another aspect of the present invention is an information processing method including the steps of: after an interactive session is initiated between a first device that does not have a camera function or a microphone function and a second device that can send and receive video data and audio data, accepting a connection request from a third device that has a camera function and a microphone function and initiating a connection to the interactive session; and, during the third device's connection to the interactive session, transmitting video data and audio data acquired by the third device to the second device and transmitting the video data to the first device.

[0019] (14) Another aspect of the present invention is a program that causes a computer to execute the steps of: after an interactive session is initiated between a first device that does not have a camera function or a microphone function and a second device that can send and receive video data and audio data, accepting a connection request from a third device that has a camera function and a microphone function and initiating a connection to the interactive session; and, during the third device's connection to the interactive session, transmitting video data and audio data acquired by the third device to the second device and transmitting the video data to the first device. [Effects of the Invention]

[0020] According to the present invention, it is possible to provide an information processing device, an information communication system, an information processing method, and a program that can easily realize an interactive session even with a device that does not have a camera function or a microphone function. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a configuration of an information communication system according to an embodiment of the present invention; [Figure 2] 10 is a flowchart illustrating processing performed by the information communication system according to the present embodiment. [Figure 3] 10A and 10B are diagrams showing an example of an image displayed on a television receiver connected to an STB in the information communication system according to the present embodiment. [Figure 4] 10 is a flowchart illustrating processing performed by the information communication system according to the present embodiment. [Figure 5] 10A and 10B are diagrams showing an example of an image displayed on a television set connected to an STB during an interactive session in the information communication system according to the present embodiment. [Figure 6] 1 is a block diagram showing an example of the internal configuration of an information processing apparatus according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0022] [Embodiment] Preferred embodiments of an information processing device, an information communication system, an information processing method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. In the drawings, identical or similar parts are designated by identical or similar reference numerals. Note that the present invention is not limited to these embodiments and includes various modifications and improvements. In other words, the components described below include those that can be easily imagined by a person skilled in the art or that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of components may be made in the present embodiment without departing from the spirit and scope of the present invention. In addition, in the drawings, the scale and number of each structure may differ from the scale and number of the actual structure to facilitate understanding of each structure.

[0023] [Information and Communication Systems] Fig. 1 is a block diagram showing the configuration of an information communication system according to this embodiment. First, an example of the configuration of the information communication system 1 will be described with reference to the figure. The information communication system 1 comprises an operator-side PC 10, a cloud system 20, an STB 30, and a user device 40. The operator-side PC 10, the cloud system 20, the STB 30, and the user device 40 are connected via a network NW such as the Internet. It can be said that the cloud system 20 is an information processing device that controls an interactive session between the operator-side PC 10, the STB 30, and the user device 40.

[0024] [Operator PC configuration] The operator's PC 10 is a computer used by an operator who provides various user services through an interactive session. An operator refers to a specialist in the user services (for example, a doctor, pharmacist, engineer, etc.). The operator's PC 10 is a device capable of transmitting and receiving video data and audio data to the STB 30 when an interactive session is started. The operator's PC 10 includes a video / audio output unit 11, a video / audio encoding unit 12, and an interactive control UI (user interface) 13. The video / audio output unit 11 outputs video to a monitor (not shown) and audio to a speaker (not shown). In other words, the operator's PC 10 can be said to be a second device capable of transmitting and receiving video data and audio data.

[0025] The video / audio encoding unit 12 encodes video captured by a camera (not shown) and audio collected by a microphone (not shown), and transmits the encoded video to the cloud system 20. The interaction control UI 13 is a user interface that controls an interaction session between the STB 30 and a user who uses the user device 40.

[0026] [Cloud system configuration] The cloud system 20 controls the connection / disconnection of an interactive session between the operator's PC 10 and the STB 30. The cloud system 20 includes a session management unit 21, a video / audio filter unit 22, a setting storage unit 23, a video / audio transfer unit 24, and a two-dimensional barcode generation unit 25.

[0027] The session management unit 21 manages an interactive session between the operator-side PC 10, the STB 30, and the user device 40. More specifically, the session management unit 21 first starts an interactive session between the operator-side PC 10 and the STB 30, and then controls the session management unit 21 to start a connection to the interactive session in response to a connection request from the user device 40. In other words, after an interactive session is started between the STB 30, which is a first device that does not have a camera function or a microphone function, and the operator-side PC 10, which is a second device that can send and receive video data and audio data, the session management unit 21 accepts a connection request from the user device 40, which is a third device that has a camera function and a microphone function, and starts a connection to the interactive session.

[0028] As a function of image processing, the video / audio filter unit 22 controls the virtual background for the image from the user device 40. The setting storage unit 23 receives and stores setting information (camera 42 on / off; microphone 41 on / off; virtual background setting) from the user device 40 for each interaction session. In other words, the setting storage unit 23 can be said to receive and store setting information for the camera function and microphone function from the user device 40, which is a third device.

[0029] Examples of setting information include information that is set by the user each time, and information that is preset for each service provided in an interaction session. For example, for a service that is used as a "problem consultation," the camera 42 may be turned off and the microphone 41 may be turned on; for a service that is used as a "health consultation," the camera 42 and microphone 41 may be turned on and a virtual background may be set (no image of the room); or for a service that is used as a "veterinary consultation," the camera 42 and microphone 41 may be turned on and no virtual background may be set (a picture of the room is taken because an animal is being photographed). The virtual background is a function that allows a virtual image to be set as the background of a video (such as a self-portrait of the user) captured by the camera 42 of the user device 40. The preset setting information is linked to a unique session ID to identify the service that is being used. That is, the setting storage unit 23 can be said to store setting information related to the camera function and microphone function of the user device 40, which is a third device, that are preset for each service that is used, each of which is associated with a unique session ID. The setting information can also be said to further include setting content related to a background image for video data of the user device 40, which is a third device.

[0030] The video / audio transfer unit 24 transfers video data (including the interactive control UI) and audio data from the operator's PC 10 to the STB 30 while an interactive session is connected between the operator's PC 10 and the STB 30 by the session management unit 21.

[0031] Furthermore, while the user device 40 is connected to the interaction session by the session management unit 21, the video and audio transfer unit 24 transfers the video data and audio data from the user device 40 to the operator's PC 10, and superimposes the video data from the user device 40 on the video from the operator's PC 10 and transfers it to the STB 30. In other words, the video and audio transfer unit 24 does not transfer the audio data from the user device 40 to the STB 30. In other words, while the session management unit 21 is connecting the user device 40, which is a third device, to the interaction session, the video and audio transfer unit 24 transmits the video data and audio data acquired by the user device 40, which is a third device, to the operator's PC 10, which is a second device, and transmits the video data to the STB 30, which is a first device. Furthermore, the video and audio transfer unit 24 superimposes the video data acquired by the user device 40, which is a third device, on the video data acquired by the operator's PC 10, which is a second device, and transmits it to the STB 30, which is a first device. Furthermore, the video / audio transfer unit 24 can be said to transmit audio data acquired from the user device 40, which is the third device, only to the operator's PC 10, which is the second device, without transmitting the audio data to the first device.

[0032] Furthermore, while the user device 40 is connected to the interaction session by the session management unit 21, the video and audio transfer unit 24 transfers the setting information (such as on / off of the camera 42 and on / off of the microphone 41 of the user device 40) stored in the setting storage unit 23 to the STB 30. In other words, it can be said that the video and audio transfer unit 24 transmits the setting information to the STB 30, which is the first device, while the user device 40, which is the third device, is connected to the interaction session.

[0033] When a user uses the STB 30 to request connection to an interactive session on the operator's PC 10, the two-dimensional barcode generation unit 25 generates a two-dimensional barcode for establishing the interactive session. An example of a two-dimensional barcode is a QR code (registered trademark). The two-dimensional barcode includes a session ID for identifying the interactive session, a connection destination URL (Uniform Resource Locator) of the cloud system 20, and the like. In other words, the two-dimensional barcode generation unit 25 can be said to be an identification data generation unit that generates identifier data that can be displayed as an image based on setting information including the session ID that identifies the interactive session.

[0034] Here, the timing for generating the two-dimensional barcode is assumed to be, first, when the process for generating the two-dimensional barcode is started upon receiving a connection request from the STB 30. In this case, the two-dimensional barcode generation unit 25 generates the two-dimensional barcode triggered by receiving the connection request from the STB 30. The timing for generating the two-dimensional barcode is assumed to be, second, when the process for generating the two-dimensional barcode is started based on the operation of the operator-side PC 10 after receiving the connection request from the STB 30. In this case, the operator initially requests a response similar to that made by operating the STB 30 remote control (for example, selecting an option on the screen), but when the operator determines that a telephone session is necessary, the process for generating the two-dimensional barcode may be started based on the operation of the operator-side PC 10.

[0035] The two-dimensional barcode is superimposed on the video data from the operator's PC 10 by the video / audio transfer unit 24 and transferred to the STB 30. In other words, the video / audio transfer unit 24 superimposes the two-dimensional barcode, which is identifier data, on the video data acquired by the operator's PC 10, which is the second device, and transmits it to the STB 30, which is the first device.

[0036] [STB configuration] STB 30 is a device that, when connected to a television receiver 60, receives operations from a user via a remote controller (hereinafter also referred to as a remote control) 50 and displays various programs and program guides on the television receiver 60. STB 30 comprises a video / audio setting operation unit 31, a video / audio output unit 32, a video / audio setting display unit 33, and a remote control operation reception unit 34. However, it does not basically have a camera function or a microphone function. In other words, STB 30 can be said to be a first device that does not have a camera function or a microphone function.

[0037] When the video / audio setting operation unit 31 receives updated setting information from the cloud system 20, it updates the UI displayed on the screen of the television receiver 60 according to the setting information. The video / audio output unit 32 outputs the video data and audio data transferred from the cloud system 20 to the externally connected television receiver 60. The video data output from the STB 30 includes video data from the operator's PC 10 and video data (picture-in-picture; sub-screen) from the user device 40. On the other hand, the audio data output from the STB 30 includes only audio data from the operator's PC 10.

[0038] The video / audio setting display unit 33 displays on the television receiver 60 a UI that allows the on / off setting of the camera 42 (video) and the microphone (audio) 41 of the user device 40 to be set by the remote control 50. The remote control operation receiving unit 34 receives operations from the remote control 50.

[0039] [User Device Configuration] The user device 40 is an electronic device, such as a smartphone or tablet terminal owned by a user, that has communication functions and also has camera and microphone functions. In other words, the user device 40 can be considered a third device having camera and microphone functions. The user device 40 includes a microphone 41, a camera 42, a video / audio encoding unit 43, a setting control unit 44, and a browser (application) 45. The microphone 41 and the camera 42 are installed on the user device. The microphone 41 collects the user's voice, and the camera 42 captures the user's video (e.g., a self-portrait). The video / audio encoding unit 43 encodes the audio collected by the microphone 41 and the video captured by the camera 42. The setting control unit 44 controls the on / off settings of the microphone 41 and the camera 42 in response to a setting change request from the cloud system 20. The browser 45 is an existing application for browsing web pages, and transmits video and audio data obtained by encoding the video and audio from the camera 42 and the microphone 41 to the cloud system 20. In this embodiment, it is possible to connect to an interactive session using an existing browser application, without installing a new application on a user device 40 such as a smartphone.

[0040] [Remote control and TV receiver] The remote control 50 is an input unit for instructing the operation of the STB 60 (channel selection, volume adjustment, etc.). In particular, the remote control 50 can be used to input selection, confirmation, cancellation, etc. to the UI displayed on the television set 60. The television set 60 is a display device for displaying video data received by the STB 30, and may also be a display capable of audio output. If the display does not have an audio output function (amplifier, speaker, etc.), a separate audio output device with audio output function can be connected to the STB 30. The television set 60, or the display and audio output device, may also be configured integrally with the STB 30.

[0041] In the configuration of the information and communication system 1 described above, it is possible to conduct an interactive session with the operator-side PC 10 using only a user device 40 such as a smartphone. However, during an interactive session, it is desirable for the conversation between the user and the operator to be conducted on a large screen so that the other person's facial expressions and other information can be clearly seen. However, the screen of a user device 40 such as a smartphone is relatively small, which is inconvenient. Furthermore, during an interactive session, it is very cumbersome to operate the user device 40 to select an option from the display screen or to set a function (for example, turning on / off the video (camera 42) or the audio (microphone 41)). Furthermore, the settings and operations required to connect a user device 40 such as a smartphone to the STB 30 via a wired or wireless connection are cumbersome.

[0042] Therefore, in this embodiment, as described above, an interactive session is established between the operator's PC 10 and the STB 30 using the screen of the television receiver 60 connected to the STB 30, and data and audio data are acquired through video on the user's side using the camera 42 and microphone 41 of the user device 40. At this time, the interactive session is enabled simply by reading the two-dimensional barcode displayed on the screen of the television receiver 60 with the camera 42 of the user device 40, without the need for any settings or operations for connecting the STB 30 and the user device 40 by wire or wirelessly.

[0043] [User device connection method] Next, connection of the user device 40 to an interaction session will be described with reference to Fig. 2. Fig. 2 is a flowchart for explaining the processing performed by the information communication system 1 according to this embodiment.

[0044] (Step S10) First, the user operates the remote control 50 to start a remote customer service application on the STB 30. The remote customer service application is an application for conducting an interactive session with the operator's PC 10, and is responsible for sending and receiving video data and audio data, and inputting operations on the remote control 50.

[0045] (Step S12) Next, when the remote customer service application is launched, the STB 30 connects to the operator's PC 10 via the cloud system 20. More specifically, the remote customer service application of the STB 30 requests the cloud system 20 to start an interactive session. The cloud system 20, having received the request, requests the operator's PC 10 to start an interactive session. The cloud system 20 then starts an interactive session between the STB 30 and the operator's PC 10. At this stage, the video data and audio data of the operator's PC 10 are output to the television set 60 connected to the STB 30 via the cloud system 20. In other words, this can be said to be a process or step in which an interactive session is started between the STB 30, which is a first device that does not have a camera function or a microphone function, and the operator's PC 10, which is a second device that can send and receive video data and audio data.

[0046] (Step 14) Next, in the cloud system 20, the two-dimensional barcode generating unit 25 generates a two-dimensional barcode including a session ID for identifying the interaction session and a connection destination URL of the cloud system 20.

[0047] (Step S16) In the cloud system 20, the video / audio transfer unit 24 superimposes the two-dimensional barcode on the video from the operator's PC 10 and transmits it to the STB 30. As a result, the two-dimensional barcode is displayed on the television receiver 60 connected to the STB 30 on the video from the operator's PC 10.

[0048] 3 is a diagram showing an example of an image displayed on a television receiver connected to an STB in the information communication system according to this embodiment. In the example shown in the figure, a two-dimensional barcode 71 is superimposed on the bottom right of an image 70 of the operator's PC 10 on the screen of the television receiver 60.

[0049] (Step S18) Returning to FIG. 2 , in the user device 40, the user uses the camera 42 to read the two-dimensional barcode 71 displayed on the television receiver 60. When the two-dimensional barcode 71 is read by the user device 40, the URL contained in the two-dimensional barcode 71 is read from the two-dimensional barcode 71 using the session ID and audio / video setting information as arguments, and the browser 45 is launched.

[0050] (Step S20) In the user device 40, the browser 45 connects to the video and audio transfer unit 24 of the cloud system 20 according to the read URL.

[0051] (Step S22) In the cloud system 20, when the session management unit 21 is accessed by the browser 45 of the user device 40, it starts connecting the user device 40 to the interactive session with the session ID specified by the two-dimensional barcode. In other words, after an interactive session is started between the STB 30, which is the first device, and the operator's PC 10, which is the second device, this can be said to be a process or step of accepting a connection request from the user device 40, which is a third device having a camera function and a microphone function, and starting a connection to the interactive session.

[0052] Thereafter, the video / audio transfer unit 24 stops superimposing the two-dimensional barcode on the video of the operation PC 10. That is, it erases the two-dimensional barcode displayed on the TV 6. In other words, when the session management unit 21 confirms that the user device 40, which is the third device, has started connecting to the interaction session, the video / audio transfer unit 24 stops transmitting the identifier data.

[0053] (Step S24) Next, in the user device 40, the setting control unit 44 saves setting information related to video and audio that is input in the user device 40 and transmitted to the operator-side PC 10. The setting information includes on / off settings for the camera 42, on / off settings for the microphone 41, and settings for the virtual background.

[0054] Through the above process, the user device 40 is connected to an interactive session with the operator's PC 10 via the cloud system 20.

[0055] [Behavior during an interactive session] Next, the operation during an interactive session will be described with reference to Fig. 4. Fig. 4 is a flowchart for explaining the processing performed by the information communication system 1 according to this embodiment.

[0056] (Step S30) In the user device 40, the browser 45 transmits the video from the camera 42 and the audio from the microphone 41 to the video / audio transfer unit 24 of the cloud system 20 based on the setting information specified by the two-dimensional barcode. Here, the setting information includes, for example, that the camera 42 is on, the microphone 41 is on, and the virtual background is number 12.

[0057] (Step S32) In the cloud system 20, the video and audio transfer unit 24 transmits the video data and audio data from the user device 40 to the operator's PC 10, and superimposes the video data (self-portrait) from the user device 40 on the video data from the operator's PC 10 and transmits it to the STB 30. At this time, the setting information stored in the setting storage unit 23 is also transmitted to the STB 30. In other words, this can be said to be a process or step of transmitting the video data and audio data acquired by the user device 40, which is a third device, to the operator's PC 10, which is a second device, and transmitting the video data to the STB 30, which is a first device, during connection to the interactive session by the user device 40, which is a third device.

[0058] (Step S34) In the STB 30, the video / audio output unit 32 displays the video data received from the cloud system 20 (video data obtained by superimposing video data from the user device 40 on video data from the operator's PC 10) on the television set 60, and also superimposes operable buttons on the UI and a virtual background on the video data from the user device 40 according to the setting information, and displays them on the television set 60. Note that the setting information transmitted to the STB 30 (specifically, camera On / Off, virtual background setting number, etc.) may be displayed on the screen in a format operable by the STB 30's remote control. Specifically, the format operable by the STB 30's remote control may be displayed as a UI component (a button that can be turned On / Off) using JavaScript or the like, and configured to be operable by the remote control.

[0059] FIG. 5 is a diagram showing an example of an image displayed on a television set connected to an STB during an interactive session in the information and communication system according to this embodiment. In the example shown in FIG. 5, an image 80 of an operator's PC 10 is displayed on the television set 60. The image 80 of the operator's PC 10 displays an image of an operator, i.e., a professional such as a doctor, pharmacist, or legal expert. The operator may wear a headset or use a microphone and speaker (not shown) depending on the situation. The UI at the lower right of the figure displays an image (user's self-portrait) 81 from the user device 40. Below the image (user's self-portrait) 81, operation buttons 82 and 83 are displayed that can be selected by operating the remote control 50 to turn the camera 42 on and off and the microphone 41 on and off according to the setting information. In addition, a virtual background 84, for example, number 12, is displayed as the background of the image (user's self-portrait) 81 according to the setting information.

[0060] The operation buttons 82 and 83 change contrast, color, and text (camera on / camera off or microphone on / off, etc.) depending on whether the camera 42 / microphone 41 is on or off. Alternatively, the operation buttons 82 and 83 may display icons depicting a camera or microphone (with a diagonal line through the icon when it is off).

[0061] During the interactive session, the user can view video and audio from the operator's PC 10 on the television set 60, and can also check the video (self-portrait) captured by the camera 42 of the user device 40 on the relatively large screen of the television set 60. The user can also check the on / off status of the microphone 41 and the camera 42 of the user device 40 by viewing the operation buttons 82 and 83 displayed on the UI on the television set 60.

[0062] At this time, the cloud system 20 does not transfer the voice data from the user device 40 to the STB 30. Therefore, the voice picked up by the microphone 41 of the user device 40, that is, the user's voice, is output only to the operator's PC 10, and is not output from the television receiver 60. Therefore, a user who is near the STB 30 does not feel uncomfortable because their own voice is output from the television receiver 60.

[0063] Furthermore, during an interactive session, the user uses the user device 40, such as a smartphone, by propping it up on a stand or the like on a table or desk. Therefore, the user can operate the remote control 50 of the STB 30 with their free hand while watching the screen of the television set 60, without looking at the screen or operating the keys of the user device 40. In other words, the user can operate the remote control 50 to turn the microphone 41 on / off, the camera 42 on / off, or update the setting state of the virtual background while gazing at the screen of the television set 60.

[0064] (Step S36) Returning to Fig. 4, in the STB 30, the remote control operation acceptance unit 34 determines whether operation buttons 82, 83 or virtual background 84 on the UI of the television set 60 have been selected by operating the remote control 50, and whether the video (camera 42) and audio (microphone 41) settings or the virtual background settings have been changed. If the video and audio or virtual background settings have been changed by operating the remote control 50 (YES in step S36), the process proceeds to step S38. On the other hand, if the video (camera) and audio (microphone) settings or the virtual background settings have not been changed (NO in step S36), the interactive session continues as is without updating the setting information.

[0065] (Step S38) If the video (camera 42) and audio (microphone 41) settings or the virtual background settings are changed by operating the remote control 50 (YES in step S36), the STB 30 sends a request to the cloud system 20 to update the setting information.

[0066] (Step S40) In the cloud system 20, when the session management unit 21 receives a request to update the setting information from the STB 30, it transmits a request to change the setting information to the user device 40. In other words, when the session management unit 21 receives an update request from the STB 30, which is the first device, to change the setting information of the camera function and microphone function of the user device 40, which is the third device, it can be said that the session management unit 21 transmits a request to change the settings of the camera function and microphone function to the user device 40, which is the third device.

[0067] (Step S42) In response to a change request from the cloud system 20, the user device 40 reflects the state of the microphone 41 on / off, the camera 42 on / off, or the virtual background according to the setting information (microphone 41 on / off, camera 42 on / off, or virtual background setting), and transmits the reflection result (updated setting information) to the cloud system 20.

[0068] (Step S44) In the cloud system 20, the setting storage unit 23 updates the stored setting information with the reflection result (updated setting information) from the user device 40. Thereafter, the session management unit 21 transmits the updated setting information to the STB 30. In other words, when the setting storage unit 23 receives updated setting information in response to the update request from the user device 40, which is the third device, it updates the stored setting information with the changed setting information, and the session management unit 21 transmits the changed setting information to the STB 30, which is the first device.

[0069] (Step S46) Upon receiving the updated setting information from the cloud system 20, the STB 30 updates the UI displayed on the television set 60. More specifically, the video / audio setting operation unit 31 updates the display form of the operation buttons 82 and 83 for controlling the on / off of the camera 42 and the microphone 41, or the virtual background 84, on the UI displayed on the television set 60, in accordance with the updated setting information. The interactive session then continues.

[0070] [Internal configuration] FIG. 6 is a block diagram showing an example of the internal configuration of an information processing device according to this embodiment. At least some of the functions of the information processing device can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses an input / output port via the bus 906. All or part of the functional units provided in the information processing device may be realized using hardware such as an ASIC, a PLD, or an FPGA. All or part of the functional units may be realized by a combination of software and hardware.

[0071] [Summary of the embodiment] According to the above-described embodiment, in the cloud system (information processing device) 20, after an interactive session is initiated between a first device that does not have a camera function or a microphone function and a second device that can transmit and receive video data and audio data, the session management unit 21 accepts a connection request from a third device that has a camera function and a microphone function and initiates a connection to the interactive session. In the cloud system (information processing device) 20, the video / audio transfer unit 24 transmits video data and audio data acquired by the third device to the second device and transmits video data to the first device during connection to the interactive session of the third device by the session management unit 21. By employing such a configuration, an interactive session using video and audio can be easily realized even with a first device that does not have a camera function or a microphone function.

[0072] Furthermore, according to the above-described embodiment, the cloud system (information processing device) 20 further includes an identification data generation unit that generates identifier data that can be displayed as video based on setting information including a session ID that identifies the interactive session. The video / audio transfer unit transmits the identifier data to the first device. By adopting such a configuration, an interactive session can be established simply by reading the identifier data displayed on the first device with the camera function of the third device, without performing any settings or operations for connecting the first device and the third device via a wired or wireless connection.

[0073] Furthermore, according to the above-described embodiment, the video / audio transfer unit superimposes the identifier data on the video data acquired by the second device and transmits the superimposed data to the first device. By adopting such a configuration, an interactive session can be established simply by reading the identifier data displayed on the first device using the camera function of the third device, without the need for any settings or operations for connecting the first device and the third device via a wired or wireless connection.

[0074] Furthermore, according to the above-described embodiment, the video and audio transfer unit stops transmitting the identifier data when the session management unit confirms that the third device has started connecting to the interactive session. By adopting such a configuration, a user in the vicinity of the first device can easily confirm that they have been connected to the interactive session.

[0075] Furthermore, according to the above-described embodiment, the video / audio transfer unit superimposes video data acquired by the third device on video data acquired by the second device and transmits the superimposed video data to the first device. By adopting such a configuration, the first device superimposes video data acquired by the third device on video data acquired by the second device, making it possible to easily realize an interactive session using video and audio even with a device that does not have a camera function or a microphone function.

[0076] Furthermore, according to the above-described embodiment, the video / audio transfer unit does not transmit audio data acquired from the third device to the first device, but transmits it only to the second device. By adopting such a configuration, the user's audio is output only to the second device, not from the first device, so that it is possible to prevent discomfort to users near the first device.

[0077] Furthermore, according to the above-described embodiment, the cloud system (information processing device) 20 further includes a setting storage unit that receives and stores setting information for the camera function and microphone function from the third device. The video / audio transfer unit transmits the setting information to the first device while the third device is connected to the interaction session. By adopting such a configuration, while connected to the interaction session, the first device can reflect the setting status of the camera function and microphone function of the third device in accordance with the received setting information.

[0078] Furthermore, according to the above-described embodiment, when the session management unit receives an update request from the first device to change the setting information of the camera function and microphone function of the third device, the session management unit transmits a setting change request for the camera function and microphone function to the third device. By adopting such a configuration, when the camera function and microphone function of the third device are changed on the first device, the third device can change the settings of the camera function and microphone function in response to the update request from the first device.

[0079] Furthermore, according to the above-described embodiment, when the setting storage unit receives updated setting information in response to an update request from the third device, it updates the stored setting information with the changed setting information. The session management unit transmits the changed setting information to the first device. By adopting such a configuration, when the settings of the camera function and microphone function are changed on the third device, the changes can be shared between the cloud system (information processing device) 20 and the first device.

[0080] Furthermore, according to the above-described embodiment, the setting storage unit stores setting information related to the camera function and microphone function of the third device, which are preset for each service to be used, each of which is associated with a unique session ID. By adopting such a configuration, since setting information for each service to be used is preset, it is possible to automatically set the camera function and microphone function of the third device according to the service to be used, thereby improving convenience.

[0081] Furthermore, according to the above-described embodiment, the setting information further includes setting details regarding a background image for the video data of the third device. By adopting such a configuration, it becomes possible to automatically set the background image for the video data of the third device depending on the service being used, thereby improving convenience.

[0082] Since the above-described embodiment makes it possible to easily realize an interactive session even with a device that does not have a camera or microphone function, it is possible to use the camera and microphone of a user's smartphone to execute an interactive session, which is a two-way service using video and audio, even with an existing device installed in the home, such as an STB that does not have a camera or microphone. Therefore, this embodiment can contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote inclusive and sustainable industrialization, and foster innovation."

[0083] In the above-described embodiment, STB30 was used as the device on the user side, but this is not limited to this. Any device that is capable of two-way communication, runs an application, and does not have a camera function or a microphone function may be used, for example, a television set (Android TV), a projector, a combination of a desktop PC and a display device (TV, monitor), or a device with a large screen in the home on which an application runs.

[0084] Note that the functions of the cloud system 20 may be implemented using electronic circuits as necessary. Furthermore, the functional units that implement the functions do not need to be included in a single device; the cloud system 20 may be configured with multiple devices. At least some of the functions of the cloud system 20 may be implemented using a computer. The computer includes a central processing unit, RAM, input / output ports, input / output devices, and a bus. The computer itself can be implemented using existing technology. The central processing unit executes instructions contained in a program loaded from RAM or the like. According to each instruction, the central processing unit writes data to RAM, reads data from RAM, and performs arithmetic and logical operations. RAM stores data and programs. Each element in RAM has an address and can be accessed using the address. Note that RAM stands for "random access memory." Input / output ports are ports through which the central processing unit exchanges data with external input / output devices. Input / output devices exchange data with the central processing unit via the input / output ports. A bus is a common communication path used within a computer. For example, the central processing unit reads and writes data from RAM via the bus. Also, for example, the central processing unit accesses an input / output port via a bus. Also, all or part of the functional units provided in the cloud system 20 may be realized using hardware such as an ASIC, a PLD, or an FPGA. Also, all or part of each functional unit may be realized by a combination of software and hardware.

[0085] In addition, all or part of the functions of each unit of the cloud system 20 in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0086] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0087] Although the embodiments of the present invention have been described above, these embodiments and their modifications are presented as examples and are not intended to limit the scope of the invention. These embodiments and their modifications can be embodied in various other forms, and various omissions, substitutions, changes, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. [Explanation of symbols]

[0088] 1...information and communication system, 10...operator's PC, 11...video and audio output unit, 12...video and audio encoding unit, 13...interaction control UI, 20...cloud system, 21...session management unit, 22...video and audio filter unit, 23...settings storage unit, 24...video and audio transfer unit, 25...two-dimensional barcode generation unit, 30...STB, 31...video and audio setting operation unit, 32...video and audio output unit, 33...video and audio setting display unit, 34...remote control operation reception unit, 40...user device, 41...microphone, 42...camera, 43...video and audio encoding unit, 44...setting control unit, 45...browser, 50...remote control unit, 60...display

Claims

1. a session management unit that, after an interactive session is initiated between a first device that does not have a camera function or a microphone function and a second device that can transmit and receive video data and audio data, receives a connection request from a third device that has a camera function and a microphone function and initiates a connection to the interactive session; a video and audio transfer unit that transmits video data and audio data acquired by the third device to the second device and transmits the video data to the first device while the third device is connected to the interactive session by the session management unit; An information processing device comprising:

2. an identification data generation unit that generates identifier data that can be displayed as a video based on setting information including a session ID that identifies the interaction session; the video and audio transfer unit transmits the identifier data to the first device; The information processing device according to claim 1 .

3. the video and audio transfer unit superimposes the identifier data on the video data acquired by the second device and transmits the superimposed identifier data to the first device; The information processing device according to claim 2 .

4. the video and audio transfer unit stops transmitting the identifier data when the session management unit confirms that the third device has started connection to the interaction session. The information processing device according to claim 3 .

5. the video and audio transfer unit superimposes the video data acquired by the third device on the video data acquired by the second device and transmits the superimposed video data to the first device; The information processing device according to claim 1 .

6. the video and audio transfer unit does not transmit the audio data acquired from the third device to the first device, but transmits it only to the second device; The information processing device according to claim 5 .

7. a setting storage unit that receives and stores setting information of the camera function and the microphone function from the third device; the video and audio transfer unit transmits the setting information to the first device while the third device is connected to the interaction session; The information processing device according to claim 5 .

8. when receiving an update request for changing setting information of the camera function and the microphone function of the third device from the first device, the session management unit transmits a setting change request for the camera function and the microphone function to the third device. The information processing device according to claim 7 .

9. the setting storage unit updates the stored setting information with the changed setting information when receiving updated setting information in response to the update request from the third device; the session management unit transmits the changed setting information to the first device; The information processing device according to claim 8 .

10. the setting storage unit stores setting information relating to a camera function and a microphone function of the third device, which are set in advance for each service to be used, each of which is associated with a unique session ID; The information processing device according to claim 7 .

11. the setting information further includes setting content related to a background image for the video data of the third device. The information processing device according to claim 10.

12. An information processing device according to any one of claims 1 to 11; the first device; the second device; and the third device; and An information and communication system comprising:

13. a step of accepting a connection request from a third device having a camera function and a microphone function and initiating a connection to the interactive session after an interactive session is initiated between a first device having no camera function and a microphone function and a second device capable of transmitting and receiving video data and audio data; transmitting video and audio data acquired by the third device to the second device while the third device is connected to the interactive session, and transmitting the video data to the first device; An information processing method comprising:

14. On the computer, a step of receiving a connection request from a third device having a camera function and a microphone function after an interactive session is initiated between a first device having no camera function and a microphone function and a second device capable of transmitting and receiving video data and audio data; and initiating a connection to the interactive session; transmitting video and audio data acquired by the third device to the second device while the third device is connected to the interactive session, and transmitting the video data to the first device; A program that executes the following.

Images