Information processing device, method for controlling information processing device, and program

Abstract

To make it possible to distribute video data in which the content of comments by a viewer is reflected.SOLUTION: Video data imaged by a plurality of imaging devices are combined and a distribution video is distributed to a viewer terminal, and when comment data are sent from the viewer terminal, the received comment data are analyzed and it is determined whether or not a comment for generating camera control data is included. If such a comment is included, the camera control data for changing an angle of view of the video data to be displayed in a superimposed manner is transmitted to an imaging device that is imaging the video data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention particularly relates to an information processing device suitable for use in distributing video to viewers, a control method for the information processing device, and a program. [Background technology]

[0002] In recent years, a function has become known in which imaging devices such as digital cameras are equipped with wireless communication capabilities, and an information processing device is connected to the imaging device to transmit video data captured by the imaging device to an external device such as a distribution server via the information processing device, thereby enabling live streaming. Furthermore, video data captured by multiple imaging devices can be synthesized into a single video data set by the information processing device, enabling multi-angle streaming. In such a system, viewers can receive and view the video data transmitted to an external device such as a distribution server, and can also send comments, enabling viewer-participation streaming.

[0003] Furthermore, in viewer participation distribution systems, comments sent by a viewer are shared with other viewers, and the broadcaster who distributes the video and other viewers can understand the intention of the viewer who sent the comment through the comment. Meanwhile, in order to realize the viewer's wishes, a technology has been proposed in which the viewer pays a fee to control the imaging device that captures the video. Patent Document 1 discloses a technology in which a viewer who pays a fee is granted the authority to operate the imaging device, enabling the viewer to operate the imaging device from their own device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-13872 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, the server must be equipped with the necessary configuration for the viewer's terminal to operate the shooting terminal, and existing servers cannot control the shooting terminal based on operation information from the viewer's side.

[0006] In view of the above-mentioned problems, the present invention has an object to make it possible to distribute video data that reflects the contents of comments made by viewers. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a video acquisition means for acquiring video data from an imaging device, a distribution means for distributing video based on the video data acquired by the video acquisition means, a comment acquisition means for acquiring comment data including comments on the video distributed by the distribution means, the comments being generated on a terminal on which the video can be viewed and shared on other terminals on which the video can be viewed, a detection means for detecting a request for the video being distributed by the distribution means from the comment data acquired by the comment acquisition means, and a control means for performing control to change the content of the video being distributed by the distribution means based on the request detected by the detection means. [Effects of the Invention]

[0008] According to the present invention, it is possible to distribute video data that reflects the content of comments made by viewers. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the overall configuration of a distribution system according to each embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the imaging apparatus. [Figure 4]FIG. 2 is a diagram illustrating the flow of data processing in the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a screen displayed on a display unit of the information processing device. [Figure 6] FIG. 10 is a diagram showing an example of a result of analyzing a plurality of comment data. [Figure 7] FIG. 10 is a diagram illustrating the flow of data processing in the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a screen displayed on a display unit of the information processing device. [Figure 9] FIG. 10 is a diagram showing an example of a result of analyzing a plurality of comment data. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the embodiments described below are examples of means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. Furthermore, each embodiment can also be combined as appropriate.

[0011] (First embodiment) <Configuration of distribution system 100> Fig. 1 is a diagram showing an example of the overall configuration of a distribution system 100 according to this embodiment. In Fig. 1, an imaging device 300 is connected to a network 103 such as the Internet, and captures video to be distributed. In the example shown in Fig. 1, a swiveling camera 300A, a digital camera 300B, and an omnidirectional camera 300C are connected as imaging devices 300.

[0012] The swivel camera 300A has a function that allows the camera's angle of view and magnification to be controlled remotely or by a program installed inside the camera. The swivel camera 300A is also called a PTZ camera and has functions for horizontal control called pan control (PAN control), vertical control called tilt control (TILT control), and zoom control (ZOOM control). The digital camera 300B is a camera typified by a digital single-lens reflex camera or a digital mirrorless single-lens camera. Depending on the type of digital camera 300B, it may have a function for controlling the camera's zoom remotely or by a program installed inside the camera. In addition, by connecting to a control device called a remote camera platform, PAN control and TILT control may be possible, just like with a swivel camera. The spherical camera 300C is also called an omnidirectional camera or a 360-degree camera, and is a camera that can capture 360-degree images. These imaging devices 300 transmit captured image data to the information processing device 200.

[0013] The information processing device 200 is connected to a network 103 such as the Internet, and receives video data transmitted from the imaging device 300. The information processing device 200 converts the received video data into a data format suitable for distribution either directly or by applying image processing or the like, and transmits the converted video data to the distribution server 101. The information processing device 200 also determines which of the multiple imaging devices 300 has video data from which to transmit to the distribution server 101.

[0014] The distribution server 101 is connected to a network 103 such as the Internet, and receives video data transmitted by the information processing device 200. The distribution server 101 simultaneously distributes the received video data to a plurality of viewer terminals 102. At this time, the distribution server 101 converts the video data into a data format and resolution suitable for each of the plurality of viewer terminals 102, and transmits the converted video data to the viewer terminal 102. The distribution server 101 also receives comment data, etc. transmitted from the viewer terminal 102 (described later), and transmits it to other viewer terminals 102 and the information processing device 200.

[0015] Viewer terminal 102 is connected to network 103 such as the Internet and receives video data transmitted by distribution server 101. In the example shown in Fig. 1, a smart device 102A, a personal computer 102B, and a smart television 102C are connected as viewer terminals 102 capable of viewing video. Viewer terminal 102 not only plays back the video data transmitted by distribution server 101, but also transmits comments input by viewers to distribution server 101 as comment data.

[0016] The viewer terminal 102 also holds user account information for logging in to the distribution server 101, and by logging in to the distribution server 101, the viewer terminal 102 can send gift information such as tips to a video distributor who is a user of the information processing device 200. In addition, by logging in to the distribution server 101, the viewer terminal 102 can also send a follow to a video distributor who is a user of the information processing device 200, thereby conveying an interest in the video distributor.

[0017] <Configuration of information processing device 200> 2 is a block diagram showing an example of the hardware configuration of an information processing device 200 in this embodiment. An example of the information processing device 200 is a smart device. A smart device refers to a mobile terminal such as a smartphone or a tablet device. In this embodiment, a smart device will be described as an example of the information processing device 200, but the information processing device is not limited to this. For example, the information processing device 200 may be a digital camera, a personal computer, or the like.

[0018] The control unit 201 controls each unit of the information processing device 200 in accordance with input signals and a program described below. Note that instead of the control unit 201 controlling the entire information processing device 200, the entire information processing device 200 may be controlled by a plurality of hardware devices sharing the processing. The imaging unit 202 converts subject light imaged by a lens included in the imaging unit 202 into an electrical signal, performs noise reduction processing, etc., and outputs the digital data as image data. The output image data is stored in the working memory 204, and then undergoes predetermined calculations and encoding processing by the control unit 201, and is recorded as a file on the recording medium 210.

[0019] The nonvolatile memory 203 is an electrically erasable and recordable nonvolatile memory. The nonvolatile memory 203 stores an OS (operating system), which is basic software executed by the control unit 201, and applications that work with the OS to realize applied functions. In this embodiment, the nonvolatile memory 203 also stores applications for communicating with the imaging device 300 and the distribution server 101. The working memory 204 is used as a buffer memory for temporarily storing image data, an image display memory for the display unit 206, a working area for the control unit 201, etc.

[0020] The operation unit 205 accepts instructions from the user to the information processing device 200. The operation unit 205 includes operation members such as a power button used by the user to instruct ON / OFF of the power of the information processing device 200, and a touch panel formed on the display unit 206. The display unit 206 displays image data, displays characters for interactive operation, etc. Note that the information processing device 200 does not necessarily have to include the display unit 206. For example, it is sufficient if the information processing device 200 can be connected to an external display device and has at least a display control function for controlling the display of the display device. The operation unit 205 and the display unit 206 configure a user interface of the information processing device 200.

[0021] The recording medium 210 records the image data output from the imaging unit 202. The recording medium 210 may be configured to be detachable from the information processing device 200, or may be built into the information processing device 200. It is sufficient that the information processing device 200 has at least a means for connecting to the recording medium 210.

[0022] The communication unit 211 is an interface for connecting to the imaging device 300 and the distribution server 101. The information processing device 200 of this embodiment transmits and receives data to and from the imaging device 300 and the distribution server 101 via the communication unit 211. In this embodiment, the communication unit 211 is an antenna, and the control unit 301 connects to the imaging device 300 via the antenna. The connection with the imaging device 300 may be direct or via an access point. A protocol for communicating data is, for example, HTTP (Hyper Text Transfer Protocol). Alternatively, PTP / IP (Picture Transfer Protocol over Internet Protocol) via a wireless LAN may also be used. Note that communication with the imaging device 300 is not limited to this. For example, the communication unit 211 may include a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, or a Wireless USB. Furthermore, a wired connection such as a USB cable, HDMI (registered trademark), IEEE 1394, or Ethernet may also be employed.

[0023] The short-range wireless communication unit 212 is a communication unit for realizing short-range wireless communication. The short-range wireless communication unit 212 is composed of an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The short-range wireless communication unit 212 realizes short-range wireless communication by outputting a modulated wireless signal from the antenna and demodulating the wireless signal received by the antenna. Here, short-range wireless communication is realized in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). Note that the non-contact proximity communication realized by the short-range wireless communication unit 212 is not limited to Bluetooth (registered trademark), and other wireless communication may be adopted.

[0024] The public network communication unit 213 is an interface used when performing public wireless communication. The information processing device 200 communicates with other devices via the public network communication unit 213. At this time, the control unit 201 inputs and outputs audio signals via the microphone 214 and the speaker 215 to realize the communication. In this embodiment, the public network communication unit 213 is an antenna, and the control unit 201 connects to the public network via the antenna. Note that a single antenna may be used as both the communication unit 211 and the public network communication unit 213.

[0025] <Configuration of imaging device 300> 3 is a block diagram showing an example of the hardware configuration of the imaging device 300 according to this embodiment. Note that, although the present embodiment shows the swivel camera 300A, the digital camera 300B, and the omnidirectional camera 300C as examples of the imaging device 300, the imaging device is not limited to these. For example, the imaging device may be a portable media player, a smart device, a personal computer, or the like.

[0026] The control unit 301 controls each unit of the imaging device 300 in accordance with input signals and a program described below. Note that instead of the control unit 301 controlling the entire imaging device 300, the entire imaging device 300 may be controlled by multiple pieces of hardware sharing the processing.

[0027] The imaging unit 302 is composed of, for example, an optical lens unit, an optical system that controls the aperture, zoom, focus, etc., and an imaging element that converts light (image) introduced through the optical lens unit into an electrical video signal. A CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) is generally used as the imaging element. Under the control of the control unit 301, the imaging unit 302 converts subject light focused by a lens included in the imaging unit 302 into an electrical signal using the imaging element, performs noise reduction processing, etc., and outputs image data as digital data. In the imaging device 300 of this embodiment, the control unit 301 encodes the image data and records it as a file on the recording medium 310 in accordance with the DCF (Design Rule for Camera File system) standard.

[0028] The nonvolatile memory 303 is an electrically erasable and recordable nonvolatile memory, and stores programs, etc., which will be described later, executed by the control unit 301. The working memory 304 is used as a buffer memory for temporarily storing image data generated by the imaging unit 302, an image display memory for the display unit 306, a working area for the control unit 301, etc.

[0029] The operation unit 305 accepts instructions from the user to the imaging device 300. The operation unit 305 includes, for example, a power button used by the user to instruct ON / OFF of the power supply to the imaging device 300, and a release switch used to instruct shooting. The release switch has SW1 and SW2 for detecting a two-stage depression state. When the release switch is pressed halfway, SW1 turns ON. This accepts instructions to perform shooting preparations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing. When the release switch is pressed fully, SW2 turns ON. This accepts instructions to perform shooting preparations.

[0030] The display unit 306 displays a viewfinder image during shooting, displays captured image data, and displays text for interactive operation. The display unit 306 does not necessarily have to be included in the imaging device 300. The imaging device 300 can be connected to an internal or external display device, and it is sufficient that the imaging device 300 has at least a display control function for controlling the display on the display device. The microphone 307 is used for inputting audio. The imaging device 300 of this embodiment transmits input audio to the information processing device 200 as audio data.

[0031] The PT control unit 308 controls the angle of view by performing PAN control and TILT control of the image capturing device 300. Note that the image capturing device 300 itself may have a PAN / TILT structure, or the image capturing device 300 may have a PAN / TILT structure by being coupled with a remote camera platform or the like.

[0032] The recording medium 310 records a file of image data output from the imaging unit 302. The recording medium 310 may be configured to be detachable from the imaging device 300, or may be built into the imaging device 300. The imaging device 300 only needs to have a means for accessing the recording medium 310.

[0033] The communication unit 311 is an interface for connecting to the information processing device 200. The imaging device 300 of this embodiment transmits and receives data to and from the information processing device 200 via the communication unit 311. For example, image data generated by the imaging unit 302 is transmitted to the information processing device 200 via the communication unit 311. Note that in this embodiment, the communication unit 311 includes an interface for communicating with the information processing device 200 via a so-called wireless LAN in accordance with the IEEE802.11 standard. The control unit 301 controls the communication unit 311 to realize wireless communication with the information processing device 200.

[0034] The short-range communication unit 312 includes, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, a communication controller, etc. The short-range communication unit 312 outputs modulated wireless signals from the antenna and demodulates wireless signals received by the antenna to achieve short-range wireless communication in accordance with the IEEE 802.15 standard (so-called Bluetooth (registered trademark)). In this embodiment, Bluetooth (registered trademark) communication employs version 4.0 of Bluetooth (registered trademark) Low Energy (hereinafter, BLE), which is low in power consumption. This Bluetooth (registered trademark) communication has a narrower communication range and shorter communication distance than wireless LAN communication, and its communication speed is slower than wireless LAN communication. On the other hand, Bluetooth (registered trademark) communication consumes less power than wireless LAN communication. The imaging device 300 of this embodiment can transmit and receive data to and from the information processing device 200 via the short-range communication unit 312. For example, when receiving a command to capture an image from the information processing device 200, the imaging unit 302 controls the imaging operation, and when receiving a command to transmit and receive data via wireless LAN communication, the communication unit 311 controls the communication unit 311 to start wireless LAN communication.

[0035] <Regarding the processing of the distribution system 100> Fig. 4 is a diagram illustrating the flow of data processing in the distribution system 100 of this embodiment. Fig. 5 is a diagram illustrating an example of a screen displayed on the display unit 206 of the information processing device 200 of this embodiment. The flow of processing in this embodiment will be described below with reference to Figs. 4 and 5.

[0036] Fig. 4(a) is a diagram showing the flow of processing in which video data generated by the imaging device 300 is synthesized by the information processing device 200 and the synthesized video data is distributed to the viewer terminal 102. As shown in Fig. 4(a), the information processing device 200 has, as functional components for synthesizing and distributing video data, video data receiving units 401A to 401C, a video data synthesis unit 402, a synthesis instruction unit 403, and a video data transmission unit 404. Each of these components is realized when the control unit 201 loads a program stored in the non-volatile memory 203 into the work memory 204 and the control unit 201 executes the program.

[0037] First, video data receiving unit 401A receives video data captured by swivel camera 300A. Similarly, video data receiving unit 401B receives video data captured by digital camera 300B, and video data receiving unit 401C receives video data captured by omnidirectional camera 300C. At this time, omnidirectional camera 300C may transmit the captured 360-degree video as video data, or may transmit video data as normal video by cutting out a portion of the video. In this embodiment, omnidirectional camera 300C cuts out a portion of the captured 360-degree video and transmits the video data as normal video.

[0038] Video data synthesis unit 402 synthesizes video data using the video data acquired by video data receiving units 401A to 401C in accordance with an instruction from synthesis instruction unit 403. Then, video data transmission unit 404 transmits the synthesized video data to distribution server 101. Here, the synthesis instruction from synthesis instruction unit 403 will be described with reference to FIG. 5.

[0039] 5(a) to 5(c) are diagrams showing example screens displayed on the display unit 206 of the information processing device 200 when issuing an instruction related to compositing. FIG. 5(a) shows an example screen displayed on the display unit 206 before the compositing instruction unit 403 issues a compositing instruction. Received video 501A is video data captured by the swivel camera 300A and received by the video data receiving unit 401A. Received video 501B is video data captured by the digital camera 300B and received by the video data receiving unit 401B. Received video 501C is video data captured by the omnidirectional camera 300C and received by the video data receiving unit 401C. The display area 502 displays the result of the video data compositing by the video data compositing unit 402, but nothing is displayed because the user has not yet issued a compositing instruction. When the user drags one of the received videos 501A to 501C onto the display area 502, the compositing instruction unit 403 issues an instruction to use the desired received video as the base for the composite video.

[0040] 5(b) shows a screen displayed on display unit 206 when received video 501B is selected as the base for the composite video. At the stage when the base image is selected, only received video 501B is selected for video 512, so video 512 is the same image as received video 501B. When the user drags received video 501A or received video 501C onto video 512, composition instruction unit 403 gives an instruction to composite the desired received video.

[0041] Fig. 5(c) shows a screen displayed on the display unit 206 when the received video 501A is selected as the video to be combined on the screen of Fig. 5(b). Note that in this embodiment, a combined video 522 in which the received video 501A is superimposed is displayed in the lower right area of ​​the received video 501B, but the method of combining the videos is not limited to this. For example, the two received videos may be combined in a split screen, or the user may be allowed to select the position for the superimposed display.

[0042] When the video data synthesis unit 402 synthesizes the video data in response to the instruction from the synthesis instruction unit 403 as described above, the video data transmission unit 404 transmits the synthesized video data to the distribution server 101 as a distribution video. The distribution server 101 transmits the video data received from the information processing device 200 to the smart device 102A, the personal computer 102B, and the smart TV 102C as a distribution video. At this time, the distribution server 101 converts the video data into a frame rate and resolution suitable for each viewer terminal and transmits the video data to the smart device 102A, the personal computer 102B, and the smart TV 102C.

[0043] 4(b) is a diagram showing the flow of processing in which comment data input to the viewer terminal 102 is transmitted to the information processing device 200 via the distribution server 101, and camera control data obtained by analyzing the comment data is transmitted to the imaging device 300. As shown in FIG. 4(b), the information processing device 200 includes a comment data receiving unit 411, a comment display unit 412, a comment data analysis unit 413, and camera control data transmitting units 414A to 414C as functional components for analyzing the comment data and controlling the camera. Each of these components is realized when the control unit 201 loads a program stored in the non-volatile memory 203 into the work memory 204 and the control unit 201 executes the program.

[0044] First, the personal computer 102B transmits comment data to the distribution server 101 based on comments entered by the user of the personal computer 102B. Here, the comment data may include account information related to the user of the personal computer 102B and gift information such as tips as user information. In addition, other viewer terminals 102, such as the smart device 102A and the smart TV 102C, can also transmit comment data to the distribution server 101 in a similar manner.

[0045] The distribution server 101 transmits the comment data received from the personal computer 102B to the information processing device 200, the smart device 102A, and the smart TV 102C. The comment data received by the information processing device 200 is also received by other viewer terminals 102, such as the smart device 102A and the smart TV 102C.

[0046] The comment data receiving unit 411 acquires comments by receiving comment data from the distribution server 101. The comment display unit 412 displays comments on the display unit 206 based on the comment data received by the comment data receiving unit 411.

[0047] Fig. 5(d) shows an example of a screen displayed on the display unit 206 when comments are displayed by the comment display unit 412. In the example of the screen in Fig. 5(d), received video 501B, which is the base, shows four people: Person A 541, Person B 542, Person C 543, and Person D 544. Received video 501A, which is superimposed and displayed at the bottom right, shows an enlarged face of Person D 544.

[0048] When the user selects the comment display button 531, a comment display screen 532 is displayed. Furthermore, when the comment display button 531 is selected while the comment display screen 532 is displayed, the comment display screen 532 is hidden. The comment display unit 412 displays on the comment display screen 532 a user icon 533 of the viewer who sent the comment data, a user name 534, an icon 535 indicating the following status, a gift sending status 536 such as a tip, and a comment 537.

[0049] The comment data analysis unit 413 analyzes the comment data received by the comment data receiving unit 411. Then, the comment data analysis unit 413 generates camera control data for changing the shooting range from the analyzed comment data. In the example shown in Fig. 5(d), the received comment data includes comment 537 requesting "I want to see Mr. B," so camera control data is generated to zoom in on Mr. B 542.

[0050] Here, one method for detecting a comment for which camera control data should be generated from multiple comments is to detect a specific string of characters related to a request such as "I want to see it." Other methods include detecting comments that start with a specific symbol (for example, "#") or detecting a predetermined string of characters. It is also possible to change the detection method depending on the viewer who made the comment. Specifically, there may be a string of characters that are only accepted in comments from viewers who have sent gifts such as tips, or from viewers who follow the viewer. As described above, various methods can be used to detect comments for which camera control data should be generated.

[0051] 5(d), the video data captured by omnidirectional camera 300C is not synthesized, and the video data captured by digital camera 300B is used as the base for the synthesized video. Therefore, comment data analysis unit 413 analyzes from the comment data that the subject being captured by swivel camera 300A should be switched from person D 544 to person B 542, and generates camera control data for swivel camera 300A. Alternatively, executable camera controls may be acquired in advance from imaging devices 300, and the camera control data may be transmitted only to imaging devices that can execute the controls.

[0052] Here, the generated camera control data may be control values ​​for PAN control, TILT control, or ZOOM control based on the results of analysis by comment data analysis unit 413, or may be information about the subject. The control value for PAN control is expressed as the current horizontal direction or the horizontal angle from a reference position, and the control value for TILT control is expressed as the current vertical direction or the vertical angle from a reference position. The information about the subject may be a person's name such as "Mr. B" or information indicating Mr. B's characteristics.

[0053] Camera control data transmission unit 414A transmits the camera control data generated by comment data analysis unit 413 to swivel camera 300A. Note that in the example of FIG. 5(d), camera control data is transmitted only to swivel camera 300A, but camera control data transmission units 414B and 414C may also similarly generate camera control data for digital camera 300B and omnidirectional camera 300C, respectively. When comment data analysis unit 413 generates camera control data for digital camera 300B, camera control data transmission unit 414B transmits the camera control data to digital camera 300B. Similarly, when comment data analysis unit 413 generates camera control data for omnidirectional camera 300C, camera control data transmission unit 414C transmits the camera control data to omnidirectional camera 300C. The camera control data transmitted to omnidirectional camera 300C may be cropping coordinates based on the results of analysis by comment data analysis unit 413, or may be information about the subject.

[0054] Swivel camera 300A controls the camera based on the camera control data received from camera control data transmitter 414A. If the received camera control data contains control values ​​related to PAN control, TILT control, or ZOOM control, control unit 301 controls PT control unit 308 of swivel camera 300A according to those control values ​​to change the angle of view of swivel camera 300A. If the received camera control data contains information related to a subject, control unit 301 controls PT control unit 308 of swivel camera 300A to change the angle of view of swivel camera 300A to search for the subject. If the subject is found, control unit 301 adjusts the angle of view of swivel camera 300A to face the subject so that the subject is at the center. Note that subject recognition results may be registered in non-volatile memory 303 of swivel camera 300A in advance, and the subject may be searched for by receiving the registered name of the corresponding subject. In this embodiment, information indicating the characteristics of person B 542 is received as camera control data, and swivel camera 300A faces person B 542.

[0055] Digital camera 300B and spherical camera 300C also perform similar control when they receive camera control data. Here, if the camera control data received by spherical camera 300C is crop coordinates, spherical camera 300C crops the captured 360-degree video according to the received crop coordinates and transmits the video data as a normal video. Also, if the camera control data received by spherical camera 300C is information about a subject, spherical camera 300C searches for the subject from the captured 360-degree video, crops out the area in which the subject appears, and transmits the video data as a normal video. FIG. 5(e) shows a screen on which a video of person B 542 is superimposed in the lower right corner by camera control based on comment 537 saying, "I want to see person B." Then, as shown in FIG. 4(a), video data transmission unit 404 transmits video data in which person B 542 appears on the video superimposed in the lower right corner to distribution server 101.

[0056] <About behavior when receiving multiple comments> Next, processing when multiple comment data are received will be described. Fig. 6 is a diagram showing an example of the results of analysis by the comment data analysis unit 413 of multiple comment data received by the comment data receiving unit 411. In Fig. 6, viewer information 601 is information about the user (viewer) of the viewer terminal 102, and includes information such as the user name, information on whether the user is following the user (video distributor) of the information processing device 200, and information on the status of gift delivery to the video distributor. In the example of Fig. 6, a viewer with the user name Viewer3 and a viewer with the user name Viewer4 are following the video distributor. In addition, the viewer with the user name Viewer2 has sent a gift of 100 points to the video distributor, and the viewer with the user name Viewer4 has sent a gift of 500 points.

[0057] The comment data analysis result 602 is the result of analysis by the comment data analysis unit 413, and is the result of detecting comment data for which camera control data should be generated from the viewer comments. In the example of Fig. 6, the viewer with the user name Viewer1 and the viewer with the user name Viewer3 have sent comment data stating that they want to see person A 541, and the viewer with the user name Viewer2 has sent comment data stating that they want to see person B 542. Note that the requested comment has not been detected in the comment data of the viewer with the user name Viewer4.

[0058] The comment data analysis unit 413 generates camera control data based on the viewer information 601 and the comment data analysis result 602. In the example of FIG. 6, it is possible to simultaneously generate and transmit camera control data to different image capture devices 300. For example, it is possible to simultaneously generate and transmit camera control data instructing the swivel camera 300A to capture person A 541 and camera control data instructing the digital camera 300B to capture person B 542. However, if the camera control data instructing the camera to capture person A 541 and the camera control data instructing the camera to capture person B 542 are instructing the same image capture device 300, these camera control data cannot be simultaneously generated and transmitted. Therefore, in this embodiment, when multiple comment data for which camera control data should be generated are detected, the camera control data to be generated from the multiple comment data is determined using the following method.

[0059] In this embodiment, the first method for determining the camera control data to be generated is to prioritize comment data from viewers who follow the video broadcaster. A second method is to prioritize comment data from viewers who have sent gifts to the video broadcaster. A third method is to prioritize the comment data analysis results for the same comment with the largest number of viewers. These three methods are described below.

[0060] First, when the first method is adopted, comment data analysis unit 413 determines the camera control data to be generated based on information on whether or not the viewer is following the video distributor in viewer information 601. In the example of Fig. 6, a viewer with the username Viewer3 is following the video distributor, so the comment of the viewer with the username Viewer3 is given priority. As a result, comment data analysis unit 413 generates camera control data that instructs swiveling camera 300A to capture person A 541.

[0061] Furthermore, when the second method is adopted, comment data analysis unit 413 determines the camera control data to be generated based on information in viewer information 601 indicating whether a gift has been sent to the video distributor. In the example of Fig. 6, the viewer with the username Viewer2 has sent a gift to the video distributor, so the comment of the viewer with the username Viewer2 is given priority. As a result, comment data analysis unit 413 generates camera control data that instructs swivel camera 300A to capture person B 542.

[0062] Furthermore, when the third method is adopted, comment data analysis unit 413 determines the camera control data to be generated based on the number of viewers with the same comment analysis result from comment data analysis result 602. In the example of FIG. 6, there are two comments requesting that person A 541 be filmed, and only one comment requesting that person B 542 be filmed. Therefore, the comment instructing that person A 541 be filmed is given priority. As a result, comment data analysis unit 413 generates camera control data instructing swivel camera 300A to film person A 541.

[0063] As described above, according to this embodiment, comment data from viewers is analyzed, and camera control data for the imaging device that generates the video data is generated to control the imaging device. This allows the content of the video to be changed and video to be distributed in accordance with the viewer's requests.

[0064] (Second embodiment) A second embodiment of the present invention will be described below. In this embodiment, comment data is transmitted from a viewer terminal to an information processing device via a distribution server, and the information processing device changes the video to be superimposed based on the analysis results of the comment data. Note that the basic configurations of the distribution system, information processing device, and imaging device 300 are the same as those shown in FIGS. 1 to 3 described in the first embodiment, and therefore will not be described here.

[0065] <About the distribution system processing> FIG. 7 is a diagram showing an example of the flow of processing for switching the video to be superimposed on the basis of comment data in the distribution system 100 according to this embodiment. FIG. 8 is a diagram showing an example of a screen displayed on the display unit 206 of the information processing device 200 according to this embodiment. Note that the processing from receiving video data to synthesizing the video data and transmitting the resulting data to the distribution server 101 is basically the same as the flow shown in FIG. 4(a). The video data synthesizing unit 702 synthesizes the video data received by the video data receiving units 701A, 701B, and 701C in accordance with instructions from the synthesis instruction unit 703. Furthermore, in this embodiment, the spherical camera 300C transmits video captured within a 360-degree range as video data.

[0066] In this embodiment as well, before the composition instruction unit 703 issues a composition instruction, the screen shown in FIG. 8(a) is displayed on the display unit 206. Note that the received videos 801A to 801C are video data captured by the swivel camera 300A, the digital camera 300B, and the omnidirectional camera 300C, respectively. First, the user selects the received video 801B from the received videos 801A to 801C and drags it to the display area 802, thereby selecting the received video that will serve as the base, and the screen shown in FIG. 8(b) appears. Thereafter, the user selects the received video 801A as the video to be composed, whereby the received video 801A is composed with the video 812 in FIG. 8(b), and a composed video 822 as shown in FIG. 8(c) is displayed. When the video data composition unit 702 has composed the video data in response to the instruction from the composition instruction unit 703 as described above, the video data transmission unit 704 transmits the composed video data to the distribution server 101. As a result, the composite video data is transmitted to the smart device 102A, the personal computer 102B, and the smart TV 102C.

[0067] Next, the comment data receiving unit 711 receives comment data from the personal computer 102B via the distribution server 101. In this case, the comment data is also transmitted to the smart device 102A and the smart TV 102C by the distribution server 101. Then, the comment display unit 712 displays the comment on the display unit 206 based on the comment data received by the comment data receiving unit 711.

[0068] As in the first embodiment, as shown in FIG. 8(d), the user of the information processing device 200 can display or hide a comment display screen 832 by selecting a comment display button 831. The comment display screen 832 displays a user icon 833, a user name 834, an icon 835 indicating a follow status, a gift sending status 836 such as a tip, and a comment 837. As in the first embodiment, it is assumed that received video 801B, which is the base of the composite video, shows person A 841, person B 842, person C 843, and person D 844, and that superimposed received video 801A shows person D 844. It is also assumed that uncomposited received video 801C shows person A 841.

[0069] In this embodiment, the comment data analysis unit 713 analyzes the comment data received by the comment data receiving unit 711 and sends a video data synthesis instruction to the synthesis instruction unit 703 based on the analyzed comment data. In the example shown in FIG. 8(d), the received comment data includes comment 837 requesting "I want to see Mr. / Ms. A," and therefore a synthesis instruction is issued to synthesize video data including Mr. / Ms. A 841. Therefore, the comment data analysis unit 713 instructs the synthesis instruction unit 703 to synthesize received video 801C captured by omnidirectional camera 300C, in which Mr. / Ms. A 841 appears. The synthesis instruction unit 703 then instructs the video data synthesis unit 702 to search for Mr. / Ms. A 841 from the received video data and synthesize received video including Mr. / Ms. A 841.

[0070] In the example shown in Fig. 8, the video data synthesis unit 702 searches for person A 841 from the 360-degree video received by the video data receiving unit 701C, and cuts out and synthesizes an area showing person A 841. Fig. 8(e) shows a screen on which a video showing person A 841 is superimposed in the lower right corner based on comment 837 requesting "I want to see person A." Note that while the example shown in Fig. 8(c) shows an example in which received video 801A captured by swivel camera 300A is synthesized, the same applies when a region including person D 844 is cut out and synthesized from video captured by omnidirectional camera 300C. In this case, after receiving the comment data, the imaging device that acquires the video to be superimposed is not changed, and only the position from which it is cut out from the video captured over a 360-degree range is changed.

[0071] <About behavior when receiving multiple comments> Next, a process when a plurality of comment data are received will be described. It is assumed that the comment data analysis unit 713 obtains, for example, the viewer information 601 and the comment data analysis result 602 in Fig. 6. In this embodiment, as in the first embodiment, the first to third methods are used when determining the video data to be instructed to be combined.

[0072] 6, the viewer with the username Viewer3 is following the video distributor, and therefore the comment of the viewer with the username Viewer3 is given priority. As a result, comment data analysis unit 713 instructs composition instruction unit 703 to compose received video 801C captured by omnidirectional camera 300C in which person A 841 appears. Then, based on the instruction from composition instruction unit 703, video data composition unit 702 searches for person A 841 from the 360-degree video received by video data receiving unit 701C, and cuts out and composes the area in which person A 841 appears.

[0073] When the second method of prioritizing comment data of a viewer who sent a gift to the video distributor is adopted, in the example of Fig. 6, a viewer with the username Viewer2 sent a gift to the video distributor. Therefore, the comment of the viewer with the username Viewer2 is prioritized. As a result, comment data analysis unit 713 instructs composition instruction unit 703 to composite received video 801C captured by omnidirectional camera 300C in which person B 842 appears. Then, based on the instruction from composition instruction unit 703, video data composition unit 702 searches for person B 842 from the 360-degree video received by video data receiving unit 701C, and cuts out and composites the area in which person B 842 appears.

[0074] Furthermore, when the comment data analysis result adopts the third method of prioritizing comments with the largest number of viewers of the same comment, in the example of Fig. 6, there are two comments requesting that person A 841 be shown, and only one comment requesting that person B 842 be shown. Therefore, the comment instructing that person A 841 be shown is given priority. As a result, comment data analysis unit 713 instructs composition instruction unit 703 to compose received video 801C captured by omnidirectional camera 300C in which person A 841 is shown. Then, based on the instruction from composition instruction unit 703, video data composition unit 702 searches for person A 841 from the 360-degree video received by video data receiving unit 701C, and cuts out and composes the area in which person A 841 is shown.

[0075] As described above, according to this embodiment, comment data from viewers is analyzed and a synthesis instruction is issued to change the video to be synthesized. This allows the content of the distributed video to be changed and video to be distributed in accordance with the viewer's wishes.

[0076] (Other embodiments) In the first and second embodiments described above, examples have been described in which, when multiple comment data are received, camera control data is generated and superimposed video data is switched based on whether or not there is a follow, whether or not there is a gift, or the number of comments. However, it is also possible to take all of these methods into consideration, and also consider the duration of the follow and the number of gift points, and generate camera control data and switch superimposed video data based on a comprehensive judgment. Hereinafter, a method for generating camera control data and switching superimposed video data based on a comprehensive judgment in the first or second embodiment will be described.

[0077] 9(a) is a diagram showing an example of an analysis result for a comment from a viewer. In FIG. 9(a), viewer information 901 and comment data analysis result 902 are basically the same as those in FIG. 6, but the viewer information 901 also includes information on the period of following, if any. In the example of FIG. 9(a), a viewer with the username Viewer3 has been following the user (video distributor) of the information processing device 200 for 500 days. Similarly, a viewer with the username Viewer4 has been following the video distributor for 100 days.

[0078] In the example of the first embodiment, the comment data analysis unit 413 generates camera control data based on the viewer information 901 and the comment data analysis result 902. In addition, in the example of the second embodiment, the comment data analysis unit 713 instructs the synthesis instruction unit 703 to synthesize video data based on the viewer information 901 and the comment data analysis result 902. Below, an example will be described in which, in order to prioritize multiple comment data, the number of comments is multiplied according to the viewer's situation and then the subject with the most comments is captured.

[0079] FIG. 9(b) is a diagram showing an example of a multiplier according to the following status 911 of a video broadcaster. As shown in FIG. 9(b), if a viewer is not following the video broadcaster, the number of comments remains the same (1x), but if the viewer is following the video broadcaster, a different multiplier is applied depending on the length of time the viewer has been following the video broadcaster. In the example of FIG. 9(b), the multiplier is set to prioritize viewers who have been following for a long time, while also prioritizing viewers who have recently followed the video broadcaster in order to attract new viewers. Therefore, if the following period is one year or more, the number of comments is multiplied by four; if the following period is one day or more but less than one year, the number of comments is multiplied by two; and if the following period is less than one day, the number of comments is multiplied by three.

[0080] 9(c) is a diagram showing an example of a multiplier according to a viewer's gift sending status 921. As shown in FIG. 9(c), if the viewer has not sent a gift to the video distributor, the number of comments remains the same (1x), but if the viewer has sent a gift to the video distributor, a multiplier is set according to the number of points. In the example of FIG. 9(c), if a gift of 100 points is sent, the number of comments is doubled, if a gift of 200 points is sent, the number of comments is tripled, and if a gift of 500 points is sent, the number of comments is six times.

[0081] FIG. 9(d) shows an example of the analysis results obtained by multiplying the number of comments by a multiplier, with reference to the multipliers shown in FIGS. 9(b) and 9(c). The viewer with the username Viewer1 does not follow the video broadcaster and has not sent a gift to the video broadcaster, so the number of comments remains the same (1x). The viewer with the username Viewer2 does not follow the video broadcaster, but has sent a gift of 100 points to the video broadcaster, so the number of comments is doubled. The viewer with the username Viewer3 has followed the video broadcaster for over a year, but has not sent a gift to the video broadcaster, so the number of comments is quadrupled. The viewer with the username Viewer4 has followed the video broadcaster for over a day but less than a year, and has sent a gift of 500 points to the video broadcaster, so the number of comments is 12x. As a result, there are a total of five comments saying they want to see Person A, two comments saying they want to see Person B, and 12 comments saying they want to see Person C. In this way, when reflecting the following status and gift sending status in the viewer information 901, camera control data is generated or a synthesis instruction is given so that person C, who has the most comments, is shown.

[0082] Furthermore, after a predetermined time has elapsed since Person C was made to appear on camera, the comment data analysis results may be referenced to determine again which comment to prioritize. In this case, this predetermined time may be proportional to the number of comments. Specifically, for example, if one minute is allowed per comment, 12 minutes after Person C is made to appear on camera, the comment data analysis results may be referenced to determine again which comment to prioritize. In the example of FIG. 9(d), since there are five comments indicating a desire to see Person A, which is the second most common, camera control data is generated or a synthesis instruction is issued to show Person A for five minutes. Note that the above example is an example in which no new comments have been analyzed in the comment data analysis results; in reality, the analysis results of new comment data received while Person C was on camera may also be included in the determination.

[0083] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0084] The disclosure of this embodiment includes the following configuration, method, and program.

[0085] (Configuration 1) an image acquisition means for acquiring image data from an imaging device; a distribution means for distributing a distribution video based on the video data acquired by the video acquisition means; a comment acquisition means for acquiring comment data including comments on the video stream distributed by the distribution means, the comments being generated on a terminal capable of viewing the video stream and being shared on other terminals capable of viewing the video stream; and a detection means for detecting a request for the video being distributed by the distribution means from the comment data acquired by the comment acquisition means; a control means for controlling the changing of the content of the video to be distributed by the distribution means based on the request detected by the detection means; An information processing device comprising:

[0086] (Configuration 2) The image capturing device further includes a synthesizing unit that synthesizes the image data acquired from the plurality of image capturing devices by the image acquiring unit, 2. The information processing device according to configuration 1, wherein the distribution means distributes the distribution video synthesized by the synthesis means. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the control means performs control to change the imaging range of the imaging device. (Configuration 4) 3. The information processing apparatus according to configuration 2, wherein the control means performs control to change the content of the video data to be synthesized by the synthesis means. (Configuration 5) the imaging device includes a swivel camera capable of at least pan control, tilt control, and zoom control; 5. The information processing device according to configuration 3 or 4, wherein the control means transmits camera control data for changing the shooting range of the swivel camera to the swivel camera. (Configuration 6) The imaging device includes a camera that captures images in a range of at least 360 degrees, The information processing device according to configuration 1, 2 or 4, wherein the control means performs control to change the content of the distributed video using a partial area of ​​the video data captured in the 360-degree range.

[0087] (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the detection means detects a request for the video to be distributed from a specific character string or symbol included in the comment. (Configuration 8) The comment data includes, as user information of the terminal that generated the comment, at least information on the status of following the broadcaster or the status of sending gifts to the broadcaster, When the comment acquisition means acquires a plurality of comment data and detects a plurality of requests for the distribution video, the detection means selects one of the plurality of detected requests based on at least the following status or the gift sending status; 8. The information processing device according to any one of configurations 1 to 7, wherein the control means performs control to change the content of the video to be distributed based on the selected one request. (Configuration 9) When the comment acquisition means acquires a plurality of comment data and detects a plurality of requests for the video to be distributed, the detection means selects one of the plurality of detected requests according to the number of comments, 8. The information processing device according to any one of configurations 1 to 7, wherein the control means performs control to change the content of the video to be distributed based on the selected one request. (Configuration 10) The information processing device described in configuration 8 is characterized in that when multiple comment data are acquired by the comment acquisition means and multiple requests for the distribution video are detected, the detection means selects one of the multiple detected requests by weighting according to the follow status, the gift sending status, and the number of comments. (Configuration 11) 11. The information processing device according to configuration 10, wherein the control means performs control to change the content of the distributed video for a time period according to the weighting.

[0088] (method) an image acquisition step of acquiring image data from an imaging device; a distribution step of distributing a distribution video based on the video data acquired in the video acquisition step; a comment acquisition step of acquiring comment data including comments on the video stream distributed in the distribution step, the comments being generated on a terminal capable of viewing the video stream and being shared on other terminals capable of viewing the video stream; a detection step of detecting a request for the video being distributed in the distribution step from the comment data acquired in the comment acquisition step; a control step of performing control to change the content of the video to be distributed in the distribution step based on the request detected in the detection step; 1. A method for controlling an information processing device, comprising:

[0089] (program) 12. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0090] 401A to 401C: video data receiving unit, 402: video data synthesis unit, 404: video data transmitting unit, 411: comment data receiving unit, 413: comment data analysis unit, 414A, 414A to 414C: camera control data transmitting unit

Claims

1. an image acquisition means for acquiring image data from an imaging device; a distribution means for distributing a distribution video based on the video data acquired by the video acquisition means; a comment acquisition means for acquiring comment data including comments on the video stream distributed by the distribution means, the comments being generated on a terminal capable of viewing the video stream and being shared on other terminals capable of viewing the video stream; and a detection means for detecting a request for the video being distributed by the distribution means from the comment data acquired by the comment acquisition means; a control means for controlling the changing of the content of the video to be distributed by the distribution means based on the request detected by the detection means; An information processing device comprising:

2. The image capturing device further includes a synthesizing unit that synthesizes the image data acquired from the plurality of image capturing devices by the image acquiring unit, 2. The information processing apparatus according to claim 1, wherein the distribution means distributes the distribution video synthesized by the synthesis means.

3. 2. The information processing apparatus according to claim 1, wherein the control means performs control to change the imaging range of the imaging device.

4. 3. The information processing apparatus according to claim 2, wherein said control means performs control to change the content of the video data to be synthesized by said synthesis means.

5. the imaging device includes a swivel camera capable of at least pan control, tilt control, and zoom control; 5. The information processing apparatus according to claim 3, wherein the control means transmits camera control data for changing the photographing range of the swivel camera to the swivel camera.

6. The imaging device includes a camera that captures images in a range of at least 360 degrees, 5. The information processing apparatus according to claim 1, wherein the control means performs control to change the content of the video to be distributed by using a partial area of ​​the video data captured in the 360-degree range.

7. 2. The information processing apparatus according to claim 1, wherein the detection means detects a request for the video to be distributed from a specific character string or symbol included in the comment.

8. The comment data includes, as user information of the terminal that generated the comment, at least information on the status of following the broadcaster or the status of sending gifts to the broadcaster, When the comment acquisition means acquires a plurality of comment data and detects a plurality of requests for the distribution video, the detection means selects one of the plurality of detected requests based on at least the following status or the gift sending status; 2. The information processing apparatus according to claim 1, wherein said control means performs control to change the content of said distribution video based on said selected one request.

9. When the comment acquisition means acquires a plurality of comment data and detects a plurality of requests for the video to be distributed, the detection means selects one of the plurality of detected requests according to the number of comments, 2. The information processing apparatus according to claim 1, wherein said control means performs control to change the content of said distribution video based on said selected one request.

10. The information processing device described in claim 8, characterized in that when multiple comment data are acquired by the comment acquisition means and multiple requests for the distribution video are detected, the detection means selects one of the multiple detected requests by weighting according to the follow status, the gift delivery status, and the number of comments.

11. 11. The information processing apparatus according to claim 10, wherein the control means performs control to change the content of the video to be distributed for a period of time according to the weighting.

12. an image acquisition step of acquiring image data from an imaging device; a distribution step of distributing a distribution video based on the video data acquired in the video acquisition step; a comment acquisition step of acquiring comment data including comments on the video stream distributed in the distribution step, the comments being generated on a terminal capable of viewing the video stream and being shared on other terminals capable of viewing the video stream; a detection step of detecting a request for the video being distributed in the distribution step from the comment data acquired in the comment acquisition step; a control step of performing control to change the content of the video to be distributed in the distribution step based on the request detected in the detection step; 1. A method for controlling an information processing device, comprising:

13. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

Images