Video imaging device, video imaging method, and program

Abstract

Video distribution server 1 streams video content received from distributor terminal 2 to viewer terminal 3. If a viewer has a problem with the video content, they enter a comment and send it to video distribution server 1, which receives the comment. Video distribution server 1 analyzes the comment and extracts information about the video content problem. If a specific extracted problem occurs a specified number of times within a specified time period, it recognizes that a specific situation has occurred. Video distribution server 1 then sends control information to distributor terminal 2 to improve the specific situation.

Description

[Technical Field]

[0001] The present invention relates to a video distribution server that relays video images from multiple cameras and distributes live streaming broadcasts to multiple viewer terminals. [Background technology]

[0002] In recent years, the expansion of network environments such as mobile phone networks and the widespread use of imaging devices have made it possible for anyone to stream moving images regardless of the type of device used or the location. In particular, the development of online video conferencing systems has led to the use of online talk broadcasts and online lessons, in which video captured from multiple devices remotely connected to each other via a network is relayed and streamed.

[0003] Patent Document 1 discloses a content distribution server that receives video streams from multiple distributor terminals connected via a network, generates streaming content, and distributes the streaming content to viewer terminals. One of the editing functions of this content distribution server is a switcher function that allows switching between videos from multiple distributors. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7062328 Summary of the Invention [Problem to be solved by the invention]

[0005] In streaming content distribution such as online talk distribution or online lessons in which multiple distributors participate and which utilizes the content distribution server described in Patent Document 1, when a problem such as image or audio disturbances in the distributed video occurs, the distributors themselves have difficulty noticing the problem, resulting in slow response times. Also, even when a viewer notices a problem such as image or audio disturbances in the distributed video and reports the problem through chat on the streaming content distribution platform, the distributor does not respond until the viewer notices the problem in their comment, which often leaves the viewer dissatisfied. [Means for solving the problem]

[0006] According to the present invention, A computer-implemented video distribution method, comprising: receiving a video captured by an imaging device; streaming the video to a viewer terminal; receiving comments from viewers of the streaming distribution from a viewer terminal; analyzing the comments and extracting feedback regarding the viewing status of the video content; When the number of feedbacks relating to a specific situation from the feedback reaches a predetermined condition, controlling the imaging device to improve the specific situation; A video distribution method comprising the steps of: [Effects of the Invention]

[0007] According to the video distribution method of the present invention, when a problem such as a disturbance in the image or sound of the video being distributed occurs during streaming content distribution, the problem can be dealt with quickly and with less burden on the distributor by providing feedback from users, thereby enabling stable distribution of streaming content to viewers. [Brief explanation of the drawings]

[0008] [Figure 1]This is a diagram of the video distribution system. [Figure 2] This is a functional configuration diagram of the video distribution server 1. [Figure 3] 10 is a flowchart showing an example of the operation of the video distribution server 1. [Figure 4] FIG. 1 is a schematic diagram of distribution content; [Figure 5] 3 is a schematic diagram of a screen displayed on the display unit 304 of the viewer terminal 3. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, identical or equivalent components, parts, processes, and signals shown in each drawing will be assigned the same reference numerals, and duplicate explanations will be omitted where appropriate. Furthermore, some components that are not important for the explanation will be omitted in each drawing. The following embodiments do not unduly limit the content of the present disclosure described in the claims, and not all of the components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0010] (Video distribution system configuration) Figure 1 shows the configuration of a video distribution system according to the present disclosure. The video distribution system comprises a video distribution server 1, a distributor terminal 2, and a viewer terminal 3, all connected via a network NW. The video distribution server 1 receives distributor videos from multiple distributor terminals 2 via the network NW, generates streaming content, and distributes the streaming content to the viewer terminal 3. For convenience, Figure 1 shows one distributor terminal 2 and one viewer terminal 3, but each system is actually made up of multiple devices.

[0011] (Configuration of video distribution server 1) 2 shows the hardware configuration and functional configuration of video distribution server 1 according to the present disclosure. Video distribution server 1 includes a control unit 101, a storage unit 102, and a communication unit 103. The control unit 101 is realized by a processor reading out an application program stored in storage unit 102, executing the processing, and transmitting and receiving data to and from a distributor terminal 2 and a viewer terminal 3 connected to the network NW via communication unit 103. This realizes the video distribution server 1 as an information processing device that performs information processing. Note that video distribution server 1 is a computer installed on a network, but it may also be a server virtually configured as a so-called cloud, or may be configured by a terminal with communication capabilities such as a personal computer or smartphone.

[0012] The storage unit 102 is a storage device for permanently or temporarily storing data and programs, and is configured with a volatile memory, a hard disk, an SSD, etc. The communication unit 103 communicates with other computers via a network NW using a wired or wireless communication standard. The network NW is configured with a mobile communication network, a wireless LAN, or other short-range wireless communication, and the communication unit 103 is configured with an interface that can communicate with the network by inputting and outputting signals.

[0013] (Control unit 101 of video distribution server 1) The control unit 101 is configured by the following functional units: The processor 101 reads out an application program stored in the storage unit 102 and executes the processing, thereby realizing each functional unit.

[0014] (Memory unit 102 of video distribution server 1) The storage unit 102 stores and holds a video distribution program for providing a streaming content distribution service and various data required for processing. The video distribution program is executed by a processor and operates as each functional unit of the control unit 101 to realize the function of a video distribution server.

[0015] (Communication unit 103 of video distribution server 1) The communication unit 103 provides an interface for communication during the operation of each functional unit of the control unit 101. When each functional unit of the control unit 101 communicates with the distributor terminal 2 and the viewer terminal 4, the communication unit 103 acts as an intermediary to communicate with the network NW.

[0016] (Configuration of distributor terminal 2) The distributor terminal 2 comprises a control unit 201, a storage unit 202, a communication unit 203, an imaging unit 204, and a display unit 205. The processor of the distributor terminal 2 functions as the control unit 201 by calling and executing an application program from the storage unit 202. The storage unit 202 stores data necessary for the processor of the distributor terminal 2 to execute the application program. The communication unit 203 provides an interface function when the user terminal 20 communicates with the video distribution server 1 via the network NW. The imaging unit 204 is made up of a camera and a microphone, and captures videos to be distributed by the streaming content distribution service. The display unit 205 displays videos captured by the imaging unit 204, and displays messages and data received from the video distribution server 1.

[0017] The hardware configuration of the distributor terminal 2 is preferably a mobile communication terminal such as a smartphone with an imaging function, but other possible devices include a digital camera with a communication function, or a tablet or laptop with an imaging function. The hardware constituting each functional component of the distributor terminal 2 includes a control unit 201 made up of a processor, a storage unit 202 made up of volatile memory, a hard disk, an SSD, etc., a communication unit 203 made up of a communication interface for a mobile communication network, a wireless LAN, or other short-range wireless communication, etc., an imaging unit 204 made up of a camera and a microphone, and a display unit 205 made up of an LCD screen, preferably with a touch panel superimposed thereon, which also serves as an input means.

[0018] (Configuration of viewer terminal 3) The viewer terminal 3 comprises a control unit 301, a memory unit 302, a communication unit 303, and a display unit 304. The processor of the viewer terminal 3 functions as the control unit 301 by calling up and executing an application program from the memory unit 302. The memory unit 302 stores data necessary for the processor of the viewer terminal 3 to execute the application program. The communication unit 303 provides an interface function when the viewer terminal 3 communicates with the video distribution server 1 via the network NW. The display unit 304 displays videos distributed from the video distribution server 1, and displays a screen for inputting messages and data received from the video distribution server 1, as well as messages to the video distribution server 1.

[0019] The hardware configuration of the viewer terminal 3 is preferably a mobile communication terminal such as a smartphone, but other possible devices include a tablet or laptop computer equipped with communication functions. As for the hardware constituting each functional component of the viewer terminal 3, the control unit 301 is composed of a processor, the storage unit 302 is composed of a volatile memory, a hard disk, an SSD, etc. The communication unit 303 is composed of a communication interface for a mobile communication network, a wireless LAN, or other short-range wireless communication, etc. The display unit 304 is a display using a liquid crystal screen, and preferably has a touch panel superimposed thereon, which also serves as an input means.

[0020] Next, each functional unit realized by the control 101 of the video distribution server 1 will be described.

[0021] (Video Receiving Unit 1011) The video receiving unit 1011 performs a process of receiving videos from multiple distributor terminals 2 via the network NW. The video receiving unit 1011 is configured to be able to simultaneously and independently receive distributor videos from multiple distributor terminals 2. The video receiving unit 1011 temporarily buffers and stores the received videos in predetermined units in the storage unit 103. Specifically, the video is stored while being updated as needed so that the videos recorded in the buffer are several seconds to several tens of seconds long at predetermined intervals. The multiple distributor terminals 2 act as imaging devices for the video distribution server 1.

[0022] (Video Streaming Department 1012) The video distribution unit 1012 creates content for streaming distribution based on the received video. The video distribution unit 1012 reads out identification information of the distributor terminal 2 and identification information of the subject captured by the distributor terminal 2, which are pre-registered in the storage unit 103. The video distribution unit 1012 creates content by arranging the received videos from multiple distributors based on the arrangement information. The arrangement information is stored in the storage unit 103 and is composed of a screen layout within the content and information on which position on the screen layout the identification information of the distributor terminal is associated. The arrangement information is edited on the screen of an administrator terminal connected to the video distribution server 1 (not shown) and recorded in the storage unit 103. The video distribution unit 1012 transmits the created content to the viewer terminal 3 via the network NW.

[0023] FIG. 4 shows an example of content delivered to the viewer terminal 3. In this example, the content screen is divided into four sections, with videos of "Mr. A," "Mr. B," "Mr. C," and "Mr. D" arranged counterclockwise from the top left of the page. The arrangement is performed by the video delivery unit 1012 based on the arrangement information and the identification information of the broadcaster terminal and the subject corresponding to each portion of the screen layout in the arrangement information. For example, the arrangement information includes that a video of Mr. A captured by broadcaster terminal A will be arranged in the top left of the page, and the identification information of the corresponding broadcaster terminal and the subject for each portion of the screen layout is also included. The arrangement information is stored in the storage unit 103 and can be referenced at any time. Note that in this embodiment, the subject is a person, but the subject can be any object, such as an animal or a landscape. The arrangement information and identification information can also be composed of information corresponding to the subject.

[0024] (Comment receiving unit 1013) The comment receiving unit 1013 accepts comments input by viewer users from the viewer terminal 3. The viewer terminal 3 displays the received content on the display unit 304. The screen displayed on the display unit 304 at this time is shown in FIG. 5. This screen is composed of a content display unit 51, a chat display unit 52, and a chat input unit 53. The content display unit 51 displays the video of the received content, and the chat display unit 52 displays comments from each user in chat format. The chat input unit 53 is a field where the viewer user inputs a comment in text, and the user inputs the comment using a software keyboard displayed on the display unit 304. The input comment is transmitted to the video distribution server 1 via the network NW by the communication unit 303 of the viewer terminal 3, and the comment receiving unit 1013 receives the comment and stores it in the memory unit 103.

[0025] (Feedback extraction unit 1014) The feedback extraction unit 1014 reads out the comments from the storage unit 103 and analyzes the comments. In analyzing the comments, it analyzes whether the image quality, sound, and imaging range of the content are mentioned, and if so, whether a specific subject is specified.

[0026] In specific comment analysis processing, the feedback extraction unit 1014 first performs a morphological analysis of the received comments to extract words. The feedback extraction unit 1014 checks for relevant words or phrases by referencing the content placement information in the storage unit 103 and the identification information of the distributor terminal and the subject. The feedback extraction unit 1014 also registers words and phrases that express the quality of the content and the viewer's expectations for the content, and the feedback extraction unit 1014 compares the similarity between these words and phrases and the morphologically analyzed words, and clusters expressions similar to words and phrases that express attention to or expectations for the content.

[0027] As a specific example, when there is some kind of problem with the audio, similar expressions in the comments such as "the sound is too quiet," "the voice is too quiet," "it's hard to hear," and "it's hard to hear" are clustered by the feedback extraction unit 1014. Similarly, when there is a problem with blown-out highlights in the video, similar expressions in the comments such as "it's blown out," "it's too bright," and "it's too bright" are clustered by the feedback extraction unit 1014. Furthermore, when a user wants to zoom in on a specific subject, similar expressions in the comments such as "zoom in on person A" and "get closer to person A" are clustered by the feedback extraction unit 1014.

[0028] The feedback extraction unit 1014 counts the number of clustered specific situations that occur within a predetermined time period based on the results of analyzing words and phrases that express cautions or requests regarding the content. If a predetermined number of specific situations are detected within the predetermined time period, the feedback extraction unit 1014 determines that control of the imaging device is necessary and notifies the imaging device control unit 1015.

[0029] Another embodiment of the feedback extraction unit 1014 analyzes comments using a large-scale language model. The large-scale language model may be included in the control unit 101 or may be used as an external server via the network NW. The feedback extraction unit 1014 detects specific situations within a predetermined time by sending processing prompts and comments to the large-scale language model and receiving processing results from the large-scale language model. Specific examples of prompts are as follows: (a) Please check the comments you will be sending to see if there are any issues or requests from users regarding the audio, image quality, or image capture range of the content, and extract the specific situation. (b) Please identify the specific situation for which the subject is being photographed. (c) Even if the expressions of specific situations are different, similar situations should be considered the same. (d) Please report if a certain number of similar situations are detected within a certain time period.

[0030] The feedback extraction unit 1014 receives the processing results of the above prompt from the large-scale language model, and if a specific situation is detected within a predetermined time, it determines that control of the imaging device is necessary and notifies the imaging device control unit 1015.

[0031] (Imaging device control unit 1015) The imaging device control unit 1015 determines the control to be applied to the corresponding imaging device based on the specific situation notified by the feedback extraction unit 1014. In this embodiment, the broadcaster terminal 2 operates as an imaging device, and the broadcaster terminal is associated with the identification information of the person who is the subject of the image, so it is possible to identify which broadcaster terminal 2 corresponds to the notification based on the notified information. A control method for the notified specific situation is registered in advance in the storage unit 103, and the imaging device control unit 1015 obtains the control method corresponding to the notification from the storage unit 103. The imaging device control unit 1015 transmits an instruction to control the imaging unit 204 to the broadcaster terminal 2 to be controlled.

[0032] As a specific example of an instruction, if the specific situation is "Person A's voice is hard to hear," and the feedback extraction unit 1014 notifies "Distributor: Person A" and "Situation: Low volume," information is generated as an instruction to control the broadcaster terminal 2 corresponding to broadcaster A to increase the microphone gain of the imaging unit 204. If the notified specific situation is about image quality, an instruction to change the camera parameters of the imaging unit 204 is generated for the corresponding broadcaster terminal. With regard to image quality, an instruction to filter the captured video may be generated, not limited to camera parameters, depending on the situation. If the notified specific situation is a change in the imaging range and a desire to zoom in on a specific subject, if the corresponding broadcaster terminal 2 can physically change the angle of view, an instruction to change the imaging direction (e.g., pan or tilt) and change the focal zoom is generated. If the angle of view cannot be changed, an instruction for so-called digital zoom, which cuts out and enlarges the range of the subject through subject recognition, is generated.

[0033] (Distributor notification unit 1016) The distributor notification unit 1016 receives comments from the viewer terminal 3 within a predetermined time period after the image capture device control unit 1015 transmits the control information, and analyzes the comments. The analysis method is the same as that of the feedback extraction unit 1014. If the analysis shows that the feedback extraction unit 1014 has already detected a situation similar to that detected by the image capture device control unit 1015, the distributor notification unit 1016 determines whether a predetermined number of cases have been detected within the predetermined time period. If it is determined that the predetermined number of cases have been detected, the distributor notification unit 1016 notifies the distributor terminal 3 corresponding to the specific situation, or an administrator terminal (not shown) that manages the distribution, that the specific situation has occurred. The notification from the distributor notification unit 1016 is made by a message in a dedicated application for content distribution, or by email, SNS, etc.

[0034] The distributor can learn from the notification by distributor notification unit 1016 that there is a problem with their own distribution and that the problem has not been resolved by control by video distribution server 1. For example, if distributor terminal A receives a notification saying, "We have received reports that A's voice is difficult to hear, but control from the video distribution server has not resolved the problem," A will know that there is some kind of problem with the microphone on his or her own distributor terminal and can take action.

[0035] Next, the flow of operation of video distribution server 1 will be explained using the flowchart in Figure 3. As already explained, content is streamed from video distribution server 1 to viewer terminal 3 by video receiving unit 1011 and video distribution unit 1012 of video distribution server 1. The flow in Figure 3 explains the series of steps in which video distribution server 1 receives comments about the streaming distribution from viewer terminal 3, analyzes them, extracts feedback, and performs control.

[0036] (Step S01: Receive comments) The comment receiving unit 1013 accepts comments input by viewer users from the viewer terminal 3. The viewer terminal 3 displays the received content on the display unit 304. As explained above, the viewer terminal 3 inputs comments into the chat input section 53 on the screen shown in FIG. 5 displayed on the display unit 304. The input comments are transmitted by the communication unit 303 of the viewer terminal 3 to the video distribution server 1 via the network NW, and the comment receiving unit 1013 receives the comments and stores them in the memory unit 103.

[0037] (Step S02: Feedback extraction) The feedback extraction unit 1014 reads out comments from the storage unit 103 and analyzes the comments. In analyzing the comments, it analyzes whether the image quality, sound, and imaging range of the content are mentioned, and if so, whether a specific subject is specified. From the analysis results, expressions similar to words and phrases expressing attention or requests regarding the content are clustered.

[0038] (Step S03: Determine predetermined conditions) The feedback extraction unit 1014 counts the number of clustered specific situations that occur within a predetermined time period based on the results of analyzing words and phrases that express cautions or wishes regarding the content. If a predetermined number of specific situations are detected within the predetermined time period, the feedback extraction unit 1014 determines that control of the imaging device is necessary, notifies the imaging device control unit 1015, and proceeds to step S04. If a predetermined number of specific situations are not detected within the predetermined time period, it is determined that there is no problem with distribution and the process ends.

[0039] (Step S04: Imaging device control) The imaging device control unit 1015 determines the control to be applied to the corresponding imaging device based on the specific situation notified by the feedback extraction unit 1014. In this embodiment, the broadcaster terminal 2 operates as an imaging device, and the broadcaster terminal is associated with the identification information of the person who is the subject of the image, so it is possible to identify which broadcaster terminal 2 corresponds to the notification based on the notified information. A control method for the notified specific situation is registered in advance in the storage unit 103, and the imaging device control unit 1015 obtains the control method corresponding to the notification from the storage unit 103. The imaging device control unit 1015 transmits an instruction to control the imaging unit 204 to the broadcaster terminal 2 to be controlled.

[0040] (Step S05: Determine if the situation is improving) The distributor notification unit 1016 receives comments from the viewer terminal 3 within a predetermined time period after the image capture device control unit 1015 transmits the control information, and analyzes the comments. The analysis method is the same as that of the feedback extraction unit 1014. If the analysis shows that the feedback extraction unit 1014 has already detected a situation similar to that of the image capture device control unit 1015, it determines whether a predetermined number of cases have been detected within the predetermined time period. If it is determined that the predetermined number of cases have been detected, the process proceeds to step S06. If the predetermined number of cases have not been detected within the predetermined time period, it is assumed that the specific situation has improved and the process ends.

[0041] (Step S06: Notification of status) The distributor notification unit 1016 notifies the distributor terminal 3 corresponding to the specific situation, or an administrator terminal (not shown) that manages distribution, that a specific situation has occurred. The notification from the distributor notification unit 1016 is made by a message in a dedicated application for content distribution, email, SNS, etc. The distributor can learn from the notification from the distributor notification unit 1016 that there is a problem with their distribution and that the problem has not been resolved by control by the video distribution server 1.

[0042] The video distribution server 1 repeats the above steps S01 to S06 while distributing content. As a result, even if a problem occurs with the content being distributed, the video distribution server 1 can immediately recognize the problem through comments from users and can respond automatically and quickly. When a distributor is filming and distributing the content themselves, it is difficult for the distributor to notice when there is a problem with their own distribution, but since the video distribution server 1 can correct the problem, the burden of paying attention to the occurrence of the problem is reduced. Furthermore, even if the corrections made by the video distribution server 1 do not improve the situation, the distributor can become aware of the problem and respond by receiving a notification from the video distribution server 1.

[0043] The above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, hard disk, or SSD, or in a storage medium such as an IC card, SD card, or DVD. [Explanation of symbols]

[0044] 1. Video distribution server 101 Control section 1011 Video receiver 1012 Video Distribution Department 1013 Comment Receiving Department 1014 Feedback Extraction Unit 1015 Imaging device control section 1016 Streamer Notification Department 2. Streamer terminal 3 Viewer terminals

Claims

1. A computer-implemented video distribution method, comprising: receiving a video captured by an imaging device; streaming the video to a viewer terminal; receiving comments from viewers of the video from the viewer terminal; analyzing the comments and extracting feedback regarding the viewing status of the video; When the number of feedbacks relating to a specific situation from the feedback reaches a predetermined condition, controlling the imaging device to improve the specific situation; Video distribution methods including.

2. the specific situation is a defect in the image quality of the video; The control of the imaging device is performed by changing camera parameters. The video distribution method according to claim 1 .

3. a step of registering a correspondence between a subject in the video and the imaging device; further comprising the specific situation is a defect in image quality for a specific subject, the control of the imaging device is performed by changing camera parameters of the imaging device corresponding to the specific subject; The video distribution method according to claim 2.

4. The specific situation is a malfunction of the audio of the video; The control of the imaging device is performed by changing microphone parameters of the imaging device. The video distribution method according to claim 1 .

5. a step of registering a correspondence between a subject in the video and the imaging device; further comprising the specific situation is an audio defect regarding a specific subject, the control of the imaging device is performed by changing a microphone parameter of the imaging device corresponding to the specific subject; The video distribution method according to claim 4.

6. the specific situation is a desire to change the imaging range of the imaging device, The control of the imaging device is performed by changing camera parameters of the imaging device. The video distribution method according to claim 1 .

7. a step of registering a correspondence between a subject in the video and the imaging device; further comprising the specific situation is a desire to zoom in on a specific subject, the control of the imaging device is performed by changing camera parameters of the imaging device corresponding to the specific subject; The video distribution method according to claim 6.

8. a step of notifying a distributor terminal that the specific situation has not improved when feedback regarding the specific situation is further extracted after the control of the imaging device; The video distribution method according to claim 1 , further comprising:

9. a step of notifying a distributor terminal corresponding to the specific subject that the specific situation has not improved when feedback regarding the specific situation is further extracted after the control of the imaging device; The video distribution method according to any one of claims 3, 5 and 7, further comprising:

10. A video distribution program that causes a computer to execute the video distribution method according to any one of claims 1 to 7.

11. A video distribution program that causes a computer to execute the video distribution method described in claim 8.

12. A video distribution program that causes a computer to execute the video distribution method described in claim 9.

13. A video distribution device that executes the video distribution method described in any one of claims 1 to 7 using a processor.

14. A video distribution device that executes the video distribution method described in claim 8 using a processor.

15. A video distribution device that executes the video distribution method described in claim 9 using a processor.

Images