Real-time distribution support server based on automatic behavior recognition and its operation method
The real-time distribution sponsorship server addresses the lack of automatic linkage in conventional streaming systems by using behavior recognition to verify mission completion and process sponsorships, enhancing viewer engagement and content creativity.
Patent Information
- Application Number
- JP2024531736
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-30
- Filing Date
- 2022-11-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The conventional sponsorship system for real-time streaming lacks automatic linkage between broadcaster mission execution and viewer sponsorship, limiting the creation of engaging and creative content.
A real-time distribution sponsorship server that utilizes automatic behavior recognition through video and audio analysis to determine mission completion and processes sponsorship payments accordingly, potentially involving IoT devices for enhanced interaction.
This system enhances viewer engagement by automatically recognizing mission completion, increasing immersion and inducing more sponsorship, while eliminating the need for manual verification, and enabling novel content experiences.
Smart Images

Figure 2024543937000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a real-time distribution (broadcast) sponsorship server and an operating method thereof, and more particularly to a real-time distribution sponsorship server and an operating method thereof, in which a viewer of a real-time distribution grants a mission and a sponsorship fee is paid to a distributor when the distributor accomplishes the mission. [Background technology]
[0002] Recently, real-time internet distribution services have become popular in various fields. For example, internet distributors are conducting real-time individual distribution using domestic and international distribution platforms such as YouTube (registered trademark), Afreeca TV, Twitch (registered trademark), and Kakao TV.
[0003] Viewers can connect to the platform to watch broadcasts and participate in chats, and can support broadcasters by sponsoring them (e.g., star balloons or donations). For example, viewers can purchase sponsorship items (or cyber money, etc.) worth a certain amount through a real-time broadcasting platform (or real-time sponsorship platform) and sponsor a broadcaster using the purchased sponsorship items. After receiving the sponsorship, the broadcaster can pay a portion of the fees to the platform and then receive a settlement equivalent to the remaining sponsorship items.
[0004] Conventional viewer-participation real-time broadcasting was conducted in a way that viewers selected and uploaded text, images, and videos they wanted to share with sponsorship, and the broadcaster confirmed the sponsorship and shared the content with the viewers, thus forming a two-way communication, or viewers gave the broadcaster a desired mission, and when the broadcaster completed the mission, the viewers confirmed it and sponsored it. Looking closely at broadcasting platforms such as AfreecaTV and Twitch, broadcasters earn financial profits through such events, and viewers get a sense of accomplishment by seeing the text, images, and videos displayed in the broadcast window, and the broadcaster's actions.
[0005] However, the conventional sponsorship method for viewer-participation real-time distribution has limitations in creating more interesting and creative content because the distributor's mission execution and the viewer's sponsorship are not automatically linked. Summary of the Invention [Problem to be solved by the invention]
[0006] The present invention has been devised in light of the above-mentioned background, and aims to provide a real-time streaming sponsorship server and an operating method thereof, which allows a sponsor of real-time streaming to grant a desired mission to a streamer, and automatically recognizes and sponsors the streamer when the streamer completes the mission.
[0007] More specifically, the present invention aims to provide a real-time Internet distribution sponsorship server and an operating method thereof, which allows a viewer of real-time Internet distribution to select a specific mission and deposit a certain amount of sponsorship money or sponsorship items, and when the Internet distributor accomplishes the mission, the deposited sponsorship money or sponsorship items are automatically paid to the Internet distributor based on the results of behavior recognition such as motion recognition and voice recognition.
[0008] The objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Means for solving the problem]
[0009] To achieve the above object, a real-time streaming sponsorship server based on automatic behavior recognition according to one embodiment of the present invention includes a sponsorship management unit that collects information on missions and sponsorship money deposit information requested by viewers of Internet broadcasts to Internet broadcasters, and processes the payment of the sponsorship money when the mission is completed; a behavior recognition unit that recognizes the behavior of the Internet broadcaster based on at least one of video and audio of the Internet broadcaster and generates a behavior recognition result; and a mission completion determination unit that compares the behavior recognition result with information on the mission and determines whether the mission is completed.
[0010] The real-time distribution support server may further include an IoT control unit that controls a distribution assistance IoT device when the mission is completed.
[0011] The information about the mission may be the name of at least one of an action, an expression, and a posture that a viewer of the Internet broadcast requests the Internet broadcaster.
[0012] Also, the information on the mission may be at least one of a video and an image regarding an action requested by the viewer of the Internet broadcast to the Internet broadcaster.
[0013] And, according to one embodiment of the present invention, a real-time streaming sponsorship system based on automatic behavior recognition includes an IoT device for behavior recognition that collects at least one of video and audio of an internet broadcaster; and a server that collects information on missions and sponsorship deposit information requested by viewers of internet broadcasts to the internet broadcaster, recognizes the behavior of the internet broadcaster based on at least one of the information on the video and audio of the internet broadcaster, generates a behavior recognition result, compares the behavior recognition result with information on the mission to determine whether the mission has been completed, and processes the payment of the sponsorship money if the mission has been completed.
[0014] The IoT device for behavior recognition can acquire depth information for each part of the Internet broadcaster's body.
[0015] The server can control a delivery assistance IoT device when the mission is completed.
[0016] The real-time distribution system according to an embodiment of the present invention includes a distribution platform system, a distributor terminal that generates distribution content in which an internet distributor appears and transmits it to the distribution platform system, and a viewer terminal that receives the distribution content in the distribution platform system and provides the distribution content to a viewer. In this case, the distribution platform system collects information on the mission requested by the viewer to the internet distributor and sponsorship money deposit information from the viewer terminal, and processes the payment of the sponsorship money when it is determined that the mission is completed based on the distribution content.
[0017] The real-time streaming sponsorship server based on automatic behavior recognition according to an embodiment of the present invention includes a sponsorship management unit that collects information on a mission presented by one of an internet broadcaster and an internet broadcast viewer and information on a reward for the mission, and processes payment of the reward when the mission is completed; a behavior recognition unit that calculates a behavioral similarity between the internet broadcaster and the viewer based on the video of the internet broadcaster and the video of the viewer; and a mission completion determination unit that determines whether the mission is completed depending on whether the behavioral similarity falls within a predetermined range. The reward information is at least one of a prize money and channel points on the condition that the mission is completed.
[0018] And, the real-time streaming sponsorship server according to one embodiment of the present invention includes a sponsorship management unit that collects information on missions and sponsorship money deposit information requested by viewers of Internet streaming to Internet broadcasters, and processes the payment of the sponsorship money when the mission is completed; a recognition unit that recognizes at least one of objects and characters appearing in the video of the Internet broadcaster and generates a recognition result; and a mission completion determination unit that determines whether the mission is completed depending on whether the recognition result matches information on the mission.
[0019] And, an operating method of a real-time distribution support server according to one embodiment of the present invention includes a mission request information receiving step of receiving mission request information at a viewer terminal; a video / audio collecting step of collecting at least one of an internet distributor's video and audio; a behavior recognition step of generating a behavior recognition result of the internet distributor based on at least one of the internet distributor's video and audio; and a behavior-mission match determination step of determining whether the behavior recognition result matches the mission request information.
[0020] The method may further include a sponsorship deposit step of receiving sponsorship deposit information at the viewer terminal and matching it with the mission request information before the video / audio collection step, and may further include a sponsorship payment step of paying the sponsorship money to the Internet distributor in one of full payment and partial payment if the behavior recognition result matches the mission request information after the behavior-mission match determination step.
[0021] After the action-mission match determination step, the method may further include an auxiliary IoT control step of operating an auxiliary IoT device for delivery when the action recognition result matches the mission request information.
[0022] In the behavior recognition step, the behavior of the Internet distributor can be recognized through a neural internetwork trained using a deep learning technique. Effect of the Invention
[0023] According to one embodiment of the present invention, when an Internet distributor carries out a mission assigned by a viewer, any IoT device can be automatically operated in conjunction with the mission to provide viewers with innovative and interesting content, which has the effect of inducing more sponsorship from the distributor's perspective.
[0024] According to one embodiment of the present invention, whether an Internet broadcaster has completed their mission can be automatically evaluated, eliminating the need for the broadcaster or viewer to separately check whether the mission has been completed. This allows the viewer and broadcaster to focus solely on the situation, which can increase the level of immersion in the broadcast.
[0025] The effects that can be obtained by the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Brief description of the drawings]
[0026] [Figure 1] 1 is a block diagram showing a configuration of a real-time distribution system according to a first embodiment of the present invention. [Diagram 2] FIG. 13 is a reference diagram showing a screen for real-time personal distribution. [Diagram 3] 1 is a block diagram showing the configuration of a real-time distribution support server according to a first embodiment of the present invention. [Figure 4] FIG. 11 is a block diagram showing a configuration of a real-time distribution system according to a second embodiment of the present invention. [Diagram 5] FIG. 11 is a block diagram showing the configuration of a real-time distribution support server according to a second embodiment of the present invention. [Figure 6] FIG. 11 is a block diagram showing a configuration of a real-time distribution system according to a third embodiment of the present invention. [Figure 7] FIG. 13 is a block diagram showing the configuration of a real-time distribution support server according to a third embodiment of the present invention. [Figure 8] FIG. 11 is a block diagram showing a configuration of a real-time distribution system according to a fourth embodiment of the present invention. [Figure 9] 4 is a flowchart illustrating an operation method of a real-time distribution support server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0027] The advantages and features of the present invention, as well as the methods for achieving the same, will become apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be embodied in various different forms, and the embodiments are provided to fully disclose the present invention and fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Meanwhile, the terms used in the present specification are for the purpose of describing the embodiments, and are not intended to limit the present invention. In the present specification, the singular form includes the plural form unless otherwise specified in the text. When used in the specification, "comprises" and / or "comprising" refer to a reference element, step, operation, and / or element, and does not exclude the presence or addition of one or more other elements, steps, operations, and / or elements.
[0028] As used in this specification, expressions such as "first," "second," "initial," or "second" may modify various embodiments or components regardless of order and / or importance, and are only used to distinguish one embodiment or component from other embodiments or components, and do not limit the embodiment or component.
[0029] In describing the present invention, if it is determined that a detailed description of related prior art may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.
[0030] In this specification, an internet broadcaster (hereinafter referred to as "broadcaster") refers to any one of the host, panelist, and participant of an internet broadcast, and may be any one of the broadcasters appearing in the video transmitted through various broadcasting platforms, such as a PD (Play Director), creator, streamer, MC (Master of Ceremonies), BJ (Broadcasting Jockey), YouTuber, or broadcaster.
[0031] In this specification, content refers to various information and its contents provided through the Internet, computer communication, etc. Specifically, it refers to information and its contents such as characters, codes, voice, audio, images, videos, etc. that are processed in digital form and distributed for use in wired / wireless telecommunication networks.
[0032] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate an overall understanding of the present invention, the same reference numerals will be used to refer to the same means regardless of the drawing numbers.
[0033] FIG. 1 is a block diagram showing the configuration of a real-time distribution system according to a first embodiment of the present invention.
[0034] As shown in FIG. 1, the real-time distribution system 1 according to the first embodiment of the present invention includes a distributor terminal 10, a distribution platform system 20, a viewer terminal 30, a distribution camera 40, a microphone 50, and a real-time distribution support server (hereinafter referred to as the “server”) 100, and may further include a distribution assistance IoT device 60.
[0035] The distributor terminal 10 generates content for distribution in real time. However, the present invention is not limited thereto, and pre-generated content may be transmitted from the outside. The distributor terminal 10 transmits content created by a distributor to the distribution platform system 20. The distributor terminal 10 may generate content by receiving video and audio from a distribution camera 40 and a microphone 50 via wired or wireless communication, but the present invention is not limited thereto. For example, the distributor terminal 10 may include a built-in camera module for photographing the distributor and a built-in microphone for collecting the distributor's audio. The distributor terminal 10 transmits the generated content to the distribution platform system 20 so that streaming distribution can be performed.
[0036] The broadcaster terminal 10 can exchange information with the server 100 through wired / wireless communication. The server 100 can transmit information regarding an action that a viewer requests from a broadcaster (hereinafter, "mission request information") and sponsorship money / sponsorship item deposit information (hereinafter, "sponsorship money deposit information") to the broadcaster terminal 10. In this case, the broadcaster terminal 10 can display the mission request information and sponsorship money deposit information on a screen so that the broadcaster can acknowledge them.
[0037] The distribution platform system 20 receives content transmitted from a distributor terminal 10. The distribution platform system 20 may be connected to a plurality of distributor terminals 10 and may have a plurality of channels connected to each distributor terminal 10. A viewer terminal 30 connects to each channel to view content transmitted by the corresponding distributor terminal 10. The distribution platform system 20 may combine various effects set by the distributor terminal 10 with the received video, and may combine chat transmitted by the viewer terminal 30 with the video.
[0038] Also, the delivery platform system 20 can exchange information with the server 100 through wired / wireless communication. The delivery platform system 20 can synthesize information provided by the server 100 (e.g., information on the viewer's mission request and the broadcaster's mission execution result, or information that sponsorship money / sponsorship items have been provided to the broadcaster) into a video. The content synthesized by the delivery platform system 20 can be transmitted to a plurality of viewer terminals 30 connected to the same channel in the delivery platform system 20. Also, when the delivery platform system 20 receives mission request information and sponsorship money deposit information from the viewer terminal 30, the delivery platform system 20 can transmit the information to the server 100. The viewer terminal 30 connects to a specific channel of the delivery platform system 20 and provides the viewer with the content transmitted by the broadcaster terminal 10 connected to the specific channel. That is, the delivery platform system 20 transmits the content received from the broadcaster terminal 10 to the viewer terminal 30 connected to the corresponding channel. The distribution platform system 20 enables the viewer terminal 30 connected to the corresponding distribution channel to provide the content transmitted in real time from the distributor terminal 10 to the viewer. When multiple viewers connect to a specific channel to view distribution, multiple viewer terminals 30 exist in the real-time distribution system 1.
[0039] The viewer terminal 30, while connected to the server 100, transmits information regarding an action that the viewer requests the broadcaster to take ("mission request information") to the server 100. If multiple broadcasters are broadcasting or participating in a broadcast on one Internet broadcasting channel, the mission request information may include information regarding the broadcaster who will perform the action requested by the viewer. In addition, the viewer terminal 30 may deposit sponsorship money or sponsorship items for the broadcaster along with transmitting the mission. The viewer terminal 30 transmits sponsorship money or sponsorship item deposit information to the server 100. The viewer terminal 30 may be installed with an application (or app) for receiving broadcasts from the broadcasting platform system 20 as well as an application (or app) for communication between the viewer terminal 30 and the server 100.
[0040] In this specification, the distributor terminal 10 or the viewer terminal 30 may be a desktop personal computer (PC), a laptop personal computer, a netbook computer, a smartphone, a smartpad, a tablet personal computer (PC), a mobile phone, a video phone, a personal communication system (PCS), a global system for mobile communication (GSM), a personal digital cellular system (PDC), a personal handyphone system (PHS), a personal digital assistant (PDA), an international mobile telecommunication (IMT)-2000, a code division multiple access (CDMA)-2000, a W-code division multiple access (W-CDMA), a wireless broadband internet (Wibro) terminal, a portable multimedia player (PMP), a wearable device (e.g. smart glasses, a head-mounted device (HMD), etc.), or a smart watch. The wireless communication device may include at least one of all kinds of wireless communication devices such as wireless watches, desktop computers, smart TVs, etc., and may further include a camera module for taking pictures and a microphone for collecting audio.
[0041] The broadcasting camera 40 and microphone 50 provide video and audio to the broadcaster terminal 10 and server 100 via wired / wireless communication. The broadcaster terminal 10 generates content using the provided video and audio. The server 100 recognizes the broadcaster's actions based on the video and audio provided by the broadcasting camera 40 and microphone 50, and determines whether the broadcaster's actions match the mission given by the viewer.
[0042] The server 100 is connected wirelessly or wired to the distributor terminal 10, the distribution platform system 20, and the viewer terminal 30 to exchange information. The server 100 collects video and audio information of the distributor from the distributor terminal 10. If the distribution camera 40 and microphone 50 are IoT devices, the server 100 can collect the video and audio of the distributor from the distribution camera 40 and microphone 50 instead of the distributor terminal 10. The server 100 can also send a control signal to the distribution assistant IoT device 60 under a predetermined condition to operate the distribution assistant IoT device 60.
[0043] The server 100 transmits a list of actions that can be requested from the broadcaster (hereinafter, "mission list") to the viewer terminal 30, and the viewer terminal 30 can display the mission list on a screen to allow the viewer to select from it. The server 100 can transmit the mission list directly to the viewer terminal 30, or can transmit the mission list indirectly to the viewer terminal 30 via the distribution platform system 20. When the viewer selects one of the actions listed in the mission list, the viewer terminal 30 transmits information regarding the action that the viewer requests from the broadcaster ("mission request information") to the server 100.
[0044] The mission request information may be in various forms. There may be a method of selecting a promised mission from a mission list, and information on the content of the viewer's requested actions (e.g., posture, facial expression) or voice may be added. In addition, a method may be applied in which the viewer specifies prohibited actions or words (voice), and if the distributor performs the action or words (voice), the sponsorship money is deducted according to the number of times. Examples are described below.
[0045] (1) A viewer can register a desired posture as a mission, and the viewer can register the desired posture as a mission by adding an image or video showing the desired posture to the mission request information through the viewer terminal device 30. As a specific example, the viewer can take a video of himself / herself doing a handstand with a smartphone, capture the handstand scene to generate an image, and transmit this image from the viewer terminal device 30 to the server 100 to register it as a mission, requesting the distributor to imitate the action (handstand) shown in the image.
[0046] (2) The viewer uses an application installed on the viewer terminal device 30 to move their joints to make a heart shape with their arms (arm heart), registering this action as a mission, and the mission is transmitted to the distributor by being displayed on the distributor terminal 10 via the viewer terminal device 30 and the server 100.
[0047] (3) A viewer can specify a facial expression that is prohibited for a certain period of time as a mission. In order to check whether such a mission is accomplished, the action recognition model of the behavior recognition unit 120 must use a model capable of analyzing facial expressions. For example, a viewer can specify a mission that the broadcaster must not frown while playing a game for 30 minutes. If a broadcaster who is broadcasting a game frowns while playing the game, the server 100 may determine that the broadcaster has failed the mission, and the sponsorship money may be deducted or not paid.
[0048] (4) Missions can be assigned in relation to video and audio. The behavior recognition unit 120 is equipped with a video processing / analysis model, and whether a mission has been completed can be confirmed by identifying characters or objects that appear in the broadcaster's video. The video processing / analysis model can be used alone, or can be used in combination with a behavior recognition model or a voice recognition model to confirm whether a mission has been completed. For example, various mission contents can be assigned in relation to video and audio, such as playing a high-difficulty game without swearing (1,000 won is deducted if a mission fails), depositing 1,000 won in sponsorship money if the title of the topmost video displayed on YouTube contains the word "news", receiving sponsorship money according to the number of people who appear in a video when playing a video on YouTube, and depositing 1,000 won if a particular character is selected by an opponent or team member of the ally in a 5:5 PvP (Player vs. Player) League of Legends game.
[0049] The mission request information of the viewer terminal 30 may be transmitted to the server 100 through the distribution platform system 20. As described above, the viewer terminal 30 may deposit sponsorship money or sponsorship items for the broadcaster along with the transmission of the mission, in which case the viewer terminal 30 transmits sponsorship money / sponsorship item deposit information ("sponsorship money deposit information") along with the mission request information to the server 100. For example, a viewer may purchase a sponsorship item (or cyber money, etc.) through the distribution platform system 20 or the server 100 and deposit the sponsorship item along with the mission request, and at this time, information on the quantity of the deposited sponsorship items is transmitted from the viewer terminal 30 to the server 100.
[0050] As another example, the server 100 transmits a mission list to the distribution platform system 20. In this case, the distribution platform system 20 combines the mission list with the video and transmits it to the viewer terminal 30, and transmits the mission request information and sponsorship deposit information received by the viewer terminal 30 to the server 100.
[0051] The server 100 may combine one or more pieces of mission request information and sponsorship deposit information received from the viewer terminal 30 or the distribution platform system 20 and transmit the combined information to the broadcaster terminal 10 or the distribution platform system 20. When the broadcaster terminal 10 receives the mission request information and sponsorship deposit information, the broadcaster terminal 10 may display the mission request information and sponsorship deposit information on a screen so that the broadcaster may acknowledge the information. When the distribution platform system 20 receives the mission request information and sponsorship deposit information, the distribution platform system 20 may combine the corresponding information with an image and transmit the combined information to a plurality of viewer terminals 30 connected to the same distribution channel.
[0052] The server 100 collects at least one of a video and a voice of the broadcaster from the broadcaster terminal 10. As another example, the server 100 may collect at least one of a broadcast video and a voice from the broadcast platform system 20. As yet another example, the server 100 may collect at least one of a broadcast video and a voice from a broadcast camera 40 and a microphone 50. The server 100 recognizes the broadcaster's behavior based on at least one of the collected video and a voice of the broadcaster, and determines whether the behavior matches the behavior (mission) requested by the viewer. In this specification, "match" is considered to include a case where the behavior of the broadcaster recognized by the server 100 matches the behavior specified by the viewer as a mission, as well as a case where the probability that the broadcaster's behavior is the behavior selected (requested) as a mission by the viewer falls within a predetermined range. For example, if the action selected by the viewer as a mission is "making a heart with fingers," and the server 100 recognizes the broadcaster's action based on the video in which the broadcaster appears and finds that there is an 80% probability that the broadcaster performed the "making a heart with fingers" action, if the reference probability is 60% or higher, it is deemed that the action recognition result matches the mission request information.
[0053] As described above, the server 100 determines whether the behavior recognition result matches the mission request information, and if there is no match, collects video and audio of the broadcaster until the end of the broadcast or for a predetermined period of time to recognize the broadcaster's behavior.
[0054] If the server 100 determines that the behavior recognition result of the distributor matches the mission request information, the server 100 transmits information that the mission completion has been completed (hereinafter, "mission completion information") to the distribution platform system 20. The distribution platform system 20 can combine the mission completion information with an image and transmit it to a plurality of viewer terminal devices 30. The server 100 can also transmit information that sponsorship money / sponsorship items will be provided (hereinafter, "sponsorship payment information") together with the mission completion information to the distribution platform system 20, in which case the distribution platform system 20 can combine both the mission completion information and the sponsorship payment information with an image and transmit them to a plurality of viewer terminal devices 30. The server 100 also processes so that, when a distributor completes a mission requested by a viewer, the deposited sponsorship money and sponsorship items are provided to the distributor who completed the mission. That is, if the viewer terminal 30 deposits sponsorship money into an account managed by the server 100 or purchases a sponsorship item through the server 100, the server 100 processes the payment of the sponsorship money or sponsorship item to the distributor when the distributor completes the mission. Meanwhile, if the distribution platform system 20 manages the deposit of sponsorship money or the viewer terminal 30 purchases a sponsorship item through the distribution platform system 20, the server 100 transmits a message to the distribution platform system 20 requesting payment of the sponsorship money or sponsorship item when the distributor completes the mission.
[0055] Meanwhile, the real-time distribution system according to the first embodiment of the present invention may further include a distribution assistant IoT device 60. In this case, the server 100 controls the distribution assistant IoT device 60. The distribution assistant IoT device 60 is a device to which the Internet of Things is applied. Incidentally, the Internet of Things means that a plurality of things are connected to each other based on the Internet. Things connected to each other based on the Internet can exchange information with each other.
[0056] For example, the distribution assistance IoT device 60 according to the present invention may be any one of an electric fan, a blower, an air conditioner, a boiler, a light, a humidifier, a dehumidifier, an air purifier, a robot vacuum cleaner, a smartphone, a PC (including a separate computer as well as a distribution computer), a speaker, a monitor (including a distribution monitor), a camera, an outlet, a multi-tap, a weighing scale, various exercise equipment such as a treadmill or a cycle machine, a printer, a fax machine, a multifunction machine, various power switch modules, a power adapter, a door lock, a videophone, a TV receiver (set-top box), a vehicle, a black box, a navigation system, an infrared sensor, a feeding device for companion animals, a play device for companion animals, and a money gun (money gun, cash gun) to which the Internet of Things is applied, and is not limited to the listed examples.
[0057] The server 100 may operate the distribution assistance IoT device 60 under a specific condition. For example, when a broadcaster performs a mission (dance) requested by a viewer in midsummer, the server 100 may operate an electric fan or an air conditioner 60 to which the Internet of Things is applied. As another example, when a broadcaster performs a mission requested by a viewer, the server 100 may operate the money gun 60 to discharge bills stored inside the money gun 60. Incidentally, a money gun is a device that can store bills inside and discharges bills stored inside to the outside when a trigger is pulled or a control signal is received. The server 100 may operate the money gun 60 to discharge bills by the amount of sponsorship provided to the broadcaster, or may control the money gun 60 to discharge a certain amount of bills simply for the visual effect of receiving sponsorship. In addition, when a broadcaster speaks a specific word, the broadcaster's keyboard may stop working or the broadcaster's monitor screen may become invisible for 10 seconds.
[0058] In another embodiment of the present invention, a mission may be given in advance, and the broadcaster may imitate the actions of the viewer during broadcast, or the viewer may imitate the broadcaster.
[0059] (1) In the case where the mission request information is "imitate the behavior of the viewer (specified viewer is designated)", the server 100 can determine whether the mission has been completed by receiving the viewer's video at the target viewer terminal 30 or receiving the target viewer's video through the distribution platform system 20 and comparing the viewer's behavior in the viewer's video with the broadcaster's behavior in the broadcaster's video. In this case, the behavior recognition unit 120 of the server 100 can calculate the similarity of the behavior (movement) using a behavior (movement) similarity comparison model, and the mission completion determination unit 130 can determine that the mission has been completed if the calculated similarity is equal to or greater than a predetermined reference value. The behavior (movement) similarity comparison model can be a model that obtains a posture sequence from the movement (joint movement) of each subject and then calculates the similarity between the sequences by applying a DTW (Dynamic Time Warping) algorithm. In this case, the viewer can compare his / her own behavior / voice with the broadcaster's behavior / voice by placing the broadcaster's video and the viewer's video on the left and right through a distribution reception application (app) installed in the viewer terminal 30. Furthermore, a viewer may view a broadcast video through the application in a manner that arranges an enlarged image of the viewer and the images of the broadcaster and other viewers in a grid pattern (video conference method), and a broadcast screen can be configured with one broadcaster and N viewers. When a broadcaster broadcasts content in which the broadcaster imitates a viewer, the broadcaster can specify a specific viewer to imitate. The broadcast screen may be configured in a manner that arranges the viewer and broadcaster on the left and right, or displays the viewer and broadcaster overlapping each other.
[0060] The server 100 can determine whether the image of the distributor matches the image of the viewer during a certain period of time designated by the distributor or the viewer, and if it determines that there is a match, it provides sponsorship money to the distributor.
[0061] (2) In the case of broadcasting content in which a viewer imitates the broadcaster (see FIG. 2), the broadcaster transmits mission request information ("viewer imitates the broadcaster's actions," "viewer imitates the broadcaster's words") and sponsorship (or prize) deposit information to the sponsorship management unit 110 of the server 100 directly or via the broadcasting platform system 20 through the broadcaster terminal 10. The server 100 collects the broadcaster's video / audio from the broadcaster terminal 10 and the viewer's video / audio from the viewer terminal 30, and can determine the viewer who completed the mission through real-time behavior / audio analysis and provide sponsorship (prize) to the relevant viewer. When the sponsorship management unit 110 of the action server 100 receives mission completion information from the mission execution determination unit 13 of the server 100, it transmits the mission completion information to the broadcasting platform system 20. The broadcasting platform system 20 can combine the mission completion information with the video and transmit it to multiple viewer terminals 30. The server 100 can utilize the behavioral similarity comparison model detailed in (1) to determine whether the mission has been completed. In addition, the server 100 can control the viewer IoT device 61 via a wired / wireless network such as the Internet through the IoT control unit 140 according to a predetermined condition such as the completion or failure of the mission, when the server 100 is given control authority over the viewer IoT device 61 in advance.
[0062] A specific example of (2) will be described. A broadcaster transmits mission request information of "viewers imitate the broadcaster's words" to the sponsorship management unit 110 through the broadcaster terminal 10, and a certain amount of prize money is accumulated in the sponsorship management unit 110. When a broadcaster says to a viewer through broadcasting, "Let's start today on a positive note. Shall we all say it together? Let's do our best!", the viewer says "Let's do our best!" on the viewer's smartphone or PC. The server 100 or the viewer terminal 30 collects video and audio of the broadcaster and the viewer from the broadcaster terminal 10 and the viewer terminal 30. The server 100 or the viewer terminal 30 recognizes the voice and calculates the voice similarity by comparing the voice recognition result of the broadcaster with the voice recognition result of the viewer, and if the similarity is equal to or greater than a predetermined value, it determines that the mission has been completed. When it is determined that several viewers have completed the mission, the sponsorship management unit 110 processes so that a predetermined amount of prize money is paid to the viewers who have completed the mission. In addition, a distributor who creates exercise content may offer a certain amount of cash reward to a viewer who exercises in the same way as the viewer. In this way, the system can be used for various campaigns.
[0063] Meanwhile, internet distribution systems have a channel point system, where channel points are accumulated through activities such as the viewer's broadcast viewing time, channel subscriptions, and broadcaster following. Viewers can use channel points to receive rewards. Examples of rewards include the right to use specific emoticons, the right to edit emoticons, sending messages in subscriber-only mode, and highlighting personal messages. The channel points required for rewards and the specific content of the rewards can be set by the broadcaster who is conducting the broadcast on the channel. If a viewer broadcasts content that imitates the broadcaster, the broadcaster can provide not only prize money but also channel points or items that are valid on the broadcasting platform to viewers who have succeeded in completing the mission.
[0064] FIG. 3 is a block diagram showing the configuration of a real-time distribution support server according to a first embodiment of the present invention.
[0065] 3, the real-time streaming sponsorship server 100 according to the first embodiment of the present invention includes a sponsorship management unit 110, a behavior recognition unit 120, and a mission execution determination unit 130. The server 100 may further include an IoT control unit 140.
[0066] The sponsorship management unit 110 provides a list (mission list) of actions (missions) that a viewer can request from a distributor to the viewer terminal 30 so that the viewer can select a mission. The sponsorship management unit 110 may provide the mission list directly to the viewer terminal 30 or indirectly through the distribution platform system 20.
[0067] Examples of missions include "making a heart with arms," "making a heart with fingers," "a cheer pose," "a horseback riding pose," "flower petal pose," "a happy expression," "laughing," "a sad expression," "pretending to cry," "dancing," "singing a song," "giving a congratulatory greeting," "saying a specific word (the word is included in the mission request information)," "making a male voice," "making a female voice," "getting angry," "pretending to be a gangster," "squatting for one minute," etc. Whether such a mission has been accomplished is determined by the operation of the behavior recognition unit 120 and the mission execution determination unit 130, and artificial intelligence may be used for the recognition and determination.
[0068] The sponsorship management unit 110 collects mission request information and sponsorship deposit information from the viewer terminal 30 or the distribution platform system 20, and matches the mission request information with the sponsorship deposit information. The sponsorship management unit 110 integrates the mission request information and sponsorship deposit information collected during a predetermined time, and transmits the integrated information to the distributor terminal 10 and the distribution platform system 20. The sponsorship management unit 110 also transmits the mission request information to the mission execution determination unit 130.
[0069] When the mission completion information is transmitted from the mission execution determination unit 13, the sponsorship management unit 110 transmits the mission execution completion information to the distribution platform system 20. The distribution platform system 20 can combine the mission execution completion information with an image and transmit the combined information to a plurality of viewer terminal devices 30. The sponsorship management unit 110 can transmit information that sponsorship money / sponsorship items will be provided ("sponsorship payment information") together with the mission execution completion information to the distribution platform system 20. In this case, the distribution platform system 20 can combine both the mission execution completion information and the sponsorship payment information with an image and transmit the combined information to a plurality of viewer terminal devices 30.
[0070] In addition, the sponsorship management unit 110 processes so that the sponsorship money or sponsorship item deposited by matching with the mission request information upon completion of the mission is paid to the distributor who completed the mission. That is, when the viewer terminal device 30 deposits sponsorship money in an account managed by the sponsorship management unit 110 or the viewer terminal device 30 purchases a sponsorship item through the sponsorship management unit 110, the sponsorship management unit 110 processes so that all or a part of the sponsorship money or sponsorship item matched with the corresponding mission is paid to the distributor upon completion of the mission of the distributor. Meanwhile, when the distribution platform system 20 manages the deposit of sponsorship money or the viewer terminal device 30 purchases a sponsorship item through the distribution platform system 20, the sponsorship management unit 110 transmits a message requesting the distribution platform system 20 to pay the sponsorship money or sponsorship item matched with the corresponding mission upon completion of the mission of the distributor.
[0071] On the other hand, even if the mission is completed, the sponsorship management unit 110 can differentially provide sponsorship money and sponsorship items to the distributors depending on the probability that the distributor performed the action requested in the mission ("action probability").
[0072] In addition, the sponsorship management unit 110 transmits mission completion information to the IoT control unit 140 when the IoT control unit 140 is included in the server 100. At this time, the sponsorship management unit 110 can transmit sponsorship payment information to the IoT control unit 140 together with the mission completion information.
[0073] The behavior recognition unit 120 collects at least one of the video in which the broadcaster appears and the voice of the broadcaster from the broadcaster terminal 10, and recognizes the broadcaster's behavior (including facial expressions and posture) through analysis. For example, the behavior recognition unit 120 can classify the posture of the broadcaster appearing in the video collected from the broadcasting camera 40 by frame, and recognize the broadcaster's behavior based on the change in posture as the sequence progresses.
[0074] The behavior recognition unit 120 may collect either the video or audio of the distributor from the distribution platform system 20 or the distribution camera 40 and microphone 50 instead of the distributor terminal 10 .
[0075] In addition, the behavior recognition unit 120 may recognize the behavior of the broadcaster by using a neural network trained through a deep learning technique. For example, the behavior recognition unit 120 may recognize the broadcaster's behavior by using a behavior recognition model generated by defining a graph structure in which joints constituting a framework are nodes and bones connecting the joints are edges, and connecting the same nodes along a time axis, and training a graph convolutional neural network (GCN). In addition, the behavior recognition unit 120 may recognize the broadcaster's voice by converting voice into text using a voice recognition model to which a deep learning technique such as a Long Short-Term Memory (LSTM) neural network is applied. The voice recognition model (voice recognition model) used by the behavior recognition unit 120 may include a general voice recognition model, but may also include a model in which voice features are extracted to create a feature vector and an acoustic model is applied to the feature vector to classify a pattern. For example, the behavior recognition unit 120 may recognize voice features or voice patterns such as gender, age, and state of pronunciation from the broadcaster's voice. Methods for extracting voice features include Linear Predictive Coefficient, Fast Fourier Transform (FFT), Cepstrum, Mel Frequency Cepstral Coefficient (MFCC), and frequency band energy. Acoustic models (pattern classification models) can be constructed using deep learning.
[0076] The behavior recognition unit 120 can recognize the behavior of the distributor by combining the behavior recognition result and the voice recognition result. The behavior recognition unit 120 may also recognize the behavior of the distributor by using a behavior recognition model that is an ensemble of a behavior recognition model and a voice recognition model. Examples of deep learning techniques that the behavior recognition unit 120 can use to recognize behavior and voice include techniques such as HMM (Hidden Markov Model), CNN (Convolutional Neural Network), GCN (Graph Convolutional Neural Network), DNN (Fully-Connected Deep Neural Network), DBN (Dynamic Bayesian Network), CRF (Conditional Random Field), Ensemble TS-LSTM, ST-GCN, AS-GCN, Wav2Vec, and SincNet.
[0077] The behavior recognition unit 120 may include a separate learning module 121. In this case, the learning module 121 receives the video and audio of the broadcaster as input, and trains a deep learning network using the names of behaviors that can be used in the mission list as labels (output) to generate a behavior recognition model.
[0078] The behavior recognition unit 120 recognizes the behavior of the distributor using at least one of the video and audio collected from the distribution camera 40 and microphone 50, and transmits the behavior recognition result to the mission execution determination unit 130. The behavior recognition result may be the name of the behavior, or may be a form in which the name of the behavior is matched with the probability value of performing that behavior ("behavior probability"). For example, the behavior recognition result may be "making a heart with fingers" or "making a heart with fingers, 80%."
[0079] Meanwhile, the behavior recognition unit 120 may be additionally equipped with an image analysis model (object detection, object recognition, optical character recognition, shore-line detection, sea-land segmentation) learned by a deep learning technique as well as a motion recognition model and a voice recognition model. Thus, the behavior recognition unit 120 may detect not only the broadcaster's posture and facial expression but also the number of objects and words / sentences appearing in the image, and transmit the detected contents to the mission execution determination unit 130 to determine whether the mission has been completed. For example, in the case of outdoor broadcasting, the content of the mission request information may be a visit to a specific franchise cafe. In this case, when the broadcaster goes in front of the cafe, the behavior recognition unit 120 may detect and recognize a word on the signboard of the cafe appearing in the broadcaster's image, and the mission execution determination unit 130 may determine that the mission has been completed if the word is included in the mission request information. In this way, it is possible to place a doll in the studio (0.01 won per second) and determine whether or not to complete a mission, such as going into water.
[0080] In addition, when the mission request information is "mimic the behavior of the viewer (a specific viewer is designated)" or "the viewer mimics the behavior of the distributor", the behavior recognition unit 120 may directly receive the viewer's video from the target viewer terminal 30 or receive the viewer's video through the distribution platform system 20, and may compare the viewer's behavior (movement) appearing in the viewer's video with the distributor's behavior (movement) appearing in the distributor's video to calculate the similarity. In this case, the behavior recognition unit 120 may calculate the similarity of the behavior (movement) using a behavior (movement) similarity comparison model, and the mission execution determination unit 130 may determine that the mission execution is completed when the calculated similarity is equal to or greater than a predetermined reference value. The behavior (movement) similarity comparison model used by the behavior recognition unit 120 may be a model that obtains a posture sequence from the movement (joint movement) of each subject and then calculates the similarity between the sequences by applying a DTW (Dynamic Time Warping) algorithm.
[0081] The mission execution determination unit 130 compares the mission request information transmitted from the sponsorship management unit 110 with the behavior recognition result transmitted from the behavior recognition unit 120 to determine whether there is a match. If there is a behavior in the mission request information that matches the behavior recognition result, the mission execution determination unit 130 transmits information that the mission execution for the specific mission has been completed (mission execution completion information) to the sponsorship management unit 110. At this time, the mission execution determination unit 130 can transmit the behavior probability in the behavior recognition result to the sponsorship management unit 110 together with the mission execution completion information.
[0082] When the real-time distribution system includes the distribution assistant IoT device 60, the IoT control unit 140 controls the distribution assistant IoT device 60. That is, the IoT control unit 140 can generate a control signal for operating the distribution assistant IoT device 60 under a predetermined condition and transmit the control signal to the distribution assistant IoT device 60 via wired / wireless communication. For example, when the distribution assistant IoT device 60 is an air conditioner, the IoT control unit 140 can operate the air conditioner 60 for a certain time upon receiving mission completion information from the sponsorship management unit 110. Also, when the distribution assistant IoT device 60 is a money gun, the IoT control unit 140 can receive mission completion information and sponsorship payment information from the sponsorship management unit 110 and control the money gun 60 to dispense a certain amount of bills.
[0083] FIG. 4 is a block diagram showing the configuration of a real-time distribution system according to a second embodiment of the present invention, and FIG. 5 is a block diagram showing the configuration of a real-time distribution support server according to the second embodiment of the present invention.
[0084] In the second embodiment of the present invention, the real-time distribution support server 100' can improve the accuracy of behavior recognition by collecting video and audio of the broadcaster using a separate IoT device 70 for behavior recognition to recognize the broadcaster's behavior and performing behavior recognition based on the collected video and audio.
[0085] As shown in FIG. 4, the real-time distribution system 1' according to the second embodiment of the present invention includes a distributor terminal 10, a distribution platform system 20, a viewer terminal 30, a distribution camera 40, a microphone 50, a real-time distribution support server (hereinafter referred to as "server") 100', and an IoT device for behavior recognition 70, and may further include a distribution assistant IoT device 60.
[0086] The real-time delivery system 1 and the real-time delivery system 1' differ in the method of collecting the video or audio of the deliverer for behavior recognition, and may also differ in the amount of information in the video data and the method of processing it. The following mainly describes the differences between the real-time delivery system 1 and the real-time delivery system 1'.
[0087] The server 100' collects at least one of the video and audio of the distributor from the IoT device 70 for behavior recognition in order to recognize the behavior of the distributor. The IoT device 70 for behavior recognition may be a camera or microphone to which Internet of Things technology is applied, and may be an IoT device equipped with both a camera and a microphone. The IoT device 70 for behavior recognition is located near the distributor terminal 10 and collects the video and audio of the distributor separately from the camera 40 or microphone 50 for distribution, and provides them to the server 100'.
[0088] The IoT device 70 for behavior recognition may be a 3D depth camera. Incidentally, 3D depth camera technology is a technology for acquiring depth information of each pixel of an image. The 3D depth camera that may be used as the IoT device 70 for behavior recognition may be any one of a stereo type, a Time-of-Flight (ToF) type, a structured pattern type, or a hybrid type in which these techniques are combined, but the scope of the present invention is not limited thereto.
[0089] The server 100' includes a sponsorship management unit 110, an action recognition unit 120', and a mission execution determination unit 130, and may further include an IoT control unit 140. The action recognition unit 120' of the server 100' may include a separate learning module. In this case, the learning module of the action recognition unit 120' may generate an action recognition model by training a deep learning network using the broadcaster's video and voice as input and the action names that may be used in the mission list as labels (output). In addition, the learning module of the action recognition unit 120' may generate an action recognition model by training a deep learning network using a dataset such as NTU RGB+D or motion data generated by Kinetic-GAN (Generative Adversarial Network).
[0090] The behavior recognition unit 120' collects at least one of video and audio of the broadcaster from the behavior recognition IoT device 70 instead of the broadcasting camera 40. When the behavior recognition IoT device 70 is a 3D depth camera, the behavior recognition unit 120' can effectively learn the behavior recognition model by performing learning and inference using depth information of each part of the body of the broadcaster to be recognized, thereby improving the accuracy of behavior recognition (inference).
[0091] FIG. 6 is a block diagram showing the configuration of a real-time distribution system according to a third embodiment of the present invention, and FIG. 7 is a block diagram showing the configuration of a real-time distribution support server according to the third embodiment of the present invention.
[0092] The third embodiment of the present invention is characterized in that the real-time distribution support server 100'' also performs the functions of a distributor terminal.
[0093] As shown in FIG. 6, the real-time distribution system 1″ according to the third embodiment of the present invention includes a distribution platform system 20′, a viewer terminal 30, a distribution IoT device 40′, and a real-time distribution support server (100″, hereinafter abbreviated as “server”), and may further include a distribution assistant IoT device 60.
[0094] The difference between real-time delivery system 1 and real-time delivery system 1" is in the data collection method for recognizing the behavior of the generator and distributor of the delivery content. The following will focus on the differences between real-time delivery system 1 and real-time delivery system 1".
[0095] The server 100″ receives distribution video and audio from the distribution IoT device 40′ via wired or wireless communication and generates content for distribution in real time. However, this is not limited to this and pre-generated content may be transmitted from outside. The distribution IoT device 40′ is a device to which Internet of Things technology is applied, and is equipped with a distribution camera and microphone and can transmit video and audio to the server 100″ via a wired / wireless communication network. The server 100″ transmits the distribution content to the distribution platform system 20′ so that streaming distribution can be performed.
[0096] When the viewer terminal 30 transmits the mission request information and the sponsorship deposit information to the server 100″ directly or through the distribution platform system 20′, the server 100″ combines one or more pieces of mission request information and sponsorship deposit information received from the viewer terminal 30 or the distribution platform system 20′ and transmits them to the viewer terminal 30 directly or through the distribution platform system 20′. The distributor can acknowledge the mission request information and the sponsorship deposit information using one of the multiple viewer terminals 30.
[0097] The server 100 ″ includes a sponsorship management unit 110 , a behavior recognition unit 120 ″, a mission execution determination unit 130 , and a distribution content generation unit 150 , and may further include an IoT control unit 140 .
[0098] The behavior recognition unit 120'' recognizes the behavior of the distributor based on the video and audio collected from the distribution IoT device 40', and the distribution content generation unit 150 generates distribution content based on the video and audio collected from the distribution IoT device 40' and transmits the generated distribution content to the distribution platform system 20'.
[0099] FIG. 8 is a block diagram showing the configuration of a real-time distribution system according to a fourth embodiment of the present invention.
[0100] The fourth embodiment of the present invention is characterized in that the real-time streaming sponsorship server 100'' is included in the streaming platform system 20'', and the streaming platform system 20'' has a function of collecting mission request information and sponsorship deposit information, and determining whether or not to complete the mission through behavior recognition of the streamer.
[0101] As shown in FIG. 8, the real-time distribution system 1'''' according to the fourth embodiment of the present invention includes a distributor terminal 10, a distribution platform system 20'', a viewer terminal 30, a distribution camera 40, a microphone 50, and may further include a distribution assistance IoT device 60. The distribution platform system 20'' includes a real-time distribution support server (100'''', hereinafter abbreviated as "server").
[0102] The following description will focus on the differences between the real-time delivery system 1 and the real-time delivery system 1''''.
[0103] The server 100'' included in the distribution platform system 20'' collects information on missions requested by viewers to Internet distributors (``mission request information'') and sponsorship deposit information from the viewer terminal 30, recognizes the distributor's behavior based on the distributed content, and compares the behavior recognition result with the mission included in the mission request information. If it determines that the mission has been completed, it processes the payment of the sponsorship money or transmits mission completion information to a sponsorship money processing unit separately included in the distribution platform system 20''. The sponsorship money processing unit can process the payment of all or a portion of the sponsorship money to the Internet distributor based on the mission completion information.
[0104] Except for the differences mentioned above, the configuration and functions of server 100'' are identical to those of server 100.
[0105] FIG. 9 is a flowchart illustrating an operation method of the real-time distribution support server 100, 100', 100'', 100'''' according to an embodiment of the present invention.
[0106] An operating method of the real-time distribution sponsorship server 100, 100', 100", 100''' according to an embodiment of the present invention includes steps S310 to S380, and may further include step S390. Also, certain steps may be omitted in the operating method. For example, steps S330 and S370 are steps of transmitting information to the outside and may be omitted. Also, in the case of real-time distribution without sponsorship, steps S320 and S380 may be omitted.
[0107] Step S310 is a step of receiving mission request information. The server 100, 100', 100", 100''' transmits a mission list to the viewer terminal 30. The server 100, 100', 100", 100''' may directly transmit the mission list to the viewer terminal 30, or may transmit or transfer the mission list to the distribution platform system 20, 20', 20", which may then combine the mission list with the video to be distributed and present it to the viewer terminal 30. When the viewer selects one of the actions listed in the mission list, the viewer terminal 30 transmits information regarding the action the viewer requests the distributor ("mission request information") to the server 100, 100', 100", 100''' directly or through the distribution platform system 20, 20', 20", and the server 100, 100', 100", 100''' receives the mission request information. If multiple broadcasters are currently broadcasting or participating in a broadcast on a single Internet broadcast channel, the mission request information may include information regarding the broadcaster who will perform the action requested by the viewer.
[0108] Step S320 is a sponsorship deposit step. Sponsorship deposit information is transmitted from the viewer terminal 30 to the server 100, 100', 100", 100'''. The viewer terminal 30 may transmit the sponsorship deposit information to the server 100, 100', 100", 100''' directly, or may transmit the sponsorship deposit information to the server 100, 100', 100", 100''' via the distribution platform system 20. The server 100, 100', 100", 100"' receives the sponsorship deposit information and matches it with mission request information transmitted from the same viewer terminal 30.
[0109] Steps S310 and S320 may be performed simultaneously. For example, when a viewer selects one of the missions ("make a finger heart") from the list of missions displayed on the viewer terminal device 30 while a game is being distributed in real time and deposits a donation through an application (or app) of the viewer terminal device 30, the server 100, 100', 100", 100''' matches the "make a finger heart" with the donation. Mission request information for "make a finger heart" can also be transmitted from a plurality of viewer terminal devices 30, in which case the server 100, 100', 100", 100''' matches a plurality of donation deposit information with the "make a finger heart" mission.
[0110] Step S330 is the step of transmitting mission request information and donation deposit information.
[0111] The servers 100, 100', 100", 100''' combine the received mission request information and sponsorship deposit information and transmit or transfer it to the broadcaster terminal 10 or the distribution platform systems 20, 20', 20". When the broadcaster terminal 10 receives the mission request information and sponsorship deposit information, the broadcaster terminal 10 can display the mission request information and sponsorship deposit information on the screen so that the broadcaster can acknowledge it. When the distribution platform systems 20, 20', 20" receive the mission request information and sponsorship deposit information, the distribution platform systems 20, 20', 20" can synthesize the corresponding information into an image and transmit it to multiple viewer terminals 30 connected to the same distribution channel.
[0112] As another example of the present invention, when a mission is selected in the viewer terminal 30, the mission can be kept secret from the distributor and other viewers (secret mission function). For example, the server 100, 100', 100''' can not transmit mission request information and sponsorship deposit information to the distributor terminal 10. In this case, the distributor will proceed with distribution without being aware of the mission request information, and viewers can enjoy and get excited by seeing the distributor's happy or surprised expression when the sponsorship money is paid or the distribution auxiliary IoT device (60, for example, a fan) is operated when the mission is completed. In addition, the server 100, 100', 100''' can prevent the mission request information and sponsorship deposit information from being transmitted not only to the distributor terminal 10 but also to the viewer terminal 30. In other words, when a viewer grants a mission, the distributor can not be informed of the mission request information (specific action or voice), and it can be made public or private to the viewers, which allows the production of more interesting content. For example, if the streamer says "good" while the mission is private, 100 won will be added to the sponsorship money, and if the streamer says "not so good," 100 won will be deducted from the sponsorship money. In addition, the time for which the secret mission is private can be set when the mission is requested, and after the time has elapsed, the streamer can check whether the mission has been completed and receive the corresponding sponsorship money.
[0113] Step S340 is a video / audio collection step. The server 100, 100', 100", 100''' collects at least one of the video of the distributor and the audio of the distributor. At least one of the distributor terminal 10, distribution platform system 20, 20', 20", distribution camera 40, distribution IoT device 40', microphone 50, and behavior recognition IoT device 70 can provide the video and audio of the distributor to the server 100, 100', 100", 100'''.
[0114] Step S350 is the behavior recognition step. The servers 100, 100', 100", 100''' recognize the behavior of the broadcaster based on at least one of the collected video and audio of the broadcaster. When the servers 100, 100', 100", 100''' recognize (infer) the behavior of the broadcaster, a neural network trained by deep learning techniques may be utilized. The behavior recognition result may be the name of the behavior, or may be in the form of matching the name of the behavior with the probability value of performing that behavior ("behavior probability"). For example, the behavior recognition result may be "making a heart with fingers" or "making a heart with fingers, 80%."
[0115] Step S360 is a step for determining whether or not there is a match between the behavior and the mission. The servers 100, 100', 100", and 100''' determine whether or not there is a match between the behavior recognition result and the mission request information. As described above, in this specification, "match" is considered to include not only a match between the behavior of the broadcaster recognized by the servers 100, 100', 100", and 100''' and the behavior specified by the viewer as the mission, but also a case where the probability that the behavior of the broadcaster is the behavior selected (requested) by the viewer as the mission falls within a predetermined range. For example, if the mission request information is "making a heart with fingers," the behavior recognition result is "making a heart with fingers, 80%," and the standard probability for determining "match" is 60%, then it is considered that the behavior recognition result and the mission request information match.
[0116] If the determination result indicates that the two do not match, the process proceeds to step S340, where the broadcaster's video and audio are additionally collected for a predetermined time period to recognize the broadcaster's behavior. For example, if the mission request information is "making a heart with fingers" but the behavior recognition result is "making a heart with arms," the mission and behavior do not match. In this case, if there are three minutes left in the mission confirmation time, the server 100, 100', 100'', 100''' will additionally collect the broadcaster's video and audio for the remaining three minutes to recognize the broadcaster's behavior and confirm whether the mission "making a heart with fingers" has been completed. If the mission is not completed within the given time, the server 100, 100', 100'', 100''' will process the task of returning the deposited sponsorship money and sponsorship items to the viewer through the sponsorship management unit 110, 221.
[0117] If the determination result is consistent, step S370 is performed. For example, if the mission request information and the behavior recognition result are consistent with "making a heart with fingers," step S370 is performed.
[0118] Step S370 is a step of transmitting mission completion information. The server 100, 100', 100", 100''' transmits or transfers the mission completion information to the distribution platform system 20, 20', 20". The distribution platform system 20, 20', 20" can combine the mission completion information with an image and transmit it to a plurality of viewer terminal devices 30. In addition, the server 100, 100', 100", 100''' can transmit or transfer information that sponsorship money / sponsorship items will be provided (hereinafter, "sponsorship payment information") together with the mission completion information to the distribution platform system 20, 20', 20". In this case, the distribution platform system 20, 20', 20" can combine both the mission completion information and the sponsorship payment information with an image and transmit them to a plurality of viewer terminal devices 30.
[0119] Step S380 is a sponsorship payment step. The servers 100, 100', 100", 100''' process the payment of all or part of the deposited sponsorship money or sponsorship items to the broadcaster who completed the mission. If the distribution platform system 20, 20', 20" manages the deposit of sponsorship money or if the viewer terminal 30 purchases a sponsorship item through the distribution platform system 20, 20', 20", the servers 100, 100', 100", 100''' transmit or deliver a message to the distribution platform system 20, 20', 20" requesting the payment of the sponsorship money or sponsorship item.
[0120] Step S390 is the auxiliary IoT control step. When the mission is completed, the server 100, 100', 100", 100''' can operate the distribution auxiliary IoT device 60 by generating a control signal and transmitting it to the distribution auxiliary IoT device 60. For example, the server 100, 100', 100", 100''' can turn on the air conditioner 60 for a certain period of time or change the set temperature, and can also make the money gun 60 dispense bills in the amount to be paid to the distributor.
[0121] In the above description, steps S310 to S390 may be further divided into additional steps or combined into fewer steps, according to embodiments of the present invention. Also, some steps may be omitted, and the order of steps may be changed, as necessary. In addition, the contents of FIGS. 1 to 8 may also be applied to the contents of FIG. 9, even if other omitted contents are included. Also, the contents described with reference to FIG. 9 may be applied to FIGS. 1 to 8.
[0122] In addition, components according to embodiments of the present invention may be embodied in the form of software or hardware such as a digital signal processor (DSP), a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) to perform a specific function.
[0123] However, the term "component" is not meant to be limited to software or hardware, and each component may be configured to reside on an addressable storage medium and to execute on one or more processors.
[0124] Thus, by way of example, components include components such as software components, object oriented software components, class components and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables.
[0125] The components and the functionality provided within those components may be combined into fewer components or further separated into additional components.
[0126] However, it will be understood that each block of the flowchart illustrations and combinations of the flowchart illustrations can be implemented by computer program instructions. These computer program instructions can be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment create means for performing the functions described in the flowchart block(s). These computer program instructions can also be used with a computer or stored in a computer readable memory that can direct the computer or other programmable data processing equipment to implement the functions in a particular manner, such that the instructions used with the computer or stored in a computer readable memory can produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions may be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on the computer or other programmable data processing device to generate a computer implemented process, such that the instructions cause the computer or other programmable data processing device to provide steps for performing the functions described in the flowchart block(s).
[0127] Also, each block may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order. For example, two blocks shown in succession may be performed substantially simultaneously, or the blocks may sometimes be performed in reverse order depending on the corresponding functions.
[0128] In this embodiment, the term "module" refers to software or hardware components such as FPGA or ASIC, and the "module" performs a certain function. However, the "module" is not limited to software or hardware. The "module" may be configured to be in an addressable storage medium and to execute one or more processors. Thus, by way of example, the "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functions provided within the "module" may be combined into a smaller number of components and "modules" or further separated into additional components and "modules". Moreover, the components and "modules" may be embodied to execute one or more CPUs within a device or a security multimedia card.
[0129] The operation methods of the real-time distribution sponsorship servers 100, 100', 100", 100''' described above have been described with reference to the flowcharts presented in the drawings. For ease of explanation, the methods have been illustrated and described as a series of blocks, but the invention is not limited to the order of the blocks, and some blocks may occur in a different order or simultaneously with other blocks than those illustrated and described herein, and various other branches, flow paths, and orders of blocks may be implemented that achieve the same or similar results. In addition, not all illustrated blocks may be required to implement the methods described herein.
[0130] Although the configuration of the present invention has been described in detail with reference to the accompanying drawings, it goes without saying that this is merely an example, and that those skilled in the art to which the present invention pertains may make various modifications and changes within the scope of the technical idea of the present invention. Therefore, the scope of protection of the present invention is defined by the claims set forth below rather than the above detailed description, and all modifications or alterations derived from the claims and their equivalents should be interpreted as being included in the technical scope of the present invention. [Explanation of symbols]
[0131] 1, 1´, 1″, 1''': Real-time distribution system 10: Distributor terminal 20, 20´, 20″: Distribution platform system 30: Viewer terminal device 40: Broadcast camera 40´: IoT equipment for distribution 50: Microphone 60: Distribution support IoT devices 70: IoT equipment for behavior recognition 100, 100´, 100″, 100''': Real-time streaming support server 110: Support Management Department 120, 120´, 120″: Behavior recognition section 130: Mission Execution and Judgment Department 140: IoT control section 150: Distribution content generation unit
Claims
1. a sponsorship management unit for collecting information on missions requested by viewers of online broadcasts to online broadcasters and sponsorship money deposit information, and for processing the payment of the sponsorship money when the missions are completed; a behavior recognition unit that recognizes the behavior of the Internet distributor based on at least one of the video and the audio of the Internet distributor and generates a behavior recognition result; and a mission execution determination unit for determining whether execution of the mission is completed by comparing the behavior recognition result with information on the mission.
2. The real-time distribution support server based on automatic behavior recognition according to claim 1 , further comprising: an IoT control unit that controls a distribution assistance IoT device when the mission is completed.
3. The information for the mission is:
2. The real-time distribution support server based on automatic behavior recognition according to claim 1, wherein the name of at least one of the actions, facial expressions, and postures requested by the viewer of the Internet distribution to the Internet distributor is included.
4. The information for the mission is:
2. The real-time distribution support server based on automatic behavior recognition according to claim 1, wherein the request is at least one of a video and an image related to the behavior requested by the viewer of the Internet distribution to the Internet distributor.
5. An IoT device for behavior recognition that collects at least one of video and audio of an Internet distributor; and a server that collects information on missions and donation deposit information requested by viewers of internet broadcasts to the internet broadcasters, recognizes the behavior of the internet broadcasters based on at least one of video and audio information of the internet broadcasters to generate a behavior recognition result, compares the behavior recognition result with information on the missions to determine whether the missions have been completed, and processes the payment of the donations if the missions have been completed.
6. The behavior recognition IoT device includes: The real-time broadcasting support system based on automatic behavior recognition according to claim 5, further comprising: acquiring depth information for each part of the body of the Internet broadcaster.
7. The server, The real-time distribution support system based on automatic behavior recognition according to claim 5, further comprising controlling a distribution assistance IoT device when the mission is completed.
8. Distribution platform system; A distributor terminal that generates distribution contents in which an Internet distributor appears and transmits the contents to the distribution platform system; and a viewer terminal that receives the distribution content in the distribution platform system and provides the distribution content to a viewer; The distribution platform system includes: A real-time distribution system that collects information on a mission requested by the viewer to the Internet distributor and information on deposit of a sponsorship fee from the viewer terminal device, and processes payment of the sponsorship fee when it is determined that the mission is completed based on the distributed content.
9. a sponsorship management unit that collects information on a mission and a reward for the mission presented by either an internet distributor or a viewer of the internet distribution, and processes the payment of the reward when the mission is completed; A behavior recognition unit that calculates a behavioral similarity between the Internet distributor and the viewer based on a video of the Internet distributor and a video of the viewer; and a mission execution determination unit that determines whether execution of the mission is completed based on whether the behavior similarity falls within a predetermined range; The information regarding the reward is: A real-time distribution sponsorship server based on automatic behavior recognition that provides at least one of a prize and channel points on the condition that the mission is completed.
10. a sponsorship management unit for collecting information on missions requested by viewers of online broadcasts to online broadcasters and sponsorship money deposit information, and for processing the payment of the sponsorship money when the missions are completed; A recognition unit that recognizes at least one of an object and a character that appears in the video of the Internet distributor and generates a recognition result; and a mission execution determining unit for determining whether execution of the mission is completed according to whether the recognition result matches information on the mission.
11. receiving mission request information at a viewer terminal; a video / audio collecting step of collecting at least one of video and audio of an Internet distributor; A behavior recognition step of generating a behavior recognition result of the Internet distributor based on at least one of a video and a voice of the Internet distributor; and and determining whether the action recognition result matches the mission request information.
12. Prior to the video / audio collection step, The method further includes a sponsorship deposit step of receiving sponsorship deposit information from the viewer terminal and matching the information with the mission request information; After the action-mission matching determination step, The method of claim 11, further comprising: a sponsorship payment step of processing the sponsorship money to be paid to the Internet distributor in one of full payment and partial payment when the behavior recognition result matches the mission request information.
13. After the action-mission matching determination step, The method of claim 11, further comprising: operating an auxiliary IoT device for streaming when the behavior recognition result matches the mission request information.
14. The behavior recognition step includes: The method of claim 11, further comprising: recognizing the behavior of the Internet broadcaster through a neural network trained by a deep learning technique.
