Live distribution system, estimation method, and program

Abstract

To reduce a processing load on a computer which estimates whether or not a viewer comment in a live distribution service is appropriate.SOLUTION: A viewer comment acquisition section (101) of a live distribution system (1) acquires a viewer comment inputted by a viewer of a live distribution service. A first estimation section (102) estimates whether or not the viewer comment is appropriate based on a first model that has learned a first training comment. A first reference determination section (103) determines whether or not a first score relating to estimation accuracy of the first model satisfies a predetermined first reference. When the first score is determined to satisfy the first reference, a second estimation section (104) does not perform estimation based on a second model that has learned a second training comment. Meanwhile, when the first score is determined not to satisfy the first reference, the second estimation section estimates whether or not the viewer comment is appropriate based on the second model.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a live distribution system, an estimation method, and a program. [Background technology]

[0002] Conventionally, live streaming services that deliver information such as video or audio in real time are known. In live streaming services, viewers may input inappropriate viewer comments, so it is necessary to estimate whether the viewer comments are appropriate. For example, Patent Literature 1 describes a technology in which training comments are manually classified as to whether they are appropriate, training data is created, and unclassified comments are classified as to whether they are appropriate based on a model that has been trained on the training data using a machine learning technique. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-506274 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when the technology of Patent Document 1 is applied to a live streaming service, the model of Patent Document 1 can estimate whether a viewer comment is appropriate or not. In this regard, since the estimation accuracy may not be sufficient with only one model, it is possible to use multiple models that have learned different training data in combination. However, if estimation based on multiple models is performed for all viewer comments, the processing load of the computer that estimates whether the viewer comment is appropriate or not increases.

[0005] One objective of the present disclosure is to reduce the processing load on a computer that estimates whether viewer comments in a live streaming service are appropriate. [Means for solving the problem]

[0006] The live streaming system of the present disclosure includes a viewer comment acquisition unit that acquires viewer comments input by viewers of a live streaming service; a first estimation unit that estimates whether the viewer comments are appropriate based on a first model from which first training comments have been learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; and a second estimation unit that, if it is determined that the first score satisfies the first criterion, does not perform estimation based on a second model from which second training comments have been learned, and, if it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comments are appropriate based on the second model. Effect of the Invention

[0007] According to the present disclosure, it is possible to reduce the processing load on a computer that estimates whether viewer comments in a live streaming service are appropriate. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of a live distribution system. [Diagram 2] FIG. 1 is a diagram showing an example of a video distributed by a live distribution service. [Diagram 3] FIG. 2 is a diagram illustrating an example of functions realized in a live distribution system. [Figure 4] FIG. 4 illustrates an example of a distribution database. [Diagram 5] FIG. 11 is a diagram showing an example of a process for estimating whether a viewer comment is appropriate or not, based on a first model and a second model. [Figure 6] FIG. 1 is a diagram illustrating an example of a process executed in a live distribution system. [Figure 7] FIG. 1 is a diagram illustrating an example of a process executed in a live distribution system. [Figure 8]FIG. 13 is a diagram illustrating an example of a function realized in a modified example. [Figure 9] FIG. 11 is a diagram illustrating an example of processing of a second model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [1. Overall configuration of the live streaming system] An example of an embodiment of a live streaming system according to the present disclosure will be described. Fig. 1 is a diagram showing an example of the overall configuration of a live streaming system. For example, the live streaming system 1 includes a server 10, an operator terminal 20, a distributor terminal 30, and a viewer terminal 40. Each of the server 10, the operator terminal 20, the distributor terminal 30, and the viewer terminal 40 can be connected to a network N such as the Internet or a LAN.

[0010] The server 10 is a server computer that provides a live distribution service. The live distribution service is a service that distributes information in real time. For example, the server 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 includes at least one processor. The storage unit 12 includes a volatile memory such as a RAM, and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication.

[0011] The operator terminal 20 is the operator's computer. The operator is a company that operates a live streaming service. For example, the operator terminal 20 is a personal computer, a tablet terminal, or a smartphone. For example, the operator terminal 20 includes a control unit 21, a memory unit 22, a communication unit 23, an operation unit 24, and a display unit 25. The physical configurations of the control unit 21, the memory unit 22, and the communication unit 23 are similar to those of the control unit 11, the memory unit 12, and the communication unit 13, respectively. The operation unit 24 is an input device such as a touch panel. The display unit 25 is a liquid crystal display or an organic EL (Electro Luminescence) display.

[0012] The broadcaster terminal 30 is a computer of the broadcaster. The broadcaster is a user who broadcasts information through a live broadcasting service. For example, the broadcaster terminal 30 is a personal computer, a tablet terminal, or a smartphone. For example, the broadcaster terminal 30 includes a control unit 31, a memory unit 32, a communication unit 33, an operation unit 34, and a display unit 35. The physical configurations of the control unit 31, the memory unit 32, the communication unit 33, the operation unit 34, and the display unit 35 are similar to those of the control unit 11, the memory unit 12, the communication unit 13, the operation unit 24, and the display unit 25, respectively. A camera 36 is connected to the broadcaster terminal 30. The camera 36 may be included inside the broadcaster terminal 30.

[0013] The viewer terminal 40 is a viewer's computer. The viewer is a user who views information through a live distribution service. For example, the viewer terminal 40 is a personal computer, a tablet terminal, or a smartphone. For example, the viewer terminal 40 includes a control unit 41, a memory unit 42, a communication unit 43, an operation unit 44, and a display unit 45. The physical configurations of the control unit 41, the memory unit 42, the communication unit 43, the operation unit 44, and the display unit 45 are similar to those of the control unit 11, the memory unit 12, the communication unit 13, the operation unit 24, and the display unit 25, respectively.

[0014] The programs stored in the storage units 12, 22, 32, and 42 may be supplied via the network N. Also, the programs stored in a computer-readable information storage medium may be supplied via a reading unit (e.g., an optical disk drive or a memory card slot) that reads the information storage medium, or an input / output unit (e.g., a USB port) that inputs and outputs data to and from an external device.

[0015] Furthermore, the live streaming system 1 only needs to include at least one computer, and is not limited to the example in Fig. 1. For example, the live streaming system 1 may include only the server 10, and the operator terminal 20, the broadcaster terminal 30, and the viewer terminal 40 may exist outside the live streaming system 1. For example, the live streaming system 1 may include only the server 10 and the operator terminal 20, and the broadcaster terminal 30 and the viewer terminal 40 may exist outside the live streaming system 1.

[0016] [2. Overview of the live streaming system] In this embodiment, a live streaming service that distributes videos promoting products is taken as an example. The live streaming service may be of various types and is not limited to the example of this embodiment. For example, the live streaming service may distribute a game commentary, a seminar, a video promoting a service, a broadcaster singing or playing, a chat with a broadcaster, or a variety show. In a live streaming service, only either an image or an audio may be distributed, instead of both an image and an audio. In this embodiment, the viewer includes both a person who only watches an image and a person who only listens to an audio.

[0017] FIG. 2 is a diagram showing an example of a video distributed by a live distribution service. For example, a distributor advertises a product sold at an online shopping mall in front of a camera 36. Distributor terminal 30 transmits the video generated by camera 36 to server 10. Server 10 distributes the video received from distributor terminal 30 to viewer terminals 40 of an unspecified number of viewers in real time. The video is displayed in real time on display unit 45 of viewer terminal 40. If a viewer likes a product in the video being viewed, they can purchase the product by selecting button B.

[0018] For example, a viewer can input a comment into the input form F. Hereinafter, a comment input by a viewer is referred to as a viewer comment. A viewer comment can include any character string. For example, a viewer comment is a message addressed to the broadcaster or other viewers. A viewer comment may not be a message addressed to others, but may be a monologue-like content. For example, a plurality of viewer comments are displayed in chronological order in the window W. As shown in FIG. 2, various viewer comments such as impressions of a product and impressions of the broadcaster are displayed in the window W. In this embodiment, the viewer comments are, in principle, made public to everyone, including the broadcaster and viewers.

[0019] For example, a viewer may enter inappropriate viewer comments. Inappropriate viewer comments include content that violates the terms of use of the live streaming service. Inappropriate viewer comments may also be viewer comments that should not be made public. For example, inappropriate viewer comments include slander, obscene content, other content that others find offensive, or content that is unclear. In the example of FIG. 2, the viewer comments entered by viewers C and E are inappropriate.

[0020] For example, it would be very time-consuming for an administrator to check all viewer comments and delete inappropriate ones. In this embodiment, in order to reduce the administrator's time and effort, whether or not a viewer comment is appropriate is estimated based on a learning model using a machine learning technique. Various techniques can be used for the machine learning technique itself. For example, the learning model may be a model using any of the machine learning techniques of supervised learning, semi-supervised learning, or unsupervised learning. Specific examples of machine learning techniques will be described later.

[0021] In this embodiment, all viewer comments are initially displayed in window W. Viewer comments that are estimated to be inappropriate by the learning model are notified to the administrator terminal 20. The administrator checks whether the viewer comment is truly inappropriate. If the administrator determines that the viewer comment is inappropriate, the administrator operates the operation unit 24 of the administrator terminal 20 to delete the viewer comment. Therefore, viewer comments that are estimated to be inappropriate by the learning model are displayed in window W until they are deleted by the administrator.

[0022] In this embodiment, in order to improve the estimation accuracy of inappropriate viewer comments, a plurality of learning models such as a first model and a second model are used in combination. However, if estimation based on both the first model and the second model is performed for all viewer comments, the processing load of the server 10 increases. For example, if the processing load of the server 10 increases, delays may occur in videos distributed by a live distribution service, or it may take a long time to detect inappropriate viewer comments.

[0023] Therefore, in the live distribution system 1 of this embodiment, when the estimation accuracy of the first model is high, the estimation result of the first model can be trusted, so estimation based on the second model is not performed. When the estimation accuracy of the first model is low, the live distribution system 1 performs estimation based on the second model, because the estimation result of the first model cannot be trusted very much. This reduces the processing load of the server 10. Details of this embodiment will be described below.

[0024] [3. Functions realized by the live streaming system] FIG. 3 is a diagram showing an example of functions realized by the live distribution system 1.

[0025] [3-1. Functions realized by the server] For example, the server 10 includes a data storage unit 100, a viewer comment acquisition unit 101, a first estimation unit 102, a first criterion determination unit 103, a second estimation unit 104, and a notification unit 105. The data storage unit 100 is realized by the memory unit 12. Each of the viewer comment acquisition unit 101, the first estimation unit 102, the first criterion determination unit 103, the second estimation unit 104, and the notification unit 105 is realized by the control unit 11.

[0026] [Data storage section] The data storage unit 100 stores data necessary for providing a live distribution service. For example, the data storage unit 100 stores a distribution database DB1.

[0027] 4 is a diagram showing an example of the distribution database DB1. The distribution database DB1 is a database that stores data related to videos being distributed by a live distribution service. For example, the distribution database DB1 stores a video ID, a distributor ID, a video title, actual data of the video, actual data of viewer comments, a viewer ID, a posting date and time, and an estimation result. In addition to videos being distributed, archives of videos that have been distributed in the past may also be stored in the distribution database DB1.

[0028] The video ID is information that can uniquely identify a video. When video distribution begins in the live distribution service, the server 10 issues a new video ID that does not overlap with other video IDs, and creates a new record in the distribution database DB1. The distributor ID is information that can uniquely identify the distributor of the video. The video title is the name of the video entered by the distributor. The actual data of the video may be in any format, for example, MPEG format, MP4 format, MOV format, or AVI format.

[0029] The actual data of the viewer comment may be in any format, for example, text format, rich text format, or message format. The viewer ID is information that can uniquely identify the viewer who inputs the viewer comment. The posting date and time is the date and time when the viewer comment was posted. The estimation result is an estimation result based on at least one of the first model M1 and the second model M2. Each time a viewer inputs a viewer comment, the actual data of the viewer comment, the viewer ID, the posting date and time, and the estimation result are stored in the distribution database DB1.

[0030] For example, the data storage unit 100 stores a trained first model M1 and a trained second model M2. Trained means that parameter adjustment has been completed. The data storage unit 100 stores the programs and parameters of the first model M1 and the second model M2. In this embodiment, a case will be described in which the learning of each of the first model M1 and the second model M2 is executed by the administrator terminal 20, but the learning of each of the first model M1 and the second model M2 may be executed by a computer other than the administrator terminal 20. The first model M1 and the second model M2 may be combined as one data. That is, the first model M1 and the second model M2 may be included in one model.

[0031] 5 is a diagram showing an example of a process for estimating whether a viewer comment is appropriate based on a first model M1 and a second model M2. Hereinafter, data input to the first model M1 is referred to as first input data. An estimation result by the first model is referred to as a first estimation result. Data input to the second model M2 is referred to as second input data. An estimation result by the second model is referred to as a second estimation result. The first input data, the second input data, the first estimation result, and the second estimation result may be in any format.

[0032] For example, the first model M1 is a model that uses fastText. fastText is a model that converts the meaning of a word into a feature vector based on text data such as an Internet encyclopedia. fastText can calculate the feature vector of a word faster than other models such as Word2Vec. For example, the viewer comments themselves are input as the first input data to the first model M1. The first model M1 performs morphological analysis on the first input data and calculates the feature vector for each word.

[0033] For example, the first model M1 calculates a feature vector of the entire first input data based on the feature vector of each word. The first model M1 outputs a first estimation result and a first score according to the feature vector of the entire first input data. Note that morphological analysis may be performed outside the first model M1, and the first input data may be a sequence of words after the viewer comments are broken down into words. Also, the feature vectors of words may be calculated outside the first model M1, and the first input data may be a sequence of feature vectors.

[0034] In this embodiment, the first model M1 outputs binary data indicating whether the viewer comments are appropriate or not as the first estimation result. The first estimation result may be any data related to the appropriateness of the viewer comments and is not limited to the example of this embodiment. For example, the first estimation result may not be binary data and may have an intermediate value. For example, when classifying the appropriateness of the viewer comments into five stages, the first estimation result may be expressed in five stages. Since the first model M1 classifies the appropriateness of the viewer comments, the first estimation result can also be said to be a classification result of the appropriateness of the viewer comments.

[0035] The first score is a score related to the estimation accuracy of the first model M1. The first score is an index indicating how confident the first model M1 is in making an estimation. The first score can also be said to be the reliability of the first estimation result. In this embodiment, a case will be described in which an index called a confidence score calculated for each first input data corresponds to the first score. The method of calculating the confidence score itself can use various known methods. For example, the first score may be calculated using Log Loss (cross entropy), AUC (Area Under the Curve), or Gini coefficient. In this embodiment, a case will be described in which the first score is expressed by a numerical value of 0 to 100, but the first score can be expressed in any numerical range.

[0036] The first score may be another index instead of the confidence score. For example, the first score may be the cosine similarity between the feature vector of the viewer comment and the feature vector of the first training comment described below. In this case, the higher the cosine similarity, the more similar the viewer comment is to the learned first training comment, and therefore the higher the accuracy of the estimation result. The first score may be an index such as an F-measure, a correct answer rate, or a matching rate.

[0037] In this embodiment, the second model M2 is a model having a conditional branch. For example, the second model M2 is a model using LightGBM (Light Gradient Boosting Machine). LightGBM is a model using a gradient boosting algorithm for decision trees. LightGBM includes multiple decision trees. LightGBM determines whether or not the conditions indicated by each of multiple conditional branches are satisfied based on data input thereto, and outputs an estimation result according to the determination result. LightGBM is known to be faster and more accurate than XGBoost, which is a type of gradient boosting algorithm for decision trees, but in this embodiment, it is slower than fastText.

[0038] In the example of FIG. 5, the second model M2 includes a plurality of condition branches such as conditions 1 to 7. For example, a plurality of items related to the characteristics of the viewer comments are input as second input data to the second model M2. Each item may be related to some characteristic of the viewer comments, for example, the number of characters in the viewer comment, the number of words included in the viewer comment, or a feature vector of the entire viewer comments. As in a modified example described later, the second input data may include other items such as a comparison result with a keyword. The conditions in the decision tree may be conditions that can be determined by the items included in the second input data. For example, the number of characters, the number of words, or a threshold value of the feature vector of the viewer comments may be set as a condition.

[0039] For example, each of the multiple condition branches included in the second model M2 has a condition that is determined based on at least one item included in the second input data. In the example of FIG. 5, the second model M2 determines whether the first condition 1 is satisfied based on the second input data. If it is determined that the condition 1 is satisfied, the second model M2 determines whether the condition 2 is satisfied based on the second input data. If it is determined that the condition 1 is not satisfied, the second model M2 determines whether the condition 3 is satisfied based on the second input data. In the same manner, the second model M2 determines whether the conditions of the tree-shaped condition branches are satisfied until the end of the decision tree is reached. The second estimation result is associated with the end of the decision tree. The second model M2 outputs the second estimation result at the end of the decision tree.

[0040] In this embodiment, the second model M2 outputs binary data indicating whether the viewer comments are appropriate or not as the second estimation result. The second estimation result may be any data related to the appropriateness of the viewer comments and is not limited to the example of this embodiment. For example, the second estimation result may not be binary data and may have an intermediate value. For example, when the appropriateness of the viewer comments is classified into five stages, the second estimation result may be expressed in five stages. The number of stages of the first estimation result and the second estimation result may be different. Since the second model M2 classifies the appropriateness of the viewer comments, the second estimation result can also be said to be a classification result of the appropriateness of the viewer comments.

[0041] In this embodiment, the first model M1 is a model that is faster than the second model M2. The first model M1 can also be said to be a model that requires less calculations than the second model M2. For example, assume that n (n is a natural number) viewer comments are the subject of estimation by the first model M1 and the second model M2. The n viewer comments estimated by the first model M1 are the same as the n viewer comments estimated by the second model M2. A first time required for the first model M1 to output all n first estimation results is shorter than a second time required for the second model M2 to output all n second estimation results.

[0042] In this embodiment, the second model M2 is assumed to have higher accuracy than the first model M1. For example, assume that n viewer comments whose correct answers are known in advance are input to each of the first model M1 and the second model M2. The n viewer comments estimated by the first model M1 and the n viewer comments estimated by the second model M2 are the same. In this case, the accuracy rate of the second model M2 is higher than that of the first model M1. It is assumed that the second model M2 has higher indices such as the matching rate or F-measure, rather than the accuracy rate, than the first model M1.

[0043] The first model M1 and the second model M2 may be models using other machine learning methods than fastText and LightGBM, and are not limited to the example of this embodiment. The first model M1 and the second model M2 may be any combination. For example, the first model M1 may be a model using Word2Vec, XGBoost, BERT, a Transformer other than BERT, a neural network, a long-short-term memory, or a support vector machine. The second model M2 may be a machine learning method other than the first model M1 among these exemplified machine learning methods. The first model M1 and the second model M2 may be models using the same machine learning method, but may be models in which different training data is learned from each other.

[0044] Furthermore, the first model M1 may be slower than the second model M2. The first model M1 may be more accurate than the second model M2. For example, contrary to the example of this embodiment, the first model M1 may be a model using LightGBM, and the second model M2 may be a model using fastText. It is only necessary that the machine learning method of the first model M1 and the machine learning method of the second model M2 are different from each other.

[0045] [Viewer Comment Collection Department] The viewer comment acquisition unit 101 acquires viewer comments entered by viewers of the live streaming service. Acquiring viewer comments means acquiring actual data of the viewer comments. In this embodiment, the viewer comment acquisition unit 101 is realized by the server 10, so the viewer comment acquisition unit 101 acquires viewer comments transmitted by the viewer comment transmission unit 401 of the viewer terminal 40. The viewer comment acquisition unit 101 associates the actual data of the viewer comment entered for a certain video with the video ID of that video, and stores the actual data in the distribution database DB1 together with the viewer ID of the viewer who entered the viewer comment and the posting date and time.

[0046] [First estimation part] The first estimation unit 102 estimates whether the viewer comment is appropriate based on the first model M1 from which the first training comment has been learned. The first training comment is a training comment to be learned by the first model M1. The first training comment is stored in a first training database DB2 described below. For example, the first estimation unit 102 inputs the viewer comment to the first model M1 as first input data. The first model M1 calculates a feature vector of the entire viewer comment based on the first input data, and outputs an estimation result according to the feature vector. The first estimation unit 102 estimates whether the viewer comment is appropriate by acquiring the estimation result of the first model M1.

[0047] [First standard judgment section] The first criterion determination unit 103 determines whether the first score satisfies a predetermined first criterion. In this embodiment, a case will be described in which the first score is calculated by the first model M1, and the first criterion determination unit 103 acquires the first score output from the first model M1. However, the first score may be calculated by inputting data indicating the internal processing of the first model M1 into a calculation algorithm for the first score. In addition, in this embodiment, a case will be described in which a higher first score means a higher estimation accuracy of the first model M1. However, it is sufficient that there is a correlation between the first score and the estimation accuracy of the first model M1, and a lower first score may mean a higher estimation accuracy of the first model M1.

[0048] The first criterion is a criterion for determining whether or not to execute estimation based on the second model M2. In this embodiment, since a higher first score means a higher estimation accuracy of the first model M1, the first criterion corresponds to the first score being equal to or greater than a predetermined first threshold. The first threshold may be a variable value that can be designated by the administrator, or may be a fixed value that is determined in advance. Note that, in the case where a lower first score means a higher estimation accuracy of the first model M1, the first criterion may correspond to the first score being less than the first threshold.

[0049] For example, the first criterion determination unit 103 determines whether the first score satisfies the first criterion by determining whether the first score is equal to or greater than a first threshold. If the first score is equal to or greater than the first threshold, the first criterion determination unit 103 determines that the first score satisfies the first criterion. If the first score is less than the first threshold, the first criterion determination unit 103 determines that the first score does not satisfy the first criterion.

[0050] [Second estimation part] If the first score is determined to satisfy the first criterion, the second estimation unit 104 does not perform estimation based on the second model M2 from which the second training comment was learned, and if the first score is determined to not satisfy the first criterion, the second estimation unit 104 estimates whether the viewer comment is appropriate or not based on the second model M2. As shown in Fig. 5, if the first score is determined to satisfy the first criterion, the second estimation unit 104 does not input second input data based on the viewer comment to the second model M2, and if the first score is determined to not satisfy the first criterion, the second estimation unit 104 inputs the second input data to the second model M2.

[0051] In this embodiment, since the second model M2 has a conditional branch, when it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring an output according to the conditional branch satisfied by the viewer comment. For example, the second model M2 generates second input data including a plurality of items related to the characteristics of the viewer comment, determines whether the viewer comment satisfies each of the conditions of the plurality of conditional branches based on the second input data, and outputs a second estimation result according to the determination result. The second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring the second estimation result of the second model M2.

[0052] [Notification Department] The notification unit 105 notifies the operator terminal 20 of at least one of the estimation results of the first model M1 and the second model M2. For example, the notification unit 105 notifies the operator terminal 20 of a viewer comment estimated to be inappropriate by the first model M1. The notification unit 105 notifies the operator terminal 20 of a viewer comment estimated to be inappropriate by the second model M2. The notification unit 105 does not need to notify the operator terminal 20 of a viewer comment estimated to be inappropriate by the first model M1 but estimated to be appropriate by the second model M2. The notification unit 105 may notify the operator terminal 20 of a viewer comment estimated to be appropriate by the first model M1 but estimated to be inappropriate by the second model M2.

[0053] The server 10 may execute a predetermined process based on at least one of the first estimation result and the second estimation result. The predetermined process is not limited to a notification by the notification unit 105. For example, a process by the display control unit 110 in Modification Example 10 described later is also an example of the predetermined process. The predetermined process may be a process other than the process by the notification unit 105 or the display control unit 110, and may be, for example, a process of waiting for a certain period of time before displaying the poster's comment, a process of lowering the display order of the poster's comment, or a process of displaying the poster's comment in a smaller size.

[0054] For example, when the server 10 determines that the first score satisfies the first criterion, the server 10 executes a predetermined process based on the first estimation result. When the server 10 determines that the first score does not satisfy the first criterion, the server 10 executes a predetermined process based on the second estimation result, not based on the first estimation result. When the server 10 determines that the first score does not satisfy the first criterion, the server 10 may execute a predetermined process based on both the first estimation result and the second estimation result.

[0055] [3-2. Functions realized by the operator's terminal] For example, the operator terminal 20 includes a data storage unit 200, a learning unit 201, and a restriction unit 202. The data storage unit 200 is realized by the storage unit 22. The learning unit 201 and the restriction unit 202 are realized by the control unit 21.

[0056] [Data storage section] The data storage unit 200 stores data necessary for operating a live distribution service. For example, the data storage unit 200 stores a first model M1 and a second model M2 before learning. The first model M1 and the second model M2 before learning are models whose parameters are initial values. The first model M1 and the second model M2 before learning are learned by the learning unit 201. For example, the data storage unit 200 stores a first training database DB2 and a second training database DB3.

[0057] The first training database DB2 is a database in which a plurality of first training data are stored. The first training data is training data to be learned by the first model M1. The first training data indicates a relationship between the first training comments (input portion) and a correct answer (output portion) indicating whether the first training comments are appropriate or not. The input portion of the first training data has the same format as the first input data. The output portion of the first training data has the same format as the first estimation result. The first training comments may be poster comments previously input in the live distribution service, or may be virtual comments prepared by the operator for training purposes. The correct answers are to be manually labeled by the operator, but a tool that automatically performs labeling may be used.

[0058] The second training database DB3 is a database in which a plurality of second training data are stored. The second training data is training data to be learned by the second model M2. The second training data includes second training comments (input portion) and correct answers (output portion) indicating whether the second training comments are appropriate or not. The input portion of the second training data has the same format as the second input data. The output portion of the second training data has the same format as the second estimation result. The second training comments may be poster comments previously input in the live distribution service, or virtual comments prepared by the operator for training purposes. The correct answers are to be manually labeled by the operator, but a tool that automatically performs labeling may be used.

[0059] [Learning Department] The learning unit 201 learns a first model M1 and a second model M2. For example, the learning unit 201 adjusts the parameters of the first model M1 so that when a first training comment included in a certain first training data is input, the correct answer included in the first training data is output. The learning unit 201 adjusts the parameters of the second model M2 so that when a second training comment included in a certain second training data is input, the correct answer included in the second training data is output.

[0060] The learning method itself of the first model M1 and the second model M2 may be a method used in machine learning methods. In the example of this embodiment, the learning unit 201 learns the first model M1 based on the learning method used in FastText. The learning unit 201 learns the second model M2 based on the learning method used in Light GBM. For example, depending on the machine learning method used for the first model M1 and the second model M2, a method such as backpropagation or gradient descent may be used to execute learning of the first model M1 and the second model M2.

[0061] [Restriction section] The limiting unit 202 limits the display of poster comments that are determined to be inappropriate by at least one of the first model M1 and the second model M2. In this embodiment, the limiting unit 202 limits the display of poster comments designated by the administrator among the poster comments notified to the administrator terminal 20. For example, the limiting unit 202 deletes poster comments that the administrator determines to be inappropriate from the distribution database DB1. The limiting unit 202 may limit the display of the poster comments by turning on a flag to restrict the display, rather than deleting the poster comments. In this case, it is assumed that this flag is stored in the distribution database DB1. Poster comments with the flag turned on are not displayed in the window W.

[0062] [3-3. Functions realized by the distributor's terminal] For example, the distributor terminal 30 includes a data storage unit 300 and a distribution unit 301. The data storage unit 300 is realized by the storage unit 32. The distribution unit 301 is realized by the control unit 31.

[0063] [Data storage section] The data storage unit 300 stores data necessary for a distributor to use a live streaming service. For example, the data storage unit 300 stores a browser for a distributor to distribute videos or an application dedicated to the live streaming service.

[0064] [Distribution Department] The distribution unit 301 distributes to viewers moving images generated by the camera 36. For example, the distribution unit 301 transmits the moving images to the server 10.

[0065] [3-4. Functions realized on the viewer device] For example, the viewer terminal 40 includes a data storage unit 400 and a viewer comment transmission unit 401. The data storage unit 400 is realized by the storage unit 42. The viewer comment transmission unit 401 is realized by the control unit 41.

[0066] [Data storage section] The data storage unit 400 stores data necessary for viewers to use the live streaming service. For example, the data storage unit 400 stores a browser for viewers to watch videos, or an application dedicated to the live streaming service.

[0067] [Viewer Comment Submission Department] The viewer comment transmitting unit 401 transmits the viewer comments to the server 10.

[0068] [4. Processing performed by the live streaming system] Figures 6 and 7 are diagrams showing an example of the processing executed in the live distribution system 1. The processing in Figures 6 and 7 is executed by the control units 11, 21, 31, and 41 operating in accordance with the programs stored in the storage units 12, 22, 32, and 42, respectively. Figures 6 and 7 show the flow of inputting viewer comments for a video distributed by a live distribution service.

[0069] 6, the distributor terminal 30 transmits a video generated by the camera 36 to the server 10 (S1). When the server 10 receives the video from the distributor terminal 30 (S2), it transmits the video to the viewer terminal 40 (S3). When the viewer terminal 40 receives the video (S4), it displays the video on the display unit 45 (S5). The viewer terminal 40 accepts viewer comments input into the input form F (S6), and transmits the viewer comments to the server 10 (S7).

[0070] When the server 10 receives the viewer comment from the viewer terminal 40 (S8), it stores the viewer comment in the distribution database DB1 and transmits a request to the viewer terminal 40 to display the viewer comment in the window W (S9). In this embodiment, a case will be described in which estimation using the first model M1 is performed after the viewer comment is displayed, but the viewer comment may be displayed after estimation using the first model M1 or the second model M2 is performed. When the viewer terminal 40 receives the request from the server 10 (S10), it displays the viewer comment in the window W (S11).

[0071] The server 10 estimates whether the viewer comment is appropriate based on the first model M1 (S12). In S12, the server 10 inputs the first input data to the first model M1 and obtains the first estimation result and the first score output from the first model M1. The first estimation result is stored in the distribution database DB1. The server 10 determines whether the first score satisfies a first criterion (S13). If it is determined that the first score satisfies the first criterion (S13: Y), the server 10 refers to the first estimation result (S14). If the estimation result of the first model M1 indicates that it is appropriate (S14: appropriate), this process ends. In this case, the viewer comment remains displayed in the window W.

[0072] If the estimation result of the first model M1 indicates that it is inappropriate in S14 (S14: inappropriate), the server 10 transmits the viewer comment to the operator terminal 20 (S15). Upon receiving the viewer comment (S16), the operator terminal 20 displays the viewer comment on the display unit 25 (S17). The operator checks the viewer comment displayed on the display unit 25. The operator terminal 20 accepts the operator's judgment result from the operation unit 24 (S18).

[0073] 7, the operator terminal 20 transmits the operator's judgment result to the server 10 (S19). When the server 10 receives the operator's judgment result (S20), it refers to the judgment result (S21). If the operator's judgment result indicates appropriateness (S21: appropriate), this process ends. In this case, the viewer comment remains displayed in the window W. If the operator's judgment result indicates inappropriateness in S21 (S21: inappropriate), the server 10 deletes the viewer comment stored in the distribution database DB1 and requests the viewer terminal 40 to delete the viewer comment from the window W (S22). When the viewer terminal 40 receives the request (S23), it deletes the viewer comment from the window W (S24), and this process ends.

[0074] In S13 of FIG. 6, if it is determined that the first score does not satisfy the first criterion (S13: N), the process moves to FIG. 7, where the server 10 estimates whether the viewer comment is appropriate or not based on the second model M2 (S25). In S25, the server 10 inputs the second input data to the second model M2 and obtains a second estimation result output from the second model M2. The second estimation result is stored in the distribution database DB1. If the estimation result of the second model M2 indicates that the comment is appropriate (S26: appropriate), the process ends. In this case, the viewer comment remains displayed in the window W. In S26, if the estimation result of the second model M2 indicates that the comment is inappropriate (S26: inappropriate), the process moves to S15. Regardless of the estimation result of the first model M1, the estimation result of the second model M2 is prioritized.

[0075] According to the live distribution system 1, when it is determined that the first score satisfies the first criterion, estimation based on the second model M2 is not performed, and when it is determined that the first score does not satisfy the first criterion, whether the viewer comment is appropriate is estimated based on the second model M2. As a result, when the first score satisfies the first criterion, the reliability of the estimation result by the first model M1 is high, so by omitting the estimation by the second model M2, the processing load of the server 10 can be reduced. Furthermore, by omitting the estimation by the second model M2, it becomes possible to estimate whether the viewer comment is appropriate or not more quickly. When the first score does not satisfy the first criterion, the reliability of the estimation result by the first model M1 is low, so by executing the estimation by the second model M2, it is possible to accurately estimate whether the viewer comment is appropriate or not. For example, by reducing the processing load of the server 10, it is possible to prevent the real-time nature of the live distribution service from being impaired, such as a delay in the video. By speeding up the estimation, it is also possible to prevent inappropriate viewer comments from remaining displayed. In a video distribution service, various videos may be distributed simultaneously, but by reducing the processing load on the server 10, it becomes possible to handle an increase in the number of videos being distributed without any problems.

[0076] In addition, in the live distribution system 1, the first model M1 is a model faster than the second model M2. This makes it possible to effectively reduce the processing load of the server 10 by first using the faster first model M1. Furthermore, it becomes possible to more quickly estimate whether the viewer comments are appropriate.

[0077] Furthermore, when it is determined that the first score does not satisfy the first criterion, the live distribution system 1 estimates whether the viewer comment is appropriate by acquiring the output of the second model M2 according to the condition branch satisfied by the viewer comment. This increases the accuracy of estimating whether the viewer comment is appropriate by the second model M2 having the condition branch.

[0078] [5. Modifications] The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit and scope of the present disclosure.

[0079] Fig. 8 is a diagram showing an example of functions realized in a modified example. As shown in Fig. 8, in the modified example described below, the server 10 includes a keyword determination unit 106, a third estimation unit 107, a second criterion determination unit 108, a fourth estimation unit 109, and a display control unit 110. These functions are realized by the control unit 11.

[0080] [Variation 1] For example, in the embodiment, the second input data input to the second model M2 is not limited to the example of the embodiment. The second input data may include the first estimation result. In the first modification, as in the embodiment, the first estimation result is exemplified as binary data indicating whether the viewer comment is appropriate or not, but may be an estimation result having an intermediate value. At least one conditional branch of the second model M2 is a condition related to the first estimation result. Depending on the first estimation result, whether the condition of the at least one conditional branch is satisfied changes.

[0081] FIG. 9 is a diagram showing an example of the processing of the second model M2. In FIG. 9, the processing of the second model M2 in the modified examples 1 to 6 is collectively shown. The second input data does not have to include all the items described in the modified examples 1 to 6, and may include only some of the items. The second model M2 in the modified example 1 has learned second training data indicating the relationship between the first estimation result of a certain second training comment and the correct answer of the appropriateness of the second training comment. The first estimation result of the second training comment may be acquired by inputting the first input data corresponding to the second training comment to the learned first model M1, or may be manually specified by the administrator. Although the contents included in the second training data are different from those in the embodiment, the learning method itself is the same as that in the embodiment.

[0082] When it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 of the first modification estimates whether the viewer comment is appropriate based on the first estimation result and the second model M2. For example, the second estimation unit 104 inputs second input data including the first estimation result to the second model M2. The second model M2 determines whether the second input data satisfies each condition of a plurality of condition branches. The second model M2 outputs a second estimation result according to the condition satisfied by the second input data. The second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring the second estimation result output from the second model M2.

[0083] When it is determined that the first score does not satisfy the first criterion, the live distribution system 1 of the first modification estimates whether the viewer comments are appropriate or not based on the first estimation result and the second model M2. Even if the first score does not satisfy the first criterion, the first estimation result may be useful to some extent, so that the estimation accuracy by the second model M2 is improved.

[0084] [Variation 2] For example, as shown in FIG. 9, the second input data may include a first score. In the second model M2 of the second modification example, second training data indicating a relationship between a first score of a second training comment and a correct answer as to whether the second training comment is appropriate or not is learned. The first score of the second training comment may be acquired by inputting the first input data corresponding to the second training comment to the learned first model M1, or may be manually specified by the administrator. Although the content included in the second training data is different from that of the embodiment, the learning method itself is the same as that of the embodiment.

[0085] When it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 of the second modification estimates whether the viewer comment is appropriate based on the first score and the second model M2. For example, the second estimation unit 104 inputs second input data including the first score to the second model M2. The second model M2 determines whether the second input data satisfies each condition of a plurality of condition branches. The second model M2 outputs a second estimation result according to the condition satisfied by the second input data. The second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring the second estimation result output from the second model M2.

[0086] When it is determined that the first score does not satisfy the first criterion, the live distribution system 1 of the second modification estimates whether the viewer comment is appropriate or not based on the first score and the second model M2. Even if the first score does not satisfy the first criterion, there may be a correlation between the first score and the appropriateness of the poster comment, so that the estimation accuracy by the second model M2 is improved.

[0087] [Variation 3] For example, as shown in FIG. 9, the second input data may include a determination result as to whether or not a keyword is included. The data storage unit 100 of the third modified example stores a keyword database DB4. For example, the keyword database DB4 stores at least one of appropriate keywords and inappropriate keywords. In the third modified example, a case will be described in which both appropriate keywords and inappropriate keywords are stored in the keyword database DB4, but the keyword database DB4 may store only either the appropriate keywords or the inappropriate keywords. The keywords stored in the keyword database DB4 are designated by the operator.

[0088] The live distribution system 1 of the third modification includes a keyword determination unit 106. The keyword determination unit 106 determines whether or not a predetermined keyword is included in the viewer comment. For example, the keyword determination unit 106 determines whether or not each of a plurality of keywords stored in the keyword database DB4 is included in the viewer comment. In the third modification, a case will be described in which the keyword determination unit 106 determines an exact match with a keyword, but the keyword determination unit 106 may also determine a partial match or a fuzzy match with a keyword (fuzzy search).

[0089] The second model M2 of the third modification has learned second training data indicating the relationship between the judgment result of whether or not a keyword is included in a second training comment and the correct answer of the appropriateness of the second training comment. The judgment result is at least one of the number of keywords, the length of the keywords, and the type of the keywords. If an appropriateness is associated with each keyword, the judgment result may be the appropriateness of the keyword. Although the content included in the second training data is different from that of the embodiment, the learning method itself is the same as that of the embodiment.

[0090] When it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 of the third modification estimates whether the viewer comment is appropriate based on the judgment result of whether the viewer comment includes a keyword and the second model M2. The meaning of the judgment result is the same as the judgment result for the second training comment. For example, the second estimation unit 104 inputs second input data including the judgment result of whether the viewer comment includes a keyword to the second model M2. The second model M2 determines whether the second input data satisfies each condition of a plurality of condition branches. The second model M2 outputs a second estimation result according to the condition satisfied by the second input data. The second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring the second estimation result output from the second model M2.

[0091] When it is determined that the first score does not satisfy the first criterion, the live distribution system 1 of the third modification estimates whether the viewer comment is appropriate or not based on the result of the determination of whether the viewer comment includes a keyword and the second model M2. This improves the estimation accuracy by the second model M2.

[0092] [Variation 4] For example, when keywords are used as in the third modification, the degree of inappropriateness may vary depending on the keyword. For this reason, a group of black words, which are the most inappropriate, and a group of gray words, which are somewhat inappropriate, may be used separately. The keyword database DB4 of the fourth modification stores a group of black words and a group of gray words. Information indicating whether a keyword is a black word or a gray word is stored in the keyword database DB4 in advance. The group of black words and the group of gray words may be stored in separate databases.

[0093] The keyword determination unit 106 of the fourth modification includes a first keyword determination unit 106A and a second keyword determination unit 106B. The first keyword determination unit 106A determines whether or not the viewer comment includes a first keyword in the first keyword group. In the fourth modification, a case where a keyword group of black words corresponds to the first keyword group will be described. In the fourth modification, a case where the first keyword determination unit 106A determines an exact match with the first keyword will be described, but the first keyword determination unit 106A may determine a partial match or a fuzzy match (fuzzy search) with the first keyword.

[0094] The second keyword determination unit 106B determines whether the viewer comment includes a second keyword in the second keyword group. In the fourth modification, a case where a keyword group of gray words corresponds to the second keyword group will be described. That is, a case where the appropriateness of the first keyword group is lower than that of the second keyword group will be described, but the appropriateness of the first keyword group may be higher than that of the second keyword group. For example, gray words may correspond to the first keyword group, and black words may correspond to the second keyword group. The first keyword group and the second keyword group are not limited to black words and gray words. For example, either the first keyword group or the second keyword group may be a white word that is an appropriate keyword. In the fourth modification, a case where the second keyword determination unit 106B determines an exact match with the second keyword will be described, but the second keyword determination unit 106B may determine a partial match or a fuzzy match (fuzzy search) with the second keyword.

[0095] The second model M2 of the modification 4 has learned second training data indicating the relationship between the judgment result of whether or not a certain second training comment includes a first keyword and the judgment result of whether or not a certain second training comment includes a second keyword, and the correct answer of this second training comment. The meaning of the judgment result is the same as in the modification 3. Although the content included in the second training data is different from that of the embodiment, the learning method itself is the same as that of the embodiment.

[0096] When it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 of the fourth modification estimates whether the viewer comment is appropriate based on a first judgment result of whether the viewer comment includes a first keyword, a second judgment result of whether the viewer comment includes a second keyword, and the second model M2. For example, the second estimation unit 104 inputs second input data including the first judgment result and the second judgment result to the second model M2. The second model M2 determines whether the second input data satisfies each condition of a plurality of condition branches. The second model M2 outputs a second estimation result according to the condition satisfied by the second input data. The second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring the second estimation result output from the second model M2.

[0097] When it is determined that the first score does not satisfy the first criterion, the live distribution system 1 of the fourth modification estimates whether the viewer comments are appropriate or not based on the first judgment result, the second judgment result, and the second model M2. This improves the estimation accuracy by the second model M2.

[0098] [Variation 5] For example, the keywords described in Variation 3 and Variation 4 may be prepared for each product attribute. In the live streaming service of Variation 5, videos introducing products are distributed, and a keyword group is prepared for each product attribute. For example, in the keyword database DB4 of Variation 5, a keyword group is prepared for each product attribute. Product attributes are information for classifying products from some perspective. Product attributes are sometimes called product categories or genres. A separate database may be prepared for each product attribute.

[0099] The keyword determination unit 106 of the fifth modified example determines whether or not a keyword is included in a viewer comment based on a group of keywords associated with attributes of a product introduced in a video that the viewer is watching. The product attributes may be specified by the distributor, or may be acquired by referring to a product database in which product attributes are defined, which is prepared in the data storage unit 100.

[0100] The keyword determination unit 106 of the modified example 5 determines whether or not a keyword is included in a viewer comment based on a group of keywords associated with product attributes in the keyword database DB4 stored in the data storage unit 100. Keyword groups associated with attributes other than the product attributes are not subject to the determination. This differs from the modified examples 3 and 4 in that the group of keywords to be determined corresponds to the product attributes, but is similar in other respects.

[0101] The live distribution system 1 of the fifth modification determines whether or not a keyword is included in a viewer comment based on a group of keywords associated with the attributes of a product introduced in a video that the viewer is watching. This allows the second model M2 to perform estimation according to the attributes of the product, thereby improving the estimation accuracy of the second model M2. For example, if the attribute of the product is clothing, a poster comment containing a word such as "butt" is often appropriate, but may be inappropriate for a product with another attribute. Therefore, by using a group of keywords according to the attribute, the estimation accuracy of the second model M2 is improved.

[0102] [Variation 6] For example, as shown in Fig. 9, the second input data may include a third estimation result by a third model M3 using a machine learning technique different from the first model M1 and the second model M2. In the sixth modification, an example is given in which the third model M3 is a model using a support vector machine. The third model M3 may be a model using a machine learning technique different from the first model M1 and the second model M2, and is not limited to a model using a support vector machine. For example, the third model M3 may be a model using a neural network.

[0103] The live distribution system 1 of the sixth modified example includes a third estimation unit 107. The third estimation unit 107 estimates whether the viewer comment is appropriate or not based on the third model M3 in which the third training comment is learned. The third model M3 learns third training data indicating a relationship between the third training comment and a correct answer as to whether the third training comment is appropriate or not. The third training data is stored in a third training database DB5 stored in the data storage unit 200. As in the embodiment, the third training data may be stored in the data storage unit 200 of the administrator terminal 20, and the learning unit 201 may execute learning of the third model M3. The learning unit 201 adjusts parameters of the third model M3 so that, when a third training comment included in certain third training data is input, an estimation result that is the correct answer included in the third training data is output.

[0104] When it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 of the sixth modification estimates whether the viewer comment is appropriate based on the estimation result of the third model M3 and the second model M2. For example, the second estimation unit 104 inputs second input data including the third estimation result to the second model M2. The second model M2 determines whether the second input data satisfies each condition of a plurality of condition branches. The second model M2 outputs a second estimation result according to the condition satisfied by the second input data. The second estimation unit 104 estimates whether the viewer comment is appropriate by acquiring the second estimation result output from the second model M2.

[0105] When it is determined that the first score does not satisfy the first criterion, the live distribution system 1 of the sixth modification estimates whether the viewer comments are appropriate or not based on the third estimation result and the second model M2. This allows the second model M2 to refer to the third estimation result, thereby improving the estimation accuracy by the second model M2.

[0106] [Variation 7] For example, in the embodiment, the second model M2 is a model that is slower but has higher accuracy than the first model M1, but the estimation accuracy of the second model M2 may be insufficient. In this case, estimation may be performed using a fourth model M4 that uses a machine learning method different from that of the first model M1 and the second model M2. In the seventh modification, a case where the fourth model M4 is a model that uses BERT is taken as an example. The fourth model M4 may be a model that uses a machine learning method different from that of the first model M1 and the second model M2, and is not limited to a model that uses BERT. For example, the fourth model M4 may be a model that uses a Transformer other than BERT. The fourth model M4 has learned fourth training data that indicates a relationship between a fourth training comment and a correct answer to whether the fourth training comment is appropriate or not. The fourth training data is stored in a fourth training database DB6 stored in the data storage unit 200.

[0107] The live distribution system 1 of the seventh modification includes a second criterion determination unit 108 and a fourth estimation unit 109. The second criterion determination unit 108 determines whether or not the second score related to the estimation accuracy of the second model M2 satisfies a predetermined second criterion. The second score differs from the first score in that the second score is a score related to the estimation accuracy of the second model M2, but is similar to the first score in other respects. The second criterion differs from the first criterion in that the second score is a criterion related to the second score, but is similar to the second criterion in other respects. Therefore, the descriptions of the first score, the first criterion, and the first model M1 in the explanation of the first score and the first criterion can be read as the second score, the second criterion, and the second model M2.

[0108] For example, the second criterion determination unit 108 acquires the second score output by the second model M2. The second criterion determination unit 108 determines whether the second score satisfies the second criterion by determining whether the second score is equal to or greater than a second threshold. If the second score is equal to or greater than the second threshold, the second criterion determination unit 108 determines that the second score satisfies the second criterion. If the second score is less than the second threshold, the second criterion determination unit 108 determines that the second score does not satisfy the second criterion.

[0109] If it is determined that the second score satisfies the second criterion, the fourth estimation unit 109 does not perform estimation based on the fourth model M4 from which the fourth training comment was learned, and if it is determined that the second score does not satisfy the second criterion, it estimates whether the viewer comment is appropriate based on the fourth model M4. If it is determined that the second score satisfies the second criterion, the fourth estimation unit 109 does not input the viewer comment to the fourth model M4, and if it is determined that the second score does not satisfy the second criterion, it inputs the viewer comment to the fourth model M4. The predetermined processing described in the embodiment is executed based on the estimation result by the fourth model M4.

[0110] In the live distribution system 1 of the seventh modification example, when it is determined that the second score satisfies the second criterion, the estimation based on the fourth model M4 from which the fourth training comment was learned is not performed, and when it is determined that the second score does not satisfy the second criterion, the estimation based on the fourth model M4 is performed to estimate whether the viewer comment is appropriate or not. As a result, even if the estimation accuracy of the second model M2 is not high, the estimation accuracy of whether the viewer comment is appropriate or not is improved by performing the estimation based on the fourth model M4. Also, when the estimation accuracy of the second model M2 is high, the second estimation result can be trusted, so that the processing load of the server 10 can be reduced by not performing the estimation based on the fourth model M4.

[0111] [Variation 8] For example, a plurality of second models M2 using different machine learning methods or a plurality of second models M2 learned from different second training data may be prepared, and the second model M2 according to the first score may be used. When it is determined that the first score does not satisfy the first criterion, the second estimation unit 104 selects one of the plurality of second models M2 based on the first score, and estimates whether the viewer comment is appropriate based on the selected second model M2.

[0112] The relationship between the first score and the second model M2 is stored in advance in the data storage unit 100. For example, the relationship is defined so that the lower the first score (the lower the accuracy of the first estimation result), the more accurate and slow the second model M2 is selected. The second estimation unit 104 may select the second model M2 associated with the first score. The estimation itself based on the second model M2 is as described in the embodiment. When a second model M2 other than the second model M2 having the conditional branch described in the embodiment is used, the second estimation result may be obtained by executing a process according to the machine learning method of the other second model M2.

[0113] In the live distribution system 1 of the eighth modification, when it is determined that the first score does not satisfy the first criterion, one of the second models M2 is selected based on the first score, and whether or not the viewer comment is appropriate is estimated based on the selected second model M2. This allows estimation based on the second model M2 according to the first score, so that it is possible to achieve a good balance between the estimation accuracy of whether or not the viewer comment is appropriate and the speed of estimating whether or not the viewer comment is appropriate. For example, when the first score does not satisfy the first criterion but the first estimation result can be trusted to a certain extent, a second estimation result with high accuracy can be obtained by high-speed processing by using the second model M2 with a certain degree of accuracy and high speed. For example, when the first score is very low and the first estimation result cannot be trusted at all, a second estimation result with high accuracy can be obtained by using the second model M2 with higher accuracy, which is not very fast.

[0114] [Variation 9] For example, an inappropriate viewer comment may be input by a particular viewer. For this reason, at least one of the first estimation unit 102 and the second estimation unit 104 may estimate whether a viewer comment is appropriate or not based on characteristics related to the viewer. The characteristics related to the viewer are characteristics related to viewer comments input in the past. For example, the characteristics are acquired such that the more times a certain viewer has input inappropriate viewer comments in the past, the higher the degree of dishonesty of this viewer.

[0115] For example, the server 10 refers to the distribution database DB1 and acquires, for each viewer, a first estimation result and a second estimation result of past viewer comments input by the viewer. The server 10 calculates the characteristics of the viewer such that the more viewer comments that are inappropriate as indicated by the first estimation result and the second estimation result, the higher the viewer's dishonesty. In the ninth modification, a case will be described in which the viewer's dishonesty is used as the characteristic, but the viewer's legitimacy may also be used as the characteristic. The server 10 may acquire characteristics according to the viewer's comment input history in other services such as SNS other than the live distribution service.

[0116] In the 9th variant, a case will be described in which both the first estimation unit 102 and the second estimation unit 104 estimate whether a viewer comment is appropriate based on characteristics related to the viewer, but only one of the first estimation unit 102 or the second estimation unit 104 may estimate whether a viewer comment is appropriate based on characteristics related to the viewer. In the 9th variant, the first training data and the second training data also include characteristics related to the viewer. The first model M1 and the second model M2 have been learned regarding characteristics related to the viewer.

[0117] For example, the first estimation unit 102 inputs first input data including characteristics related to the viewer to a first model M1. The first model M1 outputs a first estimation result according to the characteristics related to the viewer. The first estimation unit 102 acquires the first estimation result output from the first model M1. The second estimation unit 104 inputs second input data including characteristics related to the viewer to a second model M2. The second model M2 outputs a second estimation result according to the characteristics related to the viewer. The second estimation unit 104 acquires the second estimation result output from the second model M2.

[0118] Note that, instead of including the viewer's characteristics in the first input data or the second input data, a plurality of first models M1 and a plurality of second models M2 may be prepared, and the first model M1 and the second model M2 may be used according to the viewer's characteristics. For example, the first estimation unit 102 may select a first model M1 with higher accuracy as the viewer's degree of dishonesty increases. The second estimation unit 104 may select a second model M2 with higher accuracy as the viewer's degree of dishonesty increases. In other words, the first model M1 and the second model M2 may be used differently according to the viewer's characteristics.

[0119] The live distribution system 1 of the 9th modification estimates whether a viewer comment is appropriate or not based on the characteristics of the viewer. This allows the characteristics of the viewer to be taken into consideration, thereby improving the accuracy of the estimation of the viewer comments. For example, for a viewer who usually inputs inappropriate viewer comments, estimation can be performed using the first model M1 and the second model M2 that emphasize accuracy rather than speed.

[0120] [Variation 10] For example, in the embodiment, a case has been exemplified in which a viewer comment estimated to be inappropriate is sent for checking by the administrator, but a viewer comment estimated to be inappropriate may not be displayed in the window W in the first place, rather than being sent for checking by the administrator. Furthermore, a viewer comment estimated to be inappropriate by the first model M1 or the second model M2 may be displayed in the window W if it is determined to be appropriate by the administrator.

[0121] The live distribution system 1 of the tenth modification includes a display control unit 110. The display control unit 110 controls whether or not to display viewer comments based on the estimation result of at least one of the first model M1 and the second model M2. For example, for a viewer comment whose first score satisfies a first criterion, the display control unit 110 controls whether or not to display the viewer comment based on the estimation result of the first model M1. The display control unit 110 does not display viewer comments estimated to be inappropriate by the first model M1 in the window W, and displays viewer comments estimated to be appropriate by the first model M1 in the window W.

[0122] For example, for a viewer comment whose first score does not satisfy the first criterion, the display control unit 110 controls whether or not to display the viewer comment based on the estimation result of the second model M2. The display control unit 110 does not display a viewer comment estimated to be inappropriate by the second model M2 in the window W, and displays a viewer comment estimated to be appropriate by the second model M2 in the window W. Even if the viewer comment was estimated to be appropriate by the first model M1 or the second model M2, if it is determined to be inappropriate by a check by the administrator, the viewer comment may be deleted.

[0123] The live distribution system 1 of the tenth modification controls whether to display the viewer comments based on the estimation result of at least one of the first model M1 and the second model M2. This makes it possible to prevent inappropriate viewer comments from being made public.

[0124] [Other variations] For example, the above-described modifications may be combined.

[0125] For example, a function described as being realized by the server 10 may be realized by the operator terminal 20, the distributor terminal 30, the viewer terminal 40, or another computer. For example, a function described as being realized by one computer may be shared among multiple computers. Each function may be realized by at least one computer.

[0126] [6. Notes] For example, the live distribution system can be configured as follows.

[0127] (1) A viewer comment acquisition unit that acquires viewer comments input by viewers of the live streaming service; a first estimation unit that estimates whether the viewer comment is appropriate based on a first model in which a first training comment is learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a second estimation unit that, when it is determined that the first score satisfies the first criterion, does not perform estimation based on a second model from which a second training comment has been learned, and, when it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comment is appropriate based on the second model; A live streaming system including: (2) When it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on an estimation result of the first model and the second model. A live streaming system as described in (1). (3) the second estimation unit, when it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comment is appropriate or not based on the first score and the second model. A live streaming system as described in (1) or (2). (4) The live distribution system further includes a keyword determination unit that determines whether or not the viewer comments include a predetermined keyword, when it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on a determination result of whether the viewer comment includes the keyword and the second model. A live distribution system according to any one of (1) to (3). (5) The keyword determination unit is a first keyword determination unit that determines whether or not the viewer comment includes a first keyword in a first keyword group; a second keyword determination unit that determines whether or not the viewer comment includes a second keyword in a second keyword group; Including, when it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on a first determination result of whether the viewer comment includes the first keyword, a second determination result of whether the viewer comment includes the second keyword, and the second model. (4) A live distribution system as described above. (6) In the live streaming service, videos introducing products are distributed, and a group of keywords is prepared for each attribute related to the products, the keyword determination unit determines whether the viewer comment includes the keyword based on the group of keywords associated with the attribute of the product introduced in the video being viewed by the viewer; A live streaming system according to (4) or (5). (7) The live distribution system further includes a third estimation unit that estimates whether the viewer comments are appropriate based on a third model learned from a third training comment, When it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on an estimation result of the third model and the second model. A live distribution system according to any one of (1) to (6). (8) The first model is a faster model than the second model. A live distribution system according to any one of (1) to (7). (9) the second model is a model having a conditional branch, when it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate by acquiring an output according to the condition branch satisfied by the viewer comment. A live distribution system according to any one of (1) to (8). (10) The live distribution system includes: a second criterion determination unit that determines whether a second score related to the estimation accuracy of the second model satisfies a predetermined second criterion; a fourth estimation unit that, when it is determined that the second score satisfies the second criterion, does not perform estimation based on a fourth model from which a fourth training comment has been learned, and, when it is determined that the second score does not satisfy the second criterion, estimates whether the viewer comment is appropriate based on the fourth model; The live distribution system according to any one of (1) to (9), further comprising: (11) When it is determined that the first score does not satisfy the first criterion, the second estimation unit selects one of the second models based on the first score, and estimates whether the viewer comment is appropriate or not based on the selected second model. A live distribution system according to any one of (1) to (10). (12) At least one of the first estimation unit and the second estimation unit estimates whether the viewer comment is appropriate based on characteristics related to the viewer. A live distribution system according to any one of (1) to (11). (13) the live distribution system further includes a display control unit that controls whether to display the viewer comments based on an estimation result of at least one of the first model and the second model. A live distribution system according to any one of (1) to (12). [Explanation of symbols]

[0128] 1 live distribution system, 10 server, 11, 21, 31, 41 control unit, 12, 22, 32, 42 memory unit, 13, 23, 33, 43 communication unit, 20 operator terminal, 24, 34, 44 operation unit, 25, 35, 45 display unit, 30 distributor terminal, 36 camera, 40 viewer terminal, B button, F input form, N network, W window, M1 first model, M2 second model, M3 third model, M4 fourth model, 100 data storage unit, 101 viewer comment acquisition unit, 102 first estimation unit, 103 first criterion determination unit, 104 second estimation unit, 105 notification unit, 106 keyword determination unit, 107 third estimation unit, 108 second criterion determination unit, 109 fourth estimation unit, 110 display control unit, 200 data storage unit, 201 Learning unit, 202 restriction unit, 300 data storage unit, 301 distribution unit, 400 data storage unit, 401 viewer comment transmission unit, DB1 distribution database, DB2 first training database, DB3 second training database, DB4 keyword database, DB5 third training database, DB6 fourth training database.

Claims

1. A viewer comment acquisition unit that acquires viewer comments input by viewers of the live streaming service; a first estimation unit that estimates whether the viewer comment is appropriate or not based on a first model in which a first training comment has been learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a keyword determination unit that determines whether or not a predetermined keyword is included in the viewer comment; a second estimation unit that, when it is determined that the first score satisfies the first criterion, does not perform estimation based on a second model from which a second training comment has been learned, and, when it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comment is appropriate based on a determination result of whether the viewer comment includes the keyword and the second model; A live streaming system including:

2. When it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on an estimation result of the first model and the second model. The live distribution system according to claim 1 .

3. the second estimation unit, when it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comment is appropriate or not based on the first score and the second model. The live distribution system according to claim 1 or 2.

4. The keyword determination unit is a first keyword determination unit that determines whether or not the viewer comment includes a first keyword in a first keyword group; a second keyword determination unit that determines whether or not the viewer comment includes a second keyword in a second keyword group; Including, when it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on a first determination result of whether the viewer comment includes the first keyword, a second determination result of whether the viewer comment includes the second keyword, and the second model. The live distribution system according to claim 1 or 2.

5. In the live streaming service, videos introducing products are distributed, and a group of keywords is prepared for each attribute related to the products, the keyword determination unit determines whether the viewer comment includes the keyword based on the group of keywords associated with the attribute of the product introduced in the video being viewed by the viewer; The live distribution system according to claim 1 or 2.

6. The live distribution system further includes a third estimation unit that estimates whether the viewer comments are appropriate based on a third model from which third training comments are learned, When it is determined that the first score does not satisfy the first criterion, the second estimation unit estimates whether the viewer comment is appropriate or not based on an estimation result of the third model and the second model. The live distribution system according to claim 1 or 2.

7. The first model is a faster model than the second model. The live distribution system according to claim 1 or 2.

8. A viewer comment acquisition unit that acquires viewer comments input by viewers of a live streaming service; a first estimation unit that estimates whether the viewer comment is appropriate or not based on a first model in which a first training comment has been learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a second estimation unit that, when it is determined that the first score satisfies the first criterion, estimates whether the viewer comment is appropriate by acquiring an output according to the conditional branch satisfied by the viewer comment when it is determined that the first score does not satisfy the first criterion, without performing estimation based on a second model having a conditional branch learned by a second training comment; A live streaming system including:

9. The live distribution system includes: a second criterion determination unit that determines whether a second score related to the estimation accuracy of the second model satisfies a predetermined second criterion; a fourth estimation unit that, when it is determined that the second score satisfies the second criterion, does not perform estimation based on a fourth model from which a fourth training comment has been learned, and, when it is determined that the second score does not satisfy the second criterion, estimates whether the viewer comment is appropriate based on the fourth model; The live distribution system according to claim 1 , 2 or 8 , further comprising:

10. A viewer comment acquisition unit that acquires viewer comments input by viewers of a live streaming service; a first estimation unit that estimates whether the viewer comment is appropriate or not based on a first model in which a first training comment has been learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a second estimation unit that, when it is determined that the first score satisfies the first criterion, does not perform estimation based on a second model from which a second training comment has been learned, and, when it is determined that the first score does not satisfy the first criterion, selects one of a plurality of the second models based on the first score and estimates whether the viewer comment is appropriate or not based on the selected second model; A live streaming system including:

11. A viewer comment acquisition unit that acquires viewer comments input by viewers of a live streaming service; a first estimation unit that estimates whether the viewer comment is appropriate or not based on a first model in which a first training comment has been learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a second estimation unit that, when it is determined that the first score satisfies the first criterion, does not perform estimation based on a second model from which a second training comment has been learned, and, when it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comment is appropriate based on the second model; Including, At least one of the first estimation unit and the second estimation unit estimates whether the viewer comment is appropriate based on characteristics related to the viewer. Live streaming system.

12. The live distribution system further includes a display control unit that controls whether to display the viewer comments based on an estimation result of at least one of the first model and the second model. A live distribution system according to claim 1, 2, 8, 10 or 11.

13. A viewer comment acquisition step of acquiring viewer comments input by viewers of the live streaming service; a first estimation step of estimating whether the viewer comment is appropriate or not based on a first model learned from a first training comment; a first criterion determination step of determining whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a keyword determination step of determining whether or not a predetermined keyword is included in the viewer comment; a second estimation step of estimating whether the viewer comment is appropriate or not based on a determination result of whether the viewer comment includes the keyword or not and the second model when it is determined that the first score does not satisfy the first criterion, without performing estimation based on a second model from which a second training comment has been learned; and An estimation method including:

14. A viewer comment acquisition unit that acquires viewer comments input by viewers of the live streaming service; a first estimation unit that estimates whether the viewer comment is appropriate or not based on a first model in which a first training comment has been learned; a first criterion determination unit that determines whether a first score related to the estimation accuracy of the first model satisfies a predetermined first criterion; a keyword determination unit that determines whether or not a predetermined keyword is included in the viewer comment; a second estimation unit that, when it is determined that the first score satisfies the first criterion, does not perform estimation based on a second model from which a second training comment has been learned, and, when it is determined that the first score does not satisfy the first criterion, estimates whether the viewer comment is appropriate or not based on a determination result of whether the viewer comment includes the keyword and the second model; A program that makes a computer function as a

Images