Live broadcast bullet screen data crawling processing system based on keyword extraction

Abstract

The invention discloses a live broadcast bullet screen data crawling processing system based on keyword extraction, which crawls a live broadcast room bullet screen in real time, performs preprocessing, monitors a bullet screen broadcast of gateway service, acquires bullet screen data sent by a user side according to the bullet screen broadcast, collects bullet screen problem information in a live broadcast process, and sends the bullet screen problem information to the user side according to the bullet screen problem information. The bullet screen problem information comprises audience session ID, bullet screen sending time and bullet screen content. By dynamically adjusting the analysis time interval and identifying the new audience and the questioning intention thereof based on the audience session ID, the questioning of the new audience can be responded in time.

Description

Technical Field

[0001] The present invention relates to the technical field of live broadcast barrage processing, and more specifically, to a live broadcast barrage data crawling and processing system based on keyword extraction. Background Art

[0002] In the current live streaming industry, interaction between hosts and viewers primarily occurs through bullet comments. This real-time interaction greatly enhances audience engagement and makes live broadcasts more engaging. However, faced with a large and diverse number of bullet comments, hosts often struggle to respond instantly to each one. This is especially true when a large number of viewers ask the same question (e.g., product price). Repeated responses are not only time-consuming and inefficient, but also pose the risk of personal information being leaked when individuals provide their information to third-party platforms, potentially damaging their platform assets. Despite the booming live streaming industry, current oversight methods for live streaming are relatively backward. Each platform relies on either manual oversight by appointed super administrators or public reporting. With the surge in live streaming rooms and viewers, manual oversight clearly cannot meet management needs. Therefore, there is an urgent need for a new method for live streaming monitoring that can automatically monitor live broadcasts, automatically identify sensitive topics, automatically issue warnings, and operate 24 / 7. Furthermore, bullet comment data scraped through scripts relies heavily on the platform's Application Program Interface (API), which can potentially render plugin services unavailable after an API update. In order to enrich the barrage information, other interfaces are generally requested to supplement it, which leads to slower processing when there are too many barrages. The barrage data of the barrage plug-in has a large delay, affecting the live broadcast effect. Summary of the Invention

[0003] (1) Technical problems solved

[0004] In response to the problems existing in the prior art, the present invention provides a live broadcast barrage data crawling and processing system based on keyword extraction to solve the technical problems mentioned in the background technology.

[0005] (2) Technical solution

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a live broadcast barrage data crawling and processing system based on keyword extraction, comprising the following steps:

[0007] Step 1: Crawling and pre-processing the live broadcast room bullet screen in real time, monitoring the bullet screen broadcast served by the gateway, obtaining the bullet screen data sent by the user end according to the bullet screen broadcast, and collecting bullet screen problem information during the live broadcast. The bullet screen problem information includes: viewer session ID, bullet screen sending time, and bullet screen content;

[0008] Step 2: Use the BERT-based intent classification model to classify the intent of the barrage content to obtain the barrage intent classification result;

[0009] Step 3: Obtain supplementary data and cache the supplementary data; assemble the barrage data and the supplementary data into new barrage data, perform word segmentation on the barrage sentences, remove stop words, build an LDA topic model, and extract the implicit topics of the barrage;

[0010] Step 4: The new barrage data is delivered to the host terminal through user-level broadcasting, so that users of the user terminal can watch the barrage through the barrage area of ​​the host terminal.

[0011] Step 5: Dynamically adjust the analysis time interval according to the speed of barrage posting, and perform the following operations within each time interval: determine whether there is a new viewer entering the live broadcast room based on the viewer session ID, and generate a reply if a new viewer is detected and the question intention has not been answered before; determine whether the classification results of multiple barrage intentions are the same, and if the intentions are the same but there are no new viewers, and the interval between the sending times of the barrages with the same intention does not exceed the preset threshold, no reply is generated; if there is a new intention or the interval between the sending times of the barrages with the same intention exceeds the preset threshold, generate a reply corresponding to the intention;

[0012] Step 6: Compare the obtained topics with sensitive topics for similarity and identify specific sensitive topics.

[0013] The present invention is further configured such that, before monitoring the barrage broadcast of the gateway service, the method further includes: providing the anchor end with a symmetrically encrypted barrage plug-in address; upon receiving the anchor end's start-up request, decrypting the start-up request to determine whether it is a genuine request from the anchor end user.

[0014] The present invention is further configured such that the method further includes: the gateway service assigning a barrage service instance to the barrage data through a modulo operation.

[0015] The present invention is further configured such that the step 2 utilizes a BERT-based intent classification model to perform intent classification on the barrage content, specifically including the following sub-steps: S21, defining the intent category of the live barrage content and collecting corresponding barrage data; S22, pre-labeling the collected barrage data with preliminary intent, correcting the erroneous labels, and finally obtaining a stand-by barrage data set; S23, utilizing the open source BERT model as the basic model for intent classification, and performing training based on the stand-by barrage data set; S24, inputting the barrage content into the trained intent classification model, and outputting the corresponding barrage intent classification result.

[0016] The present invention is further configured such that the step one of collecting barrage question information during the live broadcast process also includes timestamping the collected barrage question information to obtain the barrage sending time.

[0017] The present invention is further configured to include an initialization timer, setting the initial value of the timer to t seconds, where t is a natural number greater than 0, and capturing and caching the barrage question information during the live broadcast through the timer; and performing an analysis operation on the barrage question information once every t seconds.

[0018] The present invention is further configured such that the supplementary data is obtained by concurrently calling corresponding services.

[0019] The present invention is further configured such that the obtaining of the supplementary data includes: concurrently calling a user service, a gift service, and a prop service to obtain the supplementary data.

[0020] (3) Beneficial effects

[0021] Compared with the existing technology, the present invention provides a live broadcast barrage data crawling and processing system based on keyword extraction, which has the following beneficial effects:

[0022] The present invention dynamically adjusts the analysis time interval and identifies new viewers and their questioning intentions based on the viewer session ID. The present invention can ensure that questions from new viewers are responded to in a timely manner, while avoiding excessive responses to repeated questions, thereby improving the audience's sense of participation and satisfaction, and enhancing the interactivity and efficiency of live broadcasts. In addition, by performing semantic analysis on the real-time related barrage of live videos, the live content can be clearly understood, and then by identifying sensitive topics, the live broadcast room that plays inappropriate content can be discovered, thereby achieving the purpose of automatic supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flowchart of the live broadcast barrage data crawling and processing system based on keyword extraction in the present invention. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meaning as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0026] See also Figure 1 The live broadcast barrage data crawling and processing system based on keyword extraction includes the following steps:

[0027] Step 1: Crawling and pre-processing the live broadcast room bullet screen in real time, monitoring the bullet screen broadcast served by the gateway, obtaining the bullet screen data sent by the user end according to the bullet screen broadcast, and collecting bullet screen problem information during the live broadcast. The bullet screen problem information includes: viewer session ID, bullet screen sending time, and bullet screen content;

[0028] Step 2: Use the BERT-based intent classification model to classify the intent of the barrage content to obtain the barrage intent classification result;

[0029] Step 3: Obtain supplementary data and cache the supplementary data; assemble the barrage data and the supplementary data into new barrage data, perform word segmentation on the barrage sentences, remove stop words, build an LDA topic model, and extract the implicit topics of the barrage;

[0030] Step 4: The new barrage data is delivered to the host terminal through user-level broadcasting, so that users of the user terminal can watch the barrage through the barrage area of ​​the host terminal.

[0031] Step 5: Dynamically adjust the analysis time interval according to the speed of barrage posting, and perform the following operations within each time interval: determine whether there is a new viewer entering the live broadcast room based on the viewer session ID, and generate a reply if a new viewer is detected and the question intention has not been answered before; determine whether the classification results of multiple barrage intentions are the same, and if the intentions are the same but there are no new viewers, and the interval between the sending times of the barrages with the same intention does not exceed the preset threshold, no reply is generated; if there is a new intention or the interval between the sending times of the barrages with the same intention exceeds the preset threshold, generate a reply corresponding to the intention;

[0032] Step 6: Compare the obtained topics with sensitive topics for similarity and identify specific sensitive topics.

[0033] In a further embodiment of the present invention, the method further includes, before monitoring the barrage broadcast of the gateway service: providing the anchor end with a symmetrically encrypted barrage plug-in address; upon receiving the broadcast start request from the anchor end, decrypting the broadcast start request to determine whether it is a real request from the anchor end user.

[0034] In a further embodiment of the present invention, the method further includes: the gateway service assigns a barrage service instance to the barrage data through a modulo operation.

[0035] Example 2:

[0036] The live broadcast barrage data crawling and processing system based on keyword extraction includes the following steps:

[0037] Step 1: Crawling and pre-processing the live broadcast room bullet screen in real time, monitoring the bullet screen broadcast served by the gateway, obtaining the bullet screen data sent by the user end according to the bullet screen broadcast, and collecting bullet screen problem information during the live broadcast. The bullet screen problem information includes: viewer session ID, bullet screen sending time, and bullet screen content;

[0038] Step 2: Use the BERT-based intent classification model to classify the intent of the barrage content to obtain the barrage intent classification result;

[0039] Step 3: Obtain supplementary data and cache the supplementary data; assemble the barrage data and the supplementary data into new barrage data, perform word segmentation on the barrage sentences, remove stop words, build an LDA topic model, and extract the implicit topics of the barrage;

[0040] Step 4: The new barrage data is delivered to the host terminal through user-level broadcasting, so that users of the user terminal can watch the barrage through the barrage area of ​​the host terminal.

[0041] Step 5: Dynamically adjust the analysis time interval according to the speed of barrage posting, and perform the following operations within each time interval: determine whether there is a new viewer entering the live broadcast room based on the viewer session ID, and generate a reply if a new viewer is detected and the question intention has not been answered before; determine whether the classification results of multiple barrage intentions are the same, and if the intentions are the same but there are no new viewers, and the interval between the sending times of the barrages with the same intention does not exceed the preset threshold, no reply is generated; if there is a new intention or the interval between the sending times of the barrages with the same intention exceeds the preset threshold, generate a reply corresponding to the intention;

[0042] Step 6: Compare the obtained topics with sensitive topics for similarity and identify specific sensitive topics.

[0043] In a further embodiment of the present invention, the step 2 uses a BERT-based intent classification model to classify the intent of the barrage content, specifically including the following sub-steps: S21, defining the intent category of the live barrage content and collecting the corresponding barrage data; S22, pre-labeling the collected barrage data with preliminary intent, correcting the wrong labels, and finally obtaining a barrage data set to be used; S23, using the open source BERT model as the basic model for intent classification, and training based on the barrage data set to be used; S24, inputting the barrage content into the trained intent classification model, and outputting the corresponding The barrage intention classification result; the barrage question information collected during the live broadcast in step one also includes timestamp marking of the collected barrage question information to obtain the barrage sending time; it also includes initializing a timer, setting the initial value of the timer to t seconds, t is a natural number greater than 0, and capturing and caching the barrage question information during the live broadcast through the timer; every t seconds, the analysis operation of the barrage question information is executed once; the supplementary data is obtained by concurrently calling the corresponding service; the acquisition of supplementary data includes: concurrently calling user services, gift services, and props services to obtain supplementary data.

[0044] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A live broadcast bullet screen data crawling and processing system based on keyword extraction, which is characterized by: The following steps are involved: Step 1: Crawling and pre-processing the live broadcast room bullet screen in real time, monitoring the bullet screen broadcast served by the gateway, obtaining the bullet screen data sent by the user end according to the bullet screen broadcast, and collecting bullet screen problem information during the live broadcast. The bullet screen problem information includes: viewer session ID, bullet screen sending time, and bullet screen content; Step 2: Use the BERT-based intent classification model to classify the intent of the barrage content to obtain the barrage intent classification result; Step 3: Obtain supplementary data and cache the supplementary data; assemble the barrage data and the supplementary data into new barrage data, perform word segmentation on the barrage sentences, remove stop words, build an LDA topic model, and extract the implicit topics of the barrage; Step 4: The new barrage data is delivered to the host terminal through user-level broadcasting, so that users of the user terminal can watch the barrage through the barrage area of ​​the host terminal. Step 5: Dynamically adjust the analysis time interval according to the speed of barrage posting, and perform the following operations within each time interval: determine whether there is a new viewer entering the live broadcast room based on the viewer session ID, and generate a reply if a new viewer is detected and the question intention has not been answered before; determine whether the classification results of multiple barrage intentions are the same, and if the intentions are the same but there are no new viewers, and the interval between the sending times of the barrages with the same intention does not exceed the preset threshold, no reply is generated; if there is a new intention or the interval between the sending times of the barrages with the same intention exceeds the preset threshold, generate a reply corresponding to the intention; Step 6: Compare the obtained topics with sensitive topics for similarity and identify specific sensitive topics.

2. The live broadcast barrage data crawling and processing system based on keyword extraction according to claim 1 is characterized by: The method further includes, before monitoring the bullet screen broadcast of the gateway service: Provide the anchor end with the symmetrically encrypted barrage plug-in address; when receiving the anchor end's broadcast request, decrypt the broadcast request to determine whether it is a real request from the anchor end user.

3. The live broadcast barrage data crawling and processing system based on keyword extraction according to claim 2 is characterized by: The method also includes: the gateway service allocates a barrage service instance to the barrage data through a modulo operation.

4. The live broadcast barrage data crawling and processing system based on keyword extraction according to any one of claims 1 to 3 is characterized in that: The second step uses a BERT-based intent classification model to classify the intent of the barrage content, specifically including the following sub-steps: S21, defining the intent category of the live barrage content and collecting corresponding barrage data; S22, pre-labeling the collected barrage data with preliminary intent, correcting incorrect labels, and finally obtaining a barrage dataset to be used; S23, using the open source BERT model as the basic model for intent classification and training it based on the barrage dataset to be used; S24. Input the barrage content into the trained intent classification model and output the corresponding barrage intent classification result.

5. The live broadcast barrage data crawling and processing system based on keyword extraction according to claim 1 is characterized by: The step 1 includes collecting barrage question information during the live broadcast process, and also includes timestamping the collected barrage question information to obtain the barrage sending time.

6. The live broadcast barrage data crawling and processing system based on keyword extraction according to claim 4 is characterized by: It also includes an initialization timer, which sets the initial value of the timer to t seconds, where t is a natural number greater than 0. The timer is used to capture and cache the barrage question information during the live broadcast; and the analysis operation of the barrage question information is performed once every t seconds.

7. The live broadcast barrage data crawling and processing system based on keyword extraction according to claim 6 is characterized by: The supplementary data is obtained by concurrently calling corresponding services.

8. The live broadcast barrage data crawling and processing system based on keyword extraction according to claim 6 is characterized by: The obtaining of the supplementary data includes: concurrently calling the user service, the gift service, and the prop service to obtain the supplementary data.

Images