Live broadcast bullet screen identification method and device based on multi-modal fusion

Abstract

The invention discloses a live broadcast bullet screen identification method and device based on multi-modal fusion. The method comprises the following steps: acquiring real-time monitoring parameters related to anchor behaviors in a live broadcast video; according to the real-time monitoring parameters, determining whether the anchor is performing bullet screen interaction; if the anchor is performing bullet screen interaction, obtaining a plurality of bullet screens occurring in a target time period; screening a target bullet screen of which the similarity score accords with a preset threshold value from the plurality of bullet screens; and sending the target bullet screen to all audiences of the live broadcasting room, so that the target bullet screen and the corresponding marks are displayed on the audience ends of all the audiences. According to the method, the interaction content of the anchor and the bullet screen is automatically identified and displayed, the burden of manually explaining or managing the bullet screen by a platform operation team is reduced, and the operation cost is greatly reduced. And when the audience enters the live broadcast room, the bullet screen discussed by the current anchor is automatically displayed, so that the audience is prevented from repeatedly asking questions, and the system performance pressure is reduced.

Description

Technical Field

[0001] The present application relates to the field of online live broadcasting, and in particular to a method and device for identifying live broadcast barrage based on multimodal fusion. Background Art

[0002] With the rapid development of live broadcast technology, real-time barrage interaction has become an important way for users to participate in live broadcast platforms. Audiences communicate with the anchor by sending barrages, which enhances the viewing experience. However, the existing technology still has the following problems in barrage interaction recognition: First, the manual operation cost is high. In traditional live broadcasts, the anchor or operator needs to manually identify and point out the barrage that is currently being interacted. This process is not only time-consuming and labor-intensive, but also increases the operating costs of the platform, making it difficult to achieve efficient management; second, the user experience is poor. When new viewers enter the live broadcast room, it is often difficult to quickly understand the current topic of discussion and they are prone to feel alienated, which affects the user retention rate. Summary of the invention

[0003] The present application provides a live broadcast barrage recognition method and device based on multimodal fusion, the purpose of which is to determine whether the anchor is interacting with the barrage through multimodal fusion technology, and to instantly display the barrage that the anchor is interacting with to the users in the live broadcast room.

[0004] In order to achieve the above objectives, this application provides the following technical solutions:

[0005] A live broadcast barrage recognition method based on multimodal fusion, comprising:

[0006] Obtain real-time monitoring parameters related to the anchor's behavior in the live video;

[0007] Determine whether the anchor is interacting with the bullet screen according to the real-time monitoring parameters;

[0008] If the anchor is interacting with the bullet screen, multiple bullet screens occurring within a target time period are obtained; the target time period is determined based on the time when the bullet screen interaction occurs;

[0009] Filtering a target bullet screen whose similarity score meets a preset threshold from the plurality of bullet screens; the similarity score is used to characterize the similarity between the bullet screen content and the host's voice content;

[0010] The target barrage is sent to all viewers in the live broadcast room, so that the target barrage and the corresponding mark are displayed on the audience end of all viewers; the mark is used to prompt the audience that the target barrage is the barrage that the anchor is currently interacting with.

[0011] Optionally, obtain real-time monitoring parameters related to the host's behavior in the live video, including:

[0012] Real-time monitoring of the anchor's live video; the live video includes the anchor's live screen and the anchor's live voice;

[0013] Performing face detection on the live broadcast screen of the anchor to obtain the eye features of the anchor;

[0014] Performing eye movement tracking on the eyeball features to obtain the time the anchor's sight stays in the bullet screen display area on the anchor's end;

[0015] Performing speech recognition on the live broadcast voice of the anchor to obtain corresponding semantic keywords;

[0016] Based on the semantic keywords, determining the core topic of the anchor's live broadcast voice;

[0017] Based on the dwell time and the core of the topic, real-time monitoring parameters related to the anchor's behavior are determined.

[0018] Optionally, judging whether the anchor is interacting with the bullet screen according to the real-time monitoring parameters includes:

[0019] Acquire the real-time monitoring parameters; the real-time monitoring parameters include the dwell time and the topic core;

[0020] If the dwell time exceeds a preset time threshold, and the core of the topic contains target keywords related to barrage interaction, it is determined that the anchor is conducting barrage interaction.

[0021] Optionally, if the anchor is interacting with a bullet screen, multiple bullet screens occurring within a target time period are obtained, including:

[0022] If the anchor is interacting with the bullet screen, the time when the bullet screen interaction occurs is determined based on the current system time;

[0023] Based on the occurrence time, determine the corresponding start timestamp and end timestamp; the start timestamp is earlier than the occurrence time, and the time difference between the start timestamp and the occurrence time is equal to the preset duration; the end timestamp is later than the occurrence time, and the time difference between the end timestamp and the occurrence time is equal to the preset duration;

[0024] Determine a target time period based on the start timestamp and the end timestamp;

[0025] From the barrage data stream obtained in real time by the anchor end, multiple barrages whose timestamps are in the target time period are obtained.

[0026] Optionally, selecting a target bullet comment whose similarity score meets a preset threshold from the plurality of bullet comments includes:

[0027] Acquire a live segment whose real-time monitoring parameters meet preset requirements from the live video; the real-time monitoring parameters include dwell time and topic core; the preset requirements are: the dwell time exceeds a preset time threshold, and the topic core includes target keywords related to barrage interaction;

[0028] Converting the voice information of the live broadcast segment into text to obtain a target text;

[0029] Obtaining a target vector corresponding to the target text and a plurality of sample vectors corresponding to the bullet comments;

[0030] Determining similarity scores of the multiple barrages based on the similarity between each of the multiple sample vectors and the target vector; the higher the similarity between the corresponding sample vector and the target vector, the higher the similarity score of the barrage;

[0031] A target bullet comment whose similarity score meets a preset threshold is selected from the plurality of bullet comments.

[0032] Optionally, the method further includes:

[0033] Obtaining the host interaction frequency in the current monitoring period; the host interaction frequency is determined based on the number of times the host interacts with the bullet screen;

[0034] According to the host interaction frequency, the monitoring frequency of the next monitoring cycle is adjusted to maintain the host interaction frequency of the next monitoring cycle within a specified range; the monitoring frequency represents a sampling interval of the real-time monitoring parameter.

[0035] Optionally, if the host interaction frequency is higher than a first threshold, the monitoring frequency of the next monitoring cycle is adjusted according to a preset upward adjustment rule so that the monitoring frequency of the next monitoring cycle after the adjustment is higher than the monitoring frequency of the next monitoring cycle before the adjustment; if the host interaction frequency is lower than a second threshold, the monitoring frequency of the next monitoring cycle is adjusted according to a preset downward adjustment rule so that the monitoring frequency of the next monitoring cycle after the adjustment is lower than the monitoring frequency of the next monitoring cycle before the adjustment.

[0036] A live broadcast barrage recognition device based on multimodal fusion, comprising:

[0037] A parameter acquisition unit, used to acquire real-time monitoring parameters related to the anchor's behavior in the live video;

[0038] A behavior determination unit, used to determine whether the anchor is interacting with the bullet screen according to the real-time monitoring parameters;

[0039] A barrage acquisition unit, used for acquiring multiple barrages occurring within a target time period if the anchor is performing barrage interaction; the target time period is determined based on the occurrence time of the barrage interaction;

[0040] A barrage screening unit, used to screen target barrages whose similarity scores meet a preset threshold from the plurality of barrages; the similarity score is used to characterize the similarity between the barrage content and the host's voice content;

[0041] The barrage display unit is used to send the target barrage to all viewers in the live broadcast room, so that the target barrage and the corresponding mark are displayed on the audience end of all viewers; the mark is used to prompt the audience that the target barrage is the barrage that the anchor is currently interacting with.

[0042] A storage medium, comprising a stored program, wherein the program, when executed by a processor, executes the live broadcast barrage recognition method based on multimodal fusion.

[0043] An electronic device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0044] The memory is used to store programs, and the processor is used to run programs, wherein the program executes the live broadcast barrage recognition method based on multimodal fusion when the processor runs it.

[0045] The technical solution provided by this application obtains real-time monitoring parameters related to the host's behavior in the live video. According to the real-time monitoring parameters, determine whether the host is interacting with the barrage. If the host is interacting with the barrage, obtain multiple barrages that occurred in the target time period. Filter the target barrages whose similarity scores meet the preset threshold from multiple barrages. Send the target barrage to all viewers in the live broadcast room, so that the target barrage and corresponding marks are displayed on the audience end of all viewers. The application can automatically identify and display the content of the host's interaction with the barrage, reducing the burden of manual explanation or management of the barrage by the platform operation team, and greatly reducing operating costs. Moreover, when the audience enters the live broadcast room, the barrage that the current host is discussing is automatically displayed to avoid the audience from asking repeated questions, thereby reducing the pressure on system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1A flowchart of a live broadcast barrage recognition method based on multimodal fusion provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of another method for identifying live bullet screen based on multimodal fusion provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of another method for identifying live bullet screen based on multimodal fusion provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of another method for identifying live bullet screen based on multimodal fusion provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the architecture of a live broadcast barrage recognition device based on multimodal fusion provided in an embodiment of the present application;

[0052] Figure 6 A flowchart of another method for identifying live bullet screen based on multimodal fusion provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a live broadcast screen at the audience end provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] like Figure 1As shown, it is a flow chart of a live broadcast barrage recognition method based on multimodal fusion provided in an embodiment of the present application, including the following steps.

[0057] S101: Acquire real-time monitoring parameters related to the anchor's behavior in the live video.

[0058] Among them, the live video includes the host's live screen and the host's live voice. The host's live screen is shot in real time based on the camera on the device to which the host belongs, and the host's live voice is collected in real time based on the microphone on the device to which the host belongs.

[0059] Optionally, the process of obtaining real-time monitoring parameters related to the anchor's behavior in the live video can be found in Figure 2 The steps are shown and the corresponding explanations.

[0060] S102: Determine whether the anchor is interacting with the bullet screen based on the real-time monitoring parameters.

[0061] Among them, the dwell time and topic core shown by the real-time monitoring parameters are used to determine whether the host is interacting with the barrage, which can achieve effective detection of barrage interaction behavior.

[0062] Optionally, if the dwell time exceeds a preset time threshold and the core of the topic contains target keywords related to barrage interaction, it is determined that the host is conducting barrage interaction.

[0063] In some examples, the preset time threshold can be adjusted according to different scenarios to ensure the accuracy of detection. For example, the preset time threshold can be set to 2 seconds.

[0064] In some examples, a preset keyword library and / or a dynamic keyword recognition algorithm can be used to identify whether the core of the topic contains target keywords related to the barrage interaction.

[0065] In some examples, the preset keyword library can be understood as a commonly used interactive keyword library preset in the system background. The interactive keyword library may include pre-defined keywords related to barrage interaction such as "someone asked", "barrage mentioned", and "question".

[0066] In some examples, dynamic keyword recognition algorithms can be used to dynamically extract key information from text and extract sentences related to interaction based on the context. For example, the host may mention the audience's nickname or question, and by analyzing the context, it can be determined whether this information is an interactive sentence containing target keywords related to barrage interaction.

[0067] In a possible implementation method, after obtaining the core of the topic, context analysis can be used to further confirm whether it is a barrage interaction. If the host mentions something like "An audience member just asked a question...", it is also possible to preliminarily determine whether the core of the topic contains target keywords related to the barrage interaction by analyzing the "question" and its context, that is, to determine whether the core of the topic is related to the barrage interaction.

[0068] It should be noted that the host's line of sight direction (which can determine the dwell time) and the host's voice information (which can determine the core of the topic) are used to comprehensively judge whether the host is interacting with the barrage. In order to reduce misjudgment and improve recognition accuracy, the dwell time must exceed the preset time threshold and the core of the topic must contain target keywords related to the barrage interaction. That is, the host's line of sight direction and the host's voice content must meet the corresponding trigger conditions at the same time to confirm that the host is responding to the barrage, thereby achieving double verification of the host's behavior.

[0069] In some examples, based on real-time monitoring parameters, it is determined whether the anchor is interacting with the bullet screen in real time. Figure 6 shown.

[0070] S103: If the anchor is interacting with barrage, multiple barrages occurring within the target time period are obtained.

[0071] Among them, the target time period is determined based on the time when the barrage interaction occurs.

[0072] In some examples, after determining that the host is interacting with the barrage, the time window mechanism can be activated to limit the barrage matching range to the barrage pool within N (manually set) seconds before and after the host mentions the interactive content (i.e., the target time period) to avoid matching early or irrelevant barrages.

[0073] Optionally, if the anchor is interacting with the bullet screen, the implementation process of obtaining multiple bullet screens that occurred during the target time period can be seen in Figure 3 The steps are shown and the corresponding explanations.

[0074] S104: Filtering target bullet comments whose similarity scores meet a preset threshold from multiple bullet comments.

[0075] Among them, the similarity score is used to characterize the similarity between the barrage content and the host's voice content.

[0076] In some examples, by calculating a similarity score for each barrage and the host's voice content, combined with fuzzy matching and semantic expansion technology, barrages with high similarity scores indicate that the semantics are closer to the host's discussion content. Through fuzzy matching, synonyms and different expressions can also be identified to ensure that different ways of asking questions can be captured. For example, "What time does the game start?" and "What is the time of today's game?" are the same in meaning, but different in literal meaning. Through semantic expansion, new words and related expressions other than preset keywords can also be identified. For example, if the host mentions "a viewer asked this question", the system can recognize the similarity of expressions such as "inquiry" and "question", thereby including all related barrages in the matching range. Finally, the system selects the most relevant barrages based on the similarity score, sorts them, and prioritizes the barrages with the highest semantic match.

[0077] It should be noted that semantic expansion is also a natural language processing technology, such as word sense disambiguation, synonym expansion, and context analysis. Fuzzy matching is a data matching technology used to compare two or more records and calculate the possibility that they belong to the same entity. Through semantic expansion and fuzzy matching, words that are semantically related to the host's voice content can be identified, even if these words do not exist in the original keyword library. The identified words are used to expand the matching range of the host's voice content to ensure accurate similarity scores. For example, using a word vector model, semantically similar words can be found to expand the matching range. Suppose the host is answering a question about the game time. The host says: "What time does the game start?" Through semantic expansion, the following different expressions can be identified without having to rely on exactly the same literal match: "When does the game start?", "What time is the game time?", "What time does today's game start?".

[0078] Optionally, the implementation process of selecting target bullet comments whose similarity scores meet the preset threshold from multiple bullet comments can be found in Figure 4 The steps are shown and the corresponding explanations.

[0079] S105: Send the target bullet screen to all viewers in the live broadcast room, so that the target bullet screen and corresponding marks are displayed on the audience terminals of all viewers.

[0080] Among them, the mark is used to remind the audience that the target barrage is the barrage that the anchor is currently interacting with.

[0081] It is understandable that after completing the barrage matching (i.e. determining the target barrage), the target barrage is displayed to all viewers in the live broadcast room through the push stream interface to ensure that the displayed target barrage is synchronized with the host's discussion content. In addition, a tag can be added to the target barrage, such as "current discussion barrage", so that the audience can identify the current interactive topic.

[0082] In some examples, when the target bullet screen is displayed on the viewer's end, it can remain highlighted for a certain period of time to prevent the viewer from missing the target bullet screen. For details, see Figure 7 shown.

[0083] Assume that in an online music live broadcast, the audience asks questions, comments and expresses their opinions on the performance to the host through barrages. After the performance, the host interacts with the audience and mentions: "Thank you for your support! If you have any questions about my songs or performances, please feel free to ask me." To this end, keywords (such as "questions", "support", "ask me") can be used to determine whether the host is interacting with the barrage, so as to prepare for subsequent barrage matching. Specifically, examples of barrages sent by the audience include: "What is this song about?" and "How did you come up with this interpretation?" By analyzing these barrages, relevant keywords are identified, and fuzzy matching technology is used to link different expressions. By introducing a semantic expansion algorithm, the system associates keywords with synonyms, such as associating "interpretation method" with "singing method". At the same time, the similarity score is calculated for each barrage and the host's voice content to identify the most relevant target barrage. Finally, the target barrage that is most relevant to the host's discussion content is screened out, and the target barrage with the highest similarity score is displayed first to highlight it in the live broadcast interface. Through the above implementation steps, viewers can obtain information related to the host's discussion in a timely manner, improve the real-time and smoothness of the interaction, help new viewers quickly understand the current discussion topic, avoid repeated questions, and thus improve the overall viewing experience.

[0084] In addition, in order to ensure good performance in different live broadcast scenarios, the monitoring frequency of the next monitoring cycle can also be adjusted according to the host interaction frequency of the current monitoring cycle.

[0085] Optionally, the host interaction frequency of the current monitoring period is obtained. The host interaction frequency is determined based on the number of times the host interacts with the barrage. According to the host interaction frequency, the monitoring frequency of the next monitoring period is adjusted to maintain the host interaction frequency of the next monitoring period within a specified range. The monitoring frequency represents the sampling interval of the real-time monitoring parameters.

[0086] Optionally, if the anchor interaction frequency is higher than the first threshold, the monitoring frequency of the next monitoring cycle is adjusted according to the preset upward adjustment rule, so that the monitoring frequency of the next monitoring cycle after the adjustment is higher than the monitoring frequency of the next monitoring cycle before the adjustment. If the anchor interaction frequency is lower than the second threshold, the monitoring frequency of the next monitoring cycle is adjusted according to the preset downward adjustment rule, so that the monitoring frequency of the next monitoring cycle after the adjustment is lower than the monitoring frequency of the next monitoring cycle before the adjustment.

[0087] It should be noted that the monitoring frequency of the next monitoring cycle is dynamically adjusted according to the host interaction frequency of the current monitoring cycle. The corresponding high-frequency threshold (such as more than 20 barrages sent per minute and the host answers more than 5 times per minute) and low-frequency threshold (such as less than 5 barrages sent per minute and the host does not respond) can be preset. The corresponding host interaction frequency is calculated every 5 seconds (i.e. the monitoring cycle) and compared with the thresholds (i.e. the high-frequency threshold and the low-frequency threshold). If the host interaction frequency is higher than the high-frequency threshold, the monitoring frequency of the next monitoring cycle is increased to once per second; if the host interaction frequency is lower than the low-frequency threshold, the monitoring frequency of the next monitoring cycle is reduced to once every 5 seconds; if the host interaction frequency is between the high-frequency threshold and the low-frequency threshold, the monitoring frequency of the next monitoring cycle remains the same (i.e. the monitoring frequency of the current monitoring cycle).

[0088] Combination Figure 2-Figure 4 The content shown in the example of the present application can automatically identify and display the content of the host's interaction with the bullet screen, reducing the burden of manual explanation or management of the bullet screen by the platform operation team, and greatly reducing operating costs. Moreover, when the audience enters the live broadcast room, the bullet screen that the current host is discussing is automatically displayed to avoid the audience from asking repeated questions, thereby reducing the pressure on system performance. And through dynamic detection frequency and intelligent bullet screen matching, the smoothness and efficiency of the interactive experience are ensured.

[0089] The process shown in S101-S105 above, based on the real-time monitoring parameters related to the anchor's behavior and combined with the similarity score corresponding to the barrage, can use multimodal fusion technology to determine whether the anchor is interacting with the barrage, and display the target barrage that the anchor is interacting with to the audience in the live broadcast room.

[0090] like Figure 2 As shown, a flow chart of another live broadcast barrage recognition method based on multimodal fusion provided in an embodiment of the present application includes the following steps.

[0091] S201: Monitor the anchor's live video in real time.

[0092] Among them, every time the anchor starts broadcasting, the live data stream can be collected to monitor the anchor's live video.

[0093] S202: Perform face detection on the live broadcast screen of the anchor to obtain the eye features of the anchor.

[0094] Among them, face detection is performed on the host's live broadcast screen to identify the host's key facial features, which include the features of various areas of interest (such as glasses, nose and mouth, etc.), and the host's eye features are determined from the features of various areas of interest.

[0095] S203: Perform eye movement tracking on the eyeball features to obtain the time the anchor's sight stays in the bullet screen display area on the anchor's end.

[0096] Among them, eye tracking of the host's eye features can determine the host's line of sight direction. On this basis, the host's screen is pre-divided into multiple areas, including a bullet screen display area, a live broadcast display area, and other display areas. The bullet screen display area is used to display the bullet screen obtained by the host, the live broadcast display area is used to display the host's live broadcast screen, and other display areas are used to display other content. According to the host's line of sight direction, combined with the position of the bullet screen display area on the screen, it can be identified whether the host's line of sight falls on the bullet screen display area, and the time the host's line of sight stays in the bullet screen display area on the host's end can be counted by a preset clock.

[0097] S204: Perform voice recognition on the live broadcast voice of the anchor to obtain corresponding semantic keywords.

[0098] Among them, by performing voice recognition on the host's live broadcast voice, the host's live broadcast voice can be converted into corresponding text, and the text can be semantically analyzed to obtain corresponding semantic keywords.

[0099] S205: Determine the core topic of the host's live broadcast voice based on the semantic keywords.

[0100] Among them, the number of semantic keywords can be at least one, and the core topic of the host's live voice can be determined based on a phrase composed of at least one semantic keyword.

[0101] S206: Based on the dwell time and the core of the topic, determine the real-time monitoring parameters related to the anchor's behavior.

[0102] Among them, the dwell time can reflect the host's viewing behavior, and the topic core can reflect the host's verbal behavior. The viewing behavior and verbal behavior can be regarded as multi-dimensional host behavior. Accordingly, based on the dwell time and the topic core, the real-time monitoring parameters related to the host's behavior are determined, which can monitor in real time whether the host's behavior is related to the barrage interaction.

[0103] The process shown in S201-S206 above utilizes the time the host's line of sight stays in the barrage display area on the host's end and the core topic of the host's live voice to obtain real-time monitoring parameters of multimodal fusion, so as to detect in real time whether the host is interacting with the barrage.

[0104] like Figure 3 As shown, a flow chart of another live broadcast barrage recognition method based on multimodal fusion provided in an embodiment of the present application includes the following steps.

[0105] S301: If the anchor is interacting with the bullet screen, the time when the bullet screen interaction occurs is determined based on the current system time.

[0106] Among them, if the anchor is interacting with the barrage, the current system time can be directly read from the anchor end, and the time when the barrage interaction occurs can be determined based on the current system time.

[0107] S302: Determine a corresponding start timestamp and end timestamp based on the occurrence time.

[0108] The start timestamp is earlier than the occurrence time, and the time difference between the start timestamp and the occurrence time is equal to the preset duration. The end timestamp is later than the occurrence time, and the time difference between the end timestamp and the occurrence time is equal to the preset duration.

[0109] In some examples, the preset duration can be set by technicians according to actual conditions, for example, it can be set to 5 seconds. Accordingly, the start timestamp can be regarded as a time point 5 seconds earlier than the occurrence time, and the end timestamp can be regarded as a time point 5 seconds later than the occurrence time.

[0110] S303: Determine a target time period based on the start timestamp and the end timestamp.

[0111] When determining the start timestamp and the end timestamp, the time period between the start timestamp and the end timestamp may be determined as the target time period.

[0112] S304: Acquire multiple bullet comments whose timestamps are in the target time period from the bullet comment data stream obtained in real time by the anchor end.

[0113] Among them, the barrage data stream obtained by the anchor in real time includes multiple real-time barrages and corresponding timestamps. After determining the target time period, multiple barrages with timestamps in the target time period can be directly read from the barrage data stream.

[0114] The process shown in S301-S304 above can utilize the occurrence time of the barrage interaction to filter the barrage related to the barrage interaction from the barrage data stream obtained in real time from the anchor end to avoid matching irrelevant barrages.

[0115] like Figure 4 As shown, a flow chart of another live broadcast barrage recognition method based on multimodal fusion provided in an embodiment of the present application includes the following steps.

[0116] S401: Acquire a live segment whose real-time monitoring parameters meet preset requirements from the live video.

[0117] The real-time monitoring parameters include the dwell time and the topic core. The preset requirements are: the dwell time exceeds the preset time threshold, and the topic core contains the target keywords related to the barrage interaction.

[0118] In some examples, if the real-time monitoring parameters of the live broadcast segment meet the preset requirements, it is determined that the host is interacting with the barrage in the live broadcast segment. To this end, a corresponding identifier can be added to the live broadcast segment so that the corresponding live broadcast segment can be obtained according to the identifier.

[0119] S402: Convert the voice information of the live broadcast segment into text to obtain a target text.

[0120] Among them, the speech recognition algorithm can be used to convert the voice information of the live broadcast segment into text to obtain the target text.

[0121] S403: Obtain a target vector corresponding to the target text and sample vectors corresponding to multiple bullet comments.

[0122] Among them, a word embedding model (such as a BERT model) can be used to convert the target text into a corresponding target vector, and to convert multiple bullet comments into corresponding sample vectors.

[0123] S404: Determine similarity scores of multiple bullet comments based on the similarities between each of the multiple sample vectors and the target vector.

[0124] Among them, the higher the similarity between the corresponding sample vector and the target vector, the higher the similarity score of the barrage.

[0125] In some examples, the cosine similarity algorithm can be used to calculate the similarity between the sample vector and the target vector. Generally speaking, the formula for calculating cosine similarity is the dot product of two vectors divided by the product of their moduli. The result value range is [-1, 1]. The closer the value is to 1, the more similar the semantics of the barrage content and the host's voice content are.

[0126] S405: Filtering target bullet comments whose similarity scores meet a preset threshold from multiple bullet comments.

[0127] Among them, when the similarity score corresponding to the barrage meets the preset threshold (for example, exceeds 0.8), it can be determined that the barrage content is semantically matched with the host's voice content.

[0128] In some examples, assuming that during a live broadcast, the live segment is "When does the game start?", barrage 1 is "When is today's game?", and barrage 2 is "When do you eat?" After generating the target vector and sample vector through the BERT model, the cosine similarity algorithm is used to calculate that the similarity score between barrage 1 and the live segment is 0.92, and the similarity score between barrage 2 and the live segment is 0.45. The preset threshold is 0.8, then barrage 1 can be determined as the target barrage.

[0129] The process shown in S401-S405 above uses the similarity between each of the multiple sample vectors and the target vector to determine the similarity scores of the multiple barrages, and screens the target barrages whose similarity scores meet the preset threshold, so as to accurately obtain the target barrages related to the host's barrage interaction behavior.

[0130] like Figure 5 As shown, it is a schematic diagram of the architecture of a live broadcast barrage recognition device based on multimodal fusion provided in an embodiment of the present application, including the units shown below.

[0131] The parameter acquisition unit 100 is used to acquire real-time monitoring parameters related to the host's behavior in the live video.

[0132] Optionally, the parameter acquisition unit 100 is specifically used for: real-time monitoring of the host's live video; the live video includes the host's live screen and the host's live voice; performing face detection on the host's live screen to obtain the host's eye features; performing eye tracking on the eye features to obtain the duration of the host's sight in the barrage display area on the host's end; performing voice recognition on the host's live voice to obtain corresponding semantic keywords; based on the semantic keywords, determining the topic core of the host's live voice; based on the dwell time and the topic core, determining the real-time monitoring parameters related to the host's behavior.

[0133] The behavior determination unit 200 is used to determine whether the anchor is interacting with the bullet screen based on real-time monitoring parameters.

[0134] Optionally, the behavior determination unit 200 is specifically used to: obtain real-time monitoring parameters; the real-time monitoring parameters include dwell time and topic core; if the dwell time exceeds a preset time threshold and the topic core contains target keywords related to barrage interaction, it is determined that the host is conducting barrage interaction.

[0135] The barrage acquisition unit 300 is used to acquire multiple barrages occurring within a target time period if the host is performing barrage interaction; the target time period is determined based on the time when the barrage interaction occurs.

[0136] Optionally, the barrage acquisition unit 300 is specifically used for: if the host is conducting a barrage interaction, determining the occurrence time of the barrage interaction based on the current system time; determining the corresponding start timestamp and end timestamp based on the occurrence time; the start timestamp is earlier than the occurrence time, and the time difference between the start timestamp and the occurrence time is equal to the preset duration; the end timestamp is later than the occurrence time, and the time difference between the end timestamp and the occurrence time is equal to the preset duration; determining the target time period based on the start timestamp and the end timestamp; and obtaining multiple barrages whose timestamps are in the target time period from the barrage data stream obtained in real time from the host end.

[0137] The barrage screening unit 400 is used to screen target barrages whose similarity scores meet a preset threshold from multiple barrages; the similarity score is used to characterize the similarity between the barrage content and the host's voice content.

[0138] Optionally, the barrage screening unit 400 is specifically used to: obtain a live segment whose real-time monitoring parameters meet preset requirements from the live video; the real-time monitoring parameters include dwell time and topic core; the preset requirements are: the dwell time exceeds the preset time threshold, and the topic core contains target keywords related to the barrage interaction; perform text conversion on the voice information of the live segment to obtain a target text; obtain a target vector corresponding to the target text, and sample vectors corresponding to multiple barrages; determine the similarity scores of multiple barrages based on the similarity between each of the multiple sample vectors and the target vector; the higher the similarity between the corresponding sample vector and the target vector, the higher the similarity score of the barrage; and screen target barrages whose similarity scores meet the preset threshold from multiple barrages.

[0139] The barrage display unit 500 is used to send the target barrage to all viewers in the live broadcast room, so that the target barrage and the corresponding mark are displayed on the audience end of all viewers; the mark is used to remind the audience that the target barrage is the barrage with which the host is currently interacting.

[0140] The monitoring adjustment unit 600 is used to: obtain the host interaction frequency of the current monitoring period; the host interaction frequency is determined based on the number of times the host interacts with the barrage; according to the host interaction frequency, the monitoring frequency of the next monitoring period is adjusted to maintain the host interaction frequency of the next monitoring period within a specified range; the monitoring frequency represents the sampling interval of the real-time monitoring parameters.

[0141] Optionally, the monitoring adjustment unit 600 is also used to: if the host interaction frequency is higher than a first threshold, adjust the monitoring frequency of the next monitoring cycle according to a preset upward adjustment rule, so that the monitoring frequency of the next monitoring cycle after the adjustment is higher than the monitoring frequency of the next monitoring cycle before the adjustment; if the host interaction frequency is lower than a second threshold, adjust the monitoring frequency of the next monitoring cycle according to a preset downward adjustment rule, so that the monitoring frequency of the next monitoring cycle after the adjustment is lower than the monitoring frequency of the next monitoring cycle before the adjustment.

[0142] The above-mentioned units, based on the real-time monitoring parameters related to the anchor's behavior and the similarity score corresponding to the barrage, can use multimodal fusion technology to determine whether the anchor is interacting with the barrage, and display the target barrage that the anchor is interacting with to the audience in the live broadcast room.

[0143] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the live broadcast barrage recognition method based on multimodal fusion provided by the present application.

[0144] The present application also provides an electronic device, including: a processor, a memory, and a bus. The processor and the memory are connected via a bus, the memory is used to store programs, and the processor is used to run the programs, wherein the live broadcast barrage recognition method based on multimodal fusion provided by the present application is executed when the programs are run.

[0145] In addition, the functions described above in the embodiments of the present application may be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0147] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A live broadcast barrage recognition method based on multimodal fusion, characterized in that: include: Obtain real-time monitoring parameters related to the anchor's behavior in the live video; Determine whether the anchor is interacting with the bullet screen according to the real-time monitoring parameters; If the anchor is interacting with barrages, multiple barrages occurring within the target time period are obtained; The target time period is determined based on the time when the barrage interaction occurs; Filtering a target bullet screen whose similarity score meets a preset threshold from the plurality of bullet screens; the similarity score is used to characterize the similarity between the bullet screen content and the host's voice content; The target bullet screen is sent to all viewers in the live broadcast room, so that the target bullet screen and corresponding marks are displayed on the viewer terminals of all viewers; The mark is used to prompt the audience that the target barrage is the barrage that the host is currently interacting with.

2. The method according to claim 1, characterized in that Get real-time monitoring parameters related to the anchor's behavior in the live video, including: Real-time monitoring of the anchor's live video; the live video includes the anchor's live screen and the anchor's live voice; Performing face detection on the live broadcast screen of the anchor to obtain the eye features of the anchor; Performing eye movement tracking on the eyeball features to obtain the time the anchor's sight stays in the bullet screen display area on the anchor's end; Performing speech recognition on the live broadcast voice of the anchor to obtain corresponding semantic keywords; Based on the semantic keywords, determining the core topic of the anchor's live broadcast voice; Based on the dwell time and the core of the topic, real-time monitoring parameters related to the anchor's behavior are determined.

3. The method according to claim 1, characterized in that According to the real-time monitoring parameters, determining whether the anchor is interacting with the bullet screen includes: Acquire the real-time monitoring parameters; the real-time monitoring parameters include the dwell time and the topic core; If the dwell time exceeds a preset time threshold, and the core of the topic contains target keywords related to barrage interaction, it is determined that the anchor is conducting barrage interaction.

4. The method according to claim 1, characterized in that If the anchor is interacting with bullet comments, multiple bullet comments occurring within the target time period are obtained, including: If the anchor is interacting with the bullet screen, the time when the bullet screen interaction occurs is determined based on the current system time; Based on the occurrence time, determine the corresponding start timestamp and end timestamp; the start timestamp is earlier than the occurrence time, and the time difference between the start timestamp and the occurrence time is equal to the preset duration; the end timestamp is later than the occurrence time, and the time difference between the end timestamp and the occurrence time is equal to the preset duration; Determine a target time period based on the start timestamp and the end timestamp; From the barrage data stream obtained in real time by the anchor end, multiple barrages whose timestamps are in the target time period are obtained.

5. The method according to claim 1, characterized in that: Selecting a target bullet comment whose similarity score meets a preset threshold from the plurality of bullet comments includes: Acquire a live segment whose real-time monitoring parameters meet preset requirements from the live video; the real-time monitoring parameters include dwell time and topic core; the preset requirements are: the dwell time exceeds a preset time threshold, and the topic core includes target keywords related to barrage interaction; Converting the voice information of the live broadcast segment into text to obtain a target text; Obtaining a target vector corresponding to the target text and a plurality of sample vectors corresponding to the bullet comments; Determining similarity scores of the multiple barrages based on the similarity between each of the multiple sample vectors and the target vector; the higher the similarity between the corresponding sample vector and the target vector, the higher the similarity score of the barrage; A target bullet comment whose similarity score meets a preset threshold is selected from the plurality of bullet comments.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining the host interaction frequency in the current monitoring period; the host interaction frequency is determined based on the number of times the host interacts with the bullet screen; According to the host interaction frequency, the monitoring frequency of the next monitoring cycle is adjusted to maintain the host interaction frequency of the next monitoring cycle within a specified range; the monitoring frequency represents a sampling interval of the real-time monitoring parameter.

7. The method according to claim 6, characterized in that If the host interaction frequency is higher than the first threshold, the monitoring frequency of the next monitoring cycle is adjusted according to the preset upward adjustment rule, so that the monitoring frequency of the next monitoring cycle after the adjustment is higher than the monitoring frequency of the next monitoring cycle before the adjustment; if the host interaction frequency is lower than the second threshold, the monitoring frequency of the next monitoring cycle is adjusted according to the preset downward adjustment rule, so that the monitoring frequency of the next monitoring cycle after the adjustment is lower than the monitoring frequency of the next monitoring cycle before the adjustment.

8. A live broadcast barrage recognition device based on multimodal fusion, characterized in that: include: A parameter acquisition unit, used to acquire real-time monitoring parameters related to the anchor's behavior in the live video; A behavior determination unit, used to determine whether the anchor is interacting with the bullet screen according to the real-time monitoring parameters; A barrage acquisition unit, used for acquiring multiple barrages occurring within a target time period if the anchor is interacting with the barrage; The target time period is determined based on the time when the barrage interaction occurs; A barrage screening unit, used to screen target barrages whose similarity scores meet a preset threshold from the plurality of barrages; The similarity score is used to characterize the similarity between the bullet screen content and the host's voice content; A barrage display unit, used to send the target barrage to all viewers in the live broadcast room, so that the target barrage and corresponding marks are displayed on the viewer terminals of all viewers; The mark is used to prompt the audience that the target barrage is the barrage that the host is currently interacting with.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the live broadcast barrage recognition method based on multimodal fusion as described in any one of claims 1-7.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program, when run by the processor, executes the live broadcast barrage recognition method based on multimodal fusion as described in any one of claims 1-7.

Images