Conversation method and device, electronic equipment and storage medium

Abstract

The invention relates to a dialogue method and device, electronic equipment and a storage medium, and the method comprises the steps: analyzing a matching score between a scene topic of a current playing scene and a plurality of interest topics in a user interest model of a watching user in real time in a media content playing process, responding to a condition that the matching score meets, generating personalized response information, and displaying the personalized response information. And the response information is output to the user, dialogue interaction with the user is actively performed, an active, timely and situation-synchronous interaction mode is realized, and the immersion and participation sense of the user in the film watching process are greatly enhanced. The limitation of intelligent media equipment on media understanding dimensions is broken through, and deep capture of artistic connotation can be realized. In addition, a user interaction mode is innovated, passive response is upgraded to active precise interaction, scene theme labels of the currently played media content are analyzed in real time, active dialogues are triggered based on the matching condition with a user interest model, real-time synchronization of the dialogues and the situation is achieved, and user emotion resonance is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dialogue method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of the Internet and the rapid maturation and popularization of large language model technology, the integration of artificial intelligence and multimedia interaction has become the core development direction of the industry. Existing intelligent media playback devices are no longer satisfied with the basic function of playing media content, but are being upgraded to human-computer interaction.

[0003] However, existing smart media playback devices are limited to passive interactive behaviors such as content recommendation before viewing, which cannot solve the problem of real-time interaction during user viewing. They suffer from a single dimension of understanding media content, an inability to synchronize user emotions and experiences during the interaction process, and a shallow level of interaction with users. Summary of the Invention

[0004] This application provides a dialogue method, device, electronic device, and storage medium to solve the problems of existing smart media playback devices having shallow user interaction levels, a single dimension of understanding of media content, and an inability to synchronize user emotions in real time during interaction.

[0005] Firstly, this application provides a dialogue method, the method comprising: During the playback of media content, a multimodal data stream of the media content is acquired, and a scene theme describing the current playback scene is extracted from the multimodal data stream. Obtain a user interest model associated with the current user, wherein the user interest model includes at least one interest topic generated based on the current user's historical interaction data; The target matching score is obtained by matching the scene theme with at least one interest theme in the user interest model. In response to the condition that the target matching score is met, response information is generated based on the multimodal data stream and the user interest model; The response information is output to the current user.

[0006] In one possible implementation, acquiring the multimodal data stream of the media content and extracting a scene theme describing the current playback scene from the multimodal data stream includes: Acquire the video data stream, audio data stream, and text data stream of the media content at the current playback moment; The video data stream is input into a pre-trained first model to identify the content in the video data stream and output the corresponding visual concept labels. Identify music information in the audio data stream and generate corresponding audio concept tags; Identify keywords and topic tags in the text data stream to obtain corresponding text concept tags; The visual concept tags, audio concept tags, and text concept tags are semantically normalized to obtain a scene theme describing the current playback scene.

[0007] In one possible implementation, the step of matching the scene topic with at least one interest topic in the user interest model to obtain a target matching score includes: For each interest topic in the user interest model, perform the following processing: Obtain the interest weights corresponding to the interest topics; Calculate the similarity between the interest topic and the scene topic; Based on the similarity and the interest weight, a matching score is obtained between the interest topic and the scene topic; The target matching score is determined based on the matching scores corresponding to the multiple interest topics.

[0008] In one possible implementation, obtaining the interest weights corresponding to interest topics in the user interest model includes: Retrieve the previous conversation data associated with the topic of interest; Analyze the previous dialogue data to obtain the calculation factor and time information; Based on the calculation factor and the old weight data of the interest topic, new weight data is determined, and a time decay factor is determined based on the time information; The interest weights corresponding to the interest topics are obtained by multiplying the new weight data and the time decay factor.

[0009] In one possible implementation, determining new weight data based on the calculated factor and the old weight data of the interest topic, and determining the time decay factor based on the time information, includes: By analyzing the calculated factors, we obtain the dialogue round factor, the proactive questioning factor, and the emotion score factor. New weight data is calculated based on the dialogue turn factor, proactive questioning factor, emotion score factor, and old weight data associated with the interest topic. Determine the difference between the current time and the time information, and calculate the time decay factor based on the difference and a preset time decay parameter.

[0010] In one possible implementation, generating response information based on the multimodal data stream and the user interest model includes: Based on the multimodal data stream and the user interest model, structured response information and plain text response information are generated; The structured response information and the plain text response information are identified as response information.

[0011] In one possible implementation, outputting the response information to the current user includes: Get the current user's account subscription status; When the account subscription status is in the first state, the structured response information in the response information is output to the current user; If the account subscription status is in the second state, the plain text response information in the response information will be output to the current user.

[0012] Secondly, this application provides a dialogue device, the device comprising: The scene theme extraction module is used to acquire the multimodal data stream of the media content during the playback process, and extract the scene theme describing the current playback scene from the multimodal data stream; The interest model acquisition module is used to acquire the user interest model associated with the current user, wherein the user interest model includes at least one interest topic generated based on the current user's historical interaction data. The target matching score determination module is used to match and calculate the scene theme with at least one interest theme in the user interest model to obtain the target matching score; The response information generation module is used to generate response information based on the multimodal data stream and the user interest model in response to the condition that the target matching score meets the requirement. The response information output module is used to output the response information to the current user.

[0013] In one possible implementation, the scene theme extraction module is specifically used for: Acquire the video data stream, audio data stream, and text data stream of the media content at the current playback moment; The video data stream is input into a pre-trained first model to identify the content in the video data stream and output the corresponding visual concept labels. Identify music information in the audio data stream and generate corresponding audio concept tags; Identify keywords and topic tags in the text data stream to obtain corresponding text concept tags; The visual concept tags, audio concept tags, and text concept tags are semantically normalized to obtain a scene theme describing the current playback scene.

[0014] In one possible implementation, the target matching score determination module includes: For each interest topic in the user interest model, perform the following processing: The weight acquisition unit is used to acquire the interest weights corresponding to the interest topics. A similarity determination unit is used to calculate the similarity between the interest topic and the scene topic; The matching score calculation unit is used to perform calculations based on the similarity and the interest weight to obtain the matching score between the interest topic and the scene topic. The matching score determination unit is used to determine the target matching score based on the matching scores corresponding to the multiple interest topics.

[0015] In one possible implementation, the weight acquisition unit includes: The dialogue data acquisition subunit is used to acquire the previous dialogue data associated with the topic of interest; The dialogue data parsing subunit is used to parse the previous dialogue data to obtain the calculation factor and time information; The weight data determination subunit is used to determine new weight data based on the calculation factor and the old weight data of the interest topic, and to determine the time decay factor based on the time information. The interest weight determination subunit is used to calculate the product of the new weight data and the time decay factor to obtain the interest weight corresponding to the interest topic.

[0016] In one possible implementation, the weight data determines the sub-unit, specifically for: By analyzing the calculated factors, we obtain the dialogue round factor, the proactive questioning factor, and the emotion score factor. New weight data is calculated based on the dialogue turn factor, proactive questioning factor, emotion score factor, and old weight data associated with the interest topic. Determine the difference between the current time and the time information, and calculate the time decay factor based on the difference and a preset time decay parameter.

[0017] In one possible implementation, the response information generation module is specifically used for: Based on the multimodal data stream and the user interest model, structured response information and plain text response information are generated; The structured response information and the plain text response information are identified as response information.

[0018] In one possible implementation, the response information output module is specifically used for: Get the current user's account subscription status; When the account subscription status is in the first state, the structured response information in the response information is output to the current user; If the account subscription status is in the second state, the plain text response information in the response information will be output to the current user.

[0019] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a dialogue program stored in the memory to implement the dialogue method described in any one of the first aspects.

[0020] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the dialogue method described in any one aspect.

[0021] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application analyzes the matching scores between the scene theme of the current playback scene and multiple interest themes in the user's interest model in real time during the playback of media content. In response to the matching score meeting the conditions, personalized response information is generated and output to the user, actively engaging in dialogue and interaction with the user. This achieves a proactive, timely, and context-synchronized interaction mode, greatly enhancing the user's immersion and participation during the viewing process. It breaks through the limitations of intelligent media devices on media understanding dimensions, synchronously analyzing multimodal data streams and generating multi-dimensional scene theme tags, enabling a deep capture of artistic connotations. Furthermore, it revolutionizes the user interaction mode, upgrading from passive response to proactive and precise interaction. It analyzes the scene theme tags of the currently playing media content in real time, triggering proactive dialogue based on the matching with the user's interest model, achieving real-time synchronization of dialogue and context, and enhancing user emotional resonance. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0025] Figure 1 A flowchart illustrating an embodiment of a dialogue method provided in this application; Figure 2 A flowchart illustrating an embodiment of another dialogue method provided in this application; Figure 3 A flowchart illustrating another embodiment of the dialogue method provided in this application; Figure 4 A block diagram of a dialogue device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0028] To address the technical challenges of existing smart media playback devices, such as shallow user interaction, limited understanding of media content, and inability to synchronize user emotions in real time, this application provides a dialogue method, device, electronic device, and storage medium. This enables proactive, timely, and context-synchronized interaction, significantly enhancing user immersion and engagement during viewing. It overcomes the limitations of smart media devices in understanding media dimensions, synchronously analyzing multimodal data streams to generate multi-dimensional scene theme tags, achieving a deep capture of artistic connotations. Furthermore, it revolutionizes user interaction, upgrading from passive response to proactive and precise interaction. It analyzes the scene theme tags of the currently playing media content in real time, triggering proactive dialogue based on matching with the user's interest model, achieving real-time synchronization between dialogue and context, and enhancing user emotional resonance.

[0029] Figure 1 A flowchart illustrating an embodiment of a dialogue method provided in this application, applied to a smart media playback device, includes the following steps: Step 101: During the playback of media content, acquire the multimodal data stream of the media content, and extract the scene theme describing the current playback scene from the multimodal data stream.

[0030] Multimodal data streams can refer to video data streams (dynamic images), audio data streams (sound-related information), and text data streams (such as subtitles) in media content. These data streams are synchronized and have timestamps.

[0031] Scene themes can be tags or concepts extracted from multimodal data streams that describe the current playback scene. These tags cover multiple dimensions such as artistic style, emotional atmosphere, and narrative function. For example, the tags or concepts for the current playback scene of media content could be "long take," "suspense," or "plot twist."

[0032] In one embodiment, during the playback of media content, video data stream, audio data stream, and text data stream of the media content at the current playback moment are acquired; the video data stream is input into a pre-trained first model to identify the content in the video data stream and output corresponding visual concept labels; music information in the audio data stream is identified to generate corresponding audio concept labels; keywords and topic labels in the text data stream are identified to obtain corresponding text concept labels; semantic normalization processing is performed on the visual concept labels, audio concept labels, and text concept labels to obtain a scene theme describing the current playback scene.

[0033] The first pre-trained model can refer to a multimodal recognition model that has been trained in advance on massive amounts of images and text data, and has the ability to convert visual images into text labels, such as the CLIP model (Contrastive Language-Image Pre-training).

[0034] Visual concept tags can refer to text tags extracted from video data streams that describe the content of the scene, covering dimensions such as shooting techniques, scene elements, and scene style.

[0035] Audio concept tags can refer to text tags that describe sound characteristics identified from audio data streams, including music style, ambient sound effects, and emotional tone.

[0036] Text concept tags can refer to core keywords or topic phrases extracted from text data streams, directly reflecting the core information of the text content.

[0037] Semantic normalization processing refers to the standardization of tags extracted from different modalities, including deduplication, synonym merging, and semantic classification, to ensure that the tag format is consistent and the meaning is clear, forming a structured set of scene themes.

[0038] Scene theme can refer to a multi-dimensional metadata set that can comprehensively depict the current playback scene after multimodal tag fusion and normalization, covering dimensions such as artistic techniques, emotional atmosphere, narrative function, and content elements.

[0039] Specifically, by accessing the media player's underlying API, video, audio, and text data streams are simultaneously acquired and integrated into a unified timestamp data structure. Keyframes are extracted from the media content and input into a pre-trained multimodal model (such as the CLIP model) to output visual concept labels, such as "a vast blue square" or "minimalist art." A sound classification model is used to identify music styles and sound events, generating audio concept labels such as "ambient music" or "ethereal." Keywords and topic tags, such as "truth" or "search," are extracted from letters or speech-to-text conversion. After extracting multimodal concept labels from the multimodal data streams of media content corresponding to the same timestamp, all labels are deduplicated, merged, and semantically normalized to generate a comprehensive set of scene topic tags. For example, the scene topic tag set might include: cosmic-scale aesthetics, minimalism, existentialist philosophy, etc.

[0040] For example, if a user is watching the "Qi XX Blue" segment from "Love, XX and Robots", the multimodal data stream corresponding to the same timestamp can be extracted for video analysis to identify cosmic-scale aesthetics and minimalism, audio analysis to identify a meditative atmosphere, and text analysis to extract the search for truth from the dialogue. Finally, the scene theme is output as: {cosmic-scale aesthetics, minimalism, meditative atmosphere, search for truth}.

[0041] Step 102: Obtain the user interest model associated with the current user. The user interest model includes at least one interest topic generated based on the current user's historical interaction data.

[0042] A user interest model can refer to a dynamic data structure that is strongly bound to a user's account. It is used to accurately depict a user's personalized preferences, contains multiple weighted interest topics, and is continuously updated based on the user's historical interaction data.

[0043] Historical interaction data can refer to relevant data generated by users in the past use of the system, including dialogue records with AI (dialogue rounds, follow-up questions), emotional feedback (such as positive replies and negative comments), and interactive behaviors (such as clicking dialogue invitations and ignoring responses).

[0044] Interest topics can refer to the core components of a user interest model, which are specific descriptions of user preferences (such as cyberpunk aesthetics or film composition). Each interest topic corresponds to a weight value (such as between 0 and 1) to represent the intensity of interest.

[0045] In one embodiment, by verifying the unique authentication identifier of the currently logged-in user, the system links it to the user's exclusive interest model storage address (a record in the database bound to the user's authentication identifier). The system then reads the user's interest model data from the database. This data is stored in JSON format and includes core fields such as the user's unique authentication identifier, subscription level, interest vectors (interest topics, keywords, weights, source dialogue identifiers, etc.), and last update time. The user's interest model is not static data; it is automatically updated with interest topics and corresponding weights each time the user interacts with the system, ensuring that the interest model reflects the user's latest preferences.

[0046] For example, suppose a user has had multiple in-depth discussions with AI regarding the cinematography of the film *Blade & Soul*. The historical interaction data includes three conversations about *Blade & Soul*, totaling eight rounds, with two proactive follow-up questions (such as "What is the core of cyberpunk aesthetics?"). All emotional feedback is positive. Based on this historical interaction data, the user's corresponding interest themes are calculated as: Cyberpunk Aesthetics (weight 0.9) and Film Cinematography and Composition (weight 0.85). The resulting interest model is {Cyberpunk Aesthetics: 0.9, Film Cinematography and Composition: 0.85}. This is merely an example; the specific methods for obtaining interest themes and their corresponding weights will be described in detail in the relevant embodiments below.

[0047] Step 103: Match the scene theme with at least one interest theme in the user interest model to obtain the target matching score.

[0048] Matching calculations can employ vector similarity algorithms, such as cosine similarity, to calculate the degree of correlation between scene topics and user interest topics. The core is to convert text labels into numerical vectors and then perform similarity calculations.

[0049] The target matching score refers to the quantitative result obtained after matching calculation. It is used to measure the degree of fit between the current scene topic and the user's interests. The higher the score, the stronger the association (the value range is usually 0-1). In one embodiment, each tag in the scene topic and the keyword of each interest topic in the user interest model are converted into a numerical vector. All tags in the scene topic are traversed, and vector similarity calculations, such as cosine similarity, are performed with each interest topic in the user interest model to obtain multiple preliminary similarity scores. The target matching score is determined from the multiple preliminary similarity scores.

[0050] For example, the cosmic-scale aesthetics and minimalism in the scene theme and {cyberpunk aesthetics: 0.9, cinematography and composition: 0.85} in the user interest model are vectorized with labels. Then, multiple preliminary similarities of 0.8, 0.5, 0.3, etc. are calculated by the cosine similarity algorithm. The maximum value of 0.8 among the multiple preliminary similarities is selected as the target matching score, and the interest theme in the corresponding interest model is determined.

[0051] Step 104: In response to the condition that the target matching score is met, generate response information based on multimodal data stream and user interest model.

[0052] Response information can refer to personalized content generated by combining the current multimodal data stream (deep contextual implications) and user interest model (historical preferences). It can take the form of thought-provoking questions, in-depth comments, etc., and aims to elicit deep user interaction.

[0053] In one embodiment, the target matching score is compared with a preset threshold (e.g., 0.6). If the target matching score is greater than the preset threshold, the response information generation process is triggered. If the score is less than or equal to the preset threshold, no response information is generated, and the system continues to perform silent analysis.

[0054] For example, if the target matching score is greater than a preset threshold, the structured prompt word information determined by the multimodal data stream and the user interest model will be input into the corresponding multimodal model. The model will generate personalized response information that meets the requirements based on the in-depth connotation of the multimodal data stream and the interest topics and their corresponding weights in the user interest model.

[0055] The structured prompts can include user background information, real-time context information, and task instructions. User background information can include the user's identity verification identifier, core interest topics and their weights, and dialogue information from which those interests originate. Real-time context information can include the currently playing media content, a scene description with a timestamp, and scene topic tags. Task instructions can include explicit requirements for response information, such as associating the user's historical interests with the current scene, ending with an open-ended question, and using a humble and curious tone.

[0056] For example, the model can generate various personalized response information based on the task instruction information in the structural cue words. For example, it can be personalized response information with plain text or personalized response information containing video content, etc. This is just an example. The specific method of generating the corresponding personalized response information will be described in detail in the relevant embodiments below.

[0057] Step 105: Output the response information to the current user.

[0058] In one embodiment, the generated personalized response information is presented to the current user in a non-intrusive output format through a user interface.

[0059] For example, the appropriate presentation format is selected based on the type of response information. If it is a plain text response, it can be displayed as a floating bubble or sidebar in the corner of the screen; if it is a response containing a video clip reference, the text and a clickable video thumbnail are displayed simultaneously. Furthermore, the response information is placed away from the core area of ​​media playback, and its transparency is adjustable. Users can choose to click to view details, ignore or close, or reply later, avoiding unnecessarily interrupting their viewing experience.

[0060] In addition, if the user clicks on the response information (such as clicking to view the interpretation), the system switches to a user-response interaction process, preparing to receive subsequent questions from the user and conduct multiple rounds of dialogue. If the user ignores it, the response information is automatically collapsed, and the system continues to silently analyze the next scenario.

[0061] The method provided in this application analyzes the matching scores between the scene theme of the current playback scene and multiple interest themes in the user's interest model during media content playback in real time. In response to matching scores meeting certain conditions, personalized response information is generated and output to the user, proactively engaging in dialogue and interaction. This achieves a proactive, timely, and context-synchronized interaction mode, greatly enhancing the user's immersion and participation during viewing. It breaks through the limitations of intelligent media devices in media understanding dimensions, synchronously analyzing multimodal data streams to generate multi-dimensional scene theme tags, enabling a deep capture of artistic connotations. Furthermore, it revolutionizes the user interaction mode, upgrading from passive response to proactive and precise interaction. It analyzes the scene theme tags of the currently playing media content in real time, triggering proactive dialogue based on the matching with the user's interest model, achieving real-time synchronization between dialogue and context, and enhancing user emotional resonance.

[0062] Figure 2 A flowchart illustrating another embodiment of the dialogue method provided in this application is shown below. Figure 1 Based on the illustrated process, this section mainly describes how to determine the target matching score according to the scene theme and interest theme, including the following steps: Step 201: During the playback of media content, acquire the multimodal data stream of the media content, and extract the scene theme describing the current playback scene from the multimodal data stream.

[0063] Step 202: Obtain the user interest model associated with the current user. The user interest model includes at least one interest topic generated based on the current user's historical interaction data.

[0064] For steps 201-202 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0065] Step 203: Perform the following processing for each interest topic in the user interest model: Step 231: Obtain the interest weights corresponding to the interest topics.

[0066] Interest weights quantify the strength of a user's preference for a particular topic, typically ranging from 0 to 1, with higher values ​​indicating stronger interest. A key feature is its dynamism; it is updated in real-time based on historical user interaction data (such as conversational behavior and emotional feedback) and adjusted using a time decay factor to prevent interest fixation (e.g., the weight of topics followed by a user six months ago will automatically decrease).

[0067] In one embodiment, the previous dialogue data associated with the topic of interest is obtained; the previous dialogue data is parsed to obtain the calculation factor and time information; based on the calculation factor and the old weight data of the topic of interest, new weight data is determined, and a time decay factor is determined based on the time information; the product of the new weight data and the time decay factor is calculated to obtain the interest weight corresponding to the topic of interest.

[0068] The aforementioned dialogue data refers to the data obtained directly from the previous dialogue related to the same interest topic after the current dialogue ends. This dialogue data includes calculated factors and time information. The calculated factors are the core input parameters used to update interest weights. They are parsed from the user's previous dialogue data related to the interest topic and may include the following key indicators: a dialogue round factor, representing the total number of dialogue rounds with the smart media playback device on the interest topic (e.g., 8 rounds of dialogue about cyberpunk aesthetics, factor value 8); a proactive questioning factor, representing whether the user followed up on the smart response to the interest topic (1.5 for follow-up questions, 1.0 for no follow-up questions); and a sentiment score factor, representing the sentiment tendency score of the user's dialogue content based on a sentiment analysis model (e.g., 1.2 for positive feedback, 1.0 for neutral feedback, and 0.8 for negative feedback).

[0069] The aforementioned time information can be key time nodes extracted from the previous dialogue data, specifically referring to the user's last interaction time related to the topic of interest (e.g., 10:00 AM on September 1, 2025), used to calculate the time decay factor to reflect the timeliness of the topic of interest in the user's interest model.

[0070] The old weight data refers to the historical weight value of the interest topic before this calculation, which is the basis for weight iteration and updates (e.g., after the user's last interaction, the weight of cyberpunk aesthetics was 0.85, which is the old weight data). In addition, if it is the user's first time following the interest topic, the old weight data is set to 0 by default.

[0071] The new weight data can be weight values ​​generated iteratively based on the calculated factors and the old weight data, reflecting the user's current basic preference strength for the topic of interest (without considering time decay).

[0072] The time decay factor can be a coefficient that simulates the natural forgetting of user interest topics over time (with a value range of 0.001-0.1). The value decreases as the difference between the current time and the last interaction time increases, ensuring that the weight is tilted towards the user's recent interest topics (e.g., if a user has not paid attention to a certain interest topic for 3 months, the time decay factor value of that interest topic will be less than 0.05).

[0073] Specifically, in the above embodiments, the specific implementation method for determining new weight data based on the old weight data of the calculated factors and interest topics, and determining the time decay factor based on time information is as follows: parse the calculated factors to obtain the dialogue turn factor, the proactive questioning factor, and the emotion score factor; calculate the new weight data based on the dialogue turn factor, the proactive questioning factor, the emotion score factor, and the old weight data associated with the interest topics; determine the difference between the current time and the time information, and calculate the time decay factor based on the difference and the preset time decay parameter.

[0074] For example, the new weight data can be calculated using the following formula (1): …Formula (1) in, Represents the old weighted data; Represents new weighted data; The learning rate is used to control the impact of this interaction on the overall weight, that is, to balance the stability and sensitivity of the user interest model. For example, it can be set to 0.1; Turn_Count represents the number of dialogue rounds; FollowUp_Factor represents the initiative to ask questions; Sentiment_Score represents the emotion score factor.

[0075] For example, the time decay factor can be calculated using the following formula (2): …………………………Formula (2) Here, Time_Decay represents the time decay factor. The time decay parameter reflects the rate at which an interest topic is forgotten. Its value ranges from [0.001, 0.1], covering different decay requirements from long-term interests to short-term hot topics. For example, a value of 0.005 requires 138 days for the interest weight to decay to half, suitable for simulating long-term interests such as personal hobbies; a value of 0.05 requires 14 days for the interest weight to decay to half, suitable for simulating current events. The above-mentioned parameters are described in the embodiments of this application. The value of can be 0.01, in which case the interest weight decays by half in about 69 days, which can better balance long-term memory and short-term hot topics. This is just an example, and the specific value is quite flexible. This application does not limit this. T represents the time period from the current time to the last conversation, which can be in days. It is determined based on the difference between the above time information (e.g., 10:00 on September 1, 2025) and the current time (e.g., 10:00 on September 15, 2025) (e.g., 14 days). e represents a mathematical constant, usually with a value of 2.71828.

[0076] For example, after calculating the new weight data and time decay factor of the corresponding interest topic, the product of the new weight data and the time decay factor can be calculated to obtain the current interest weight of the interest topic, as shown in the following formula (3): ……………………Formula (3) in, The current interest weight representing the topic of interest. Represents the new weight data; Time_Decay represents the time decay factor.

[0077] Step 232: Calculate the similarity between the interest topic and the scene topic.

[0078] Similarity can be a numerical value (ranging from 0 to 1) that quantifies the semantic connection between an interest topic and a scene topic. It is calculated using a vector similarity algorithm (such as cosine similarity). The higher the value, the higher the matching degree between the two (e.g., the similarity between cyberpunk aesthetics and cosmic scale aesthetics is 0.8).

[0079] In one embodiment, the labels of interest topics and scene topics are uniformly converted into text sequences. A pre-trained word embedding model is used to convert the text sequences into low-dimensional numerical vectors. The cosine of the angle between the two vectors is calculated to obtain the similarity.

[0080] Step 233: Perform calculations based on similarity and interest weights to obtain the matching score between the interest topic and the scene topic.

[0081] The setting operation can refer to the multiplicative calculation logic of similarity and interest weights corresponding to interest topics. The core purpose is to enable the combination of high interest intensity and high semantic matching to obtain a higher matching score, and to avoid accidental triggering of interaction in cases of low interest but high similarity (such as the user's slightly concerned topic matching the scene) or high interest but low similarity (such as the user's heavily concerned topic having a weak connection with the scene).

[0082] The matching score can be the final correlation value between a single interest topic and a scene topic after a set calculation, reflecting the comprehensive matching level between the interest topic and the user's interests and content value in the current scene.

[0083] For example, suppose a user's interest topic and its corresponding interest weight are: (Minimalist, 0.3). The similarity between this interest topic and the scene topic (Minimalist Scene) is 0.9. Then the matching score between this interest topic and the scene topic is 0.3 × 0.9 = 0.27. As another example, cyberpunk aesthetics (weight 0.8). The similarity between this interest topic and the scene topic cosmic scale aesthetics is 0.9. Then the matching score between this interest topic (cyberpunk aesthetics) and the scene topic (cosmic scale aesthetics) is 0.8 × 0.9 = 0.72.

[0084] Specifically, the matching score can be calculated using the following formula (4): Matching score = Similarity × Interest weight……………………Formula (4) The aforementioned similarity refers to the similarity between interest topics and scene topics.

[0085] Step 234: Determine the target matching score based on the matching scores corresponding to multiple interest topics.

[0086] The target matching score can be the maximum value selected from the matching scores corresponding to all interest topics, used to determine whether to trigger an active interaction. Choosing the maximum value instead of the average or multiple values ​​is to ensure that the interaction focus is clear (initiating dialogue only based on the point that the user is most interested in and that best matches the scene), avoiding triggering excessive interactions.

[0087] In one embodiment, the maximum value is selected from the matching scores of multiple interest topics and scene topics to determine the target matching score.

[0088] For example, assuming that the calculated matching scores of multiple interest topics and scene topics (cosmic scale aesthetics) are 0.72 (cyberpunk aesthetics), 0.25 (minimalist theme), and 0.18 (existentialist philosophy), then 0.72 is determined as the target matching score.

[0089] Step 204: In response to the condition that the target matching score is met, generate response information based on the multimodal data stream and user interest model.

[0090] Step 205: Output the response information to the current user.

[0091] For steps 204-205 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0092] pass Figure 2 The description of the illustrated embodiment, based on quantitative calculation factors such as dialogue turns, follow-up questions, and emotional feedback in the analyzed dialogue data, and combined with a time decay factor, transforms the user interest model from a static, fixed label into a dynamically evolving, numerical user interest model. This achieves dynamism, quantification, and timeliness of the user interest model, thereby ensuring high relevance and timeliness of user interaction. By selecting the maximum matching score corresponding to multiple interest topics as the decision-making basis rather than the average, it ensures that each proactive dialogue is based on a clear topic that is most interesting to the user and best matches the scenario, making the interaction focus prominent and the reasons sufficient, greatly improving the user experience.

[0093] Figure 3A flowchart illustrating another embodiment of the dialogue method provided in this application is shown. Figure 1 Based on the illustrated process, this section mainly describes how to generate and output response information, including the following steps: Step 301: During the playback of media content, acquire the multimodal data stream of the media content, and extract the scene theme describing the current playback scene from the multimodal data stream.

[0094] Step 302: Obtain the user interest model associated with the current user. The user interest model includes at least one interest topic generated based on the current user's historical interaction data.

[0095] Step 303: Match the scene theme with at least one interest theme in the user interest model to obtain the target matching score.

[0096] For steps 301-303 above, please refer to the detailed description of the relevant embodiments above.

[0097] Step 304: In response to the target matching score meeting the condition, generate structured response information and plain text response information based on multimodal data stream and user interest model.

[0098] Step 305: Determine the structured response information and plain text response information as response information.

[0099] The following is a unified description of steps 304-305 above: Structured response information can be conformal response data containing text content and multimodal reference data, expressed in a machine-parseable format. Its core feature is multimodal relevance, including not only text interpretation but also timestamps pointing to specific segments of media content, source identifiers, and other information. This can trigger multimodal interactions such as video clip capture and picture-in-picture playback. For example, when analyzing how a shot conveys despair, in addition to text interpretation, it includes timestamp data from 00:58 to 01:05, supporting synchronized playback of the corresponding video clip.

[0100] Plain text responses can be single-modal responses containing only text content, without multimodal references. Their core characteristics are simplicity and basicity. They are mainly used to answer factual and simplified questions and do not involve complex functions such as media clip extraction or in-depth analysis of audiovisual language.

[0101] In one embodiment, after determining that the target matching score meets the conditions, a highly structured prompt word is constructed. This prompt word may include system roles, user background information, current real-time context information, and key instruction information. The prompt word is then input into an advanced multimodal model to generate a response message that simultaneously contains structured response information and plain text response information.

[0102] For example, the response information output by the model can be in the following form: { "structured_response": { "text_content": "You see, this blue cube on a cosmic scale, its minimalist style is consistent with the architectural aesthetics of 'Blade Runner XX,' both exploring the relationship between grandeur and nothingness." "media_reference": { "start_timestamp": "00:15:35.000", "end_timestamp": "00:15:45.000" } }, "plain_text_response": "This cosmic-scale blue cube, with its minimalist style, is consistent with the architectural aesthetics of *Blade & Soul*, both exploring the relationship between grandeur and nothingness." }

[0103] Step 306: Obtain the current user's account subscription status.

[0104] Step 307: If the account subscription status is in the first state, output the structured response information from the response information to the current user.

[0105] Step 308: If the account subscription status is in the second state, output the plain text response information in the response message to the current user.

[0106] The following is a unified description of steps 306-308 above: Account subscription status refers to the service permission identifier bound to a user's account, used to distinguish the level of service a user can enjoy, and is divided into two core statuses. The first status: usually refers to the paid membership status, where the user has purchased premium services and can use the full functionality of the system (such as obtaining structured response information and watching multimodal response clips); the second status: usually refers to the free user status, where the user has not paid and can only use basic functions (such as obtaining plain text response information).

[0107] The response information can be a collection of interactive content that the system ultimately outputs to the user. It consists of both structured response information and plain text response information. The corresponding type of output is selected according to the user's subscription status to ensure that users with different permissions receive an appropriate interactive experience.

[0108] In one embodiment, if the user's account subscription status is in a first state, the video segment in the structured response information is parsed, the timestamp of the video segment is extracted, the timestamp is converted into parameters that the player can recognize, and the front end presents it in the form of text and clickable video thumbnails. The user can play the video segment by clicking a button. If the user's account subscription status is in a second state, the plain text response information is parsed, and the front end presents the plain text content in the form of lightweight text bubbles.

[0109] pass Figure 3 The detailed description of the illustrated embodiment shows that, based on the user's account subscription status information, it accurately matches the needs of different users. Paid users enjoy deep multimodal interaction, enhancing immersion and interaction accuracy, and satisfying the needs of advanced users for artistic interpretation and empirical verification, while free users obtain basic value information. Simultaneously, the multimodal referencing module only triggers video capture for paid users, reducing the frequency of player calls and lowering the interface call costs for media copyright holders (such as some copyright holders charging per segment capture). Based on different adaptation dimensions, it can optimize resource scheduling and reduce operating costs.

[0110] Figure 4 A block diagram of a dialogue device provided in this application embodiment, such as Figure 4 As shown, Figure 4 A dialogue device provided in this application embodiment includes: The scene theme extraction module 41 is used to acquire the multimodal data stream of the media content during the playback of the media content, and extract the scene theme describing the current playback scene from the multimodal data stream. Interest model acquisition module 42 is used to acquire a user interest model associated with the current user, wherein the user interest model includes at least one interest topic generated based on the current user's historical interaction data. The target matching score determination module 43 is used to match and calculate the scene theme with at least one interest theme in the user interest model to obtain the target matching score; The response information generation module 44 is used to generate response information based on the multimodal data stream and the user interest model in response to the matching score meeting the condition. The response information output module 45 is used to output the response information to the current user.

[0111] In one possible implementation, the scene theme extraction module 41 is specifically used for: Acquire the video data stream, audio data stream, and text data stream of the media content at the current playback moment; The video data stream is input into a pre-trained first model to identify the content in the video data stream and output the corresponding visual concept labels. Identify music information in the audio data stream and generate corresponding audio concept tags; Identify keywords and topic tags in the text data stream to obtain corresponding text concept tags; The visual concept tags, audio concept tags, and text concept tags are semantically normalized to obtain a scene theme describing the current playback scene.

[0112] In one possible implementation, the target matching score determination module 43 includes: For each interest topic in the user interest model, perform the following processing: The weight acquisition unit is used to acquire the interest weights corresponding to the interest topics. A similarity determination unit is used to calculate the similarity between the interest topic and the scene topic; The matching score calculation unit is used to perform calculations based on the similarity and the interest weight to obtain the matching score between the interest topic and the scene topic. The matching score determination unit is used to determine the target matching score based on the matching scores corresponding to the multiple interest topics.

[0113] In one possible implementation, the weight acquisition unit includes: The dialogue data acquisition subunit is used to acquire the previous dialogue data associated with the topic of interest; The dialogue data parsing subunit is used to parse the previous dialogue data to obtain the calculation factor and time information; The weight data determination subunit is used to determine new weight data based on the calculation factor and the old weight data of the interest topic, and to determine the time decay factor based on the time information. The interest weight determination subunit is used to calculate the product of the new weight data and the time decay factor to obtain the interest weight corresponding to the interest topic.

[0114] In one possible implementation, the weight data determines the sub-unit, specifically for: By analyzing the calculated factors, we obtain the dialogue round factor, the proactive questioning factor, and the emotion score factor. New weight data is calculated based on the dialogue turn factor, proactive questioning factor, emotion score factor, and old weight data associated with the interest topic. Determine the difference between the current time and the time information, and calculate the time decay factor based on the difference and a preset time decay parameter.

[0115] In one possible implementation, the response information generation module 44 is specifically used for: Based on the multimodal data stream and the user interest model, structured response information and plain text response information are generated; The structured response information and the plain text response information are identified as response information.

[0116] In one possible implementation, the response information output module 45 is specifically used for: Get the current user's account subscription status; When the account subscription status is in the first state, the structured response information in the response information is output to the current user; If the account subscription status is in the second state, the plain text response information in the response information will be output to the current user.

[0117] like Figure 5 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the dialogue method provided in any of the foregoing method embodiments, including: During the playback of media content, a multimodal data stream of the media content is acquired, and a scene theme describing the current playback scene is extracted from the multimodal data stream. Obtain a user interest model associated with the current user, wherein the user interest model includes at least one interest topic generated based on the current user's historical interaction data; The target matching score is obtained by matching the scene theme with at least one interest theme in the user interest model. In response to the matching score meeting the condition, response information is generated based on the multimodal data stream and the user interest model; The response information is output to the current user.

[0118] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the dialogue method provided in any of the foregoing method embodiments.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0122] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A dialogue method, characterized in that, The method includes: During the playback of media content, a multimodal data stream of the media content is acquired, and a scene theme describing the current playback scene is extracted from the multimodal data stream. Obtain a user interest model associated with the current user, wherein the user interest model includes at least one interest topic generated based on the current user's historical interaction data; The target matching score is obtained by matching the scene theme with at least one interest theme in the user interest model. In response to the condition that the target matching score is met, response information is generated based on the multimodal data stream and the user interest model; The response information is output to the current user.

2. The method according to claim 1, characterized in that, The step of acquiring the multimodal data stream of the media content and extracting the scene theme describing the current playback scene from the multimodal data stream includes: Acquire the video data stream, audio data stream, and text data stream of the media content at the current playback moment; The video data stream is input into a pre-trained first model to identify the content in the video data stream and output the corresponding visual concept labels. Identify music information in the audio data stream and generate corresponding audio concept tags; Identify keywords and topic tags in the text data stream to obtain corresponding text concept tags; The visual concept tags, audio concept tags, and text concept tags are semantically normalized to obtain a scene theme describing the current playback scene.

3. The method according to claim 1, characterized in that, The step of matching the scene theme with at least one interest theme in the user interest model to obtain a target matching score includes: For each interest topic in the user interest model, perform the following processing: Obtain the interest weights corresponding to the interest topics; Calculate the similarity between the interest topic and the scene topic; Based on the similarity and the interest weight, a matching score is obtained between the interest topic and the scene topic; The target matching score is determined based on the matching scores corresponding to the multiple interest topics.

4. The method according to claim 3, characterized in that, The step of obtaining the interest weights corresponding to the interest topics includes: Retrieve the previous conversation data associated with the topic of interest; Analyze the previous dialogue data to obtain the calculation factor and time information; Based on the calculation factor and the old weight data of the interest topic, new weight data is determined, and a time decay factor is determined based on the time information; The interest weights corresponding to the interest topics are obtained by multiplying the new weight data and the time decay factor.

5. The method according to claim 4, characterized in that, The process of determining new weight data based on the old weight data of the calculated factors and the interest topics, and determining the time decay factor based on the time information, includes: By analyzing the calculated factors, we obtain the dialogue round factor, the proactive questioning factor, and the emotion score factor. New weight data is calculated based on the dialogue turn factor, proactive questioning factor, emotion score factor, and old weight data associated with the interest topic. Determine the difference between the current time and the time information, and calculate the time decay factor based on the difference and a preset time decay parameter.

6. The method according to claim 1, characterized in that, The generation of response information based on the multimodal data stream and the user interest model includes: Based on the multimodal data stream and the user interest model, structured response information and plain text response information are generated; The structured response information and the plain text response information are identified as response information.

7. The method according to claim 1 or 6, characterized in that, The step of outputting the response information to the current user includes: Get the current user's account subscription status; When the account subscription status is in the first state, the structured response information in the response information is output to the current user; If the account subscription status is in the second state, the plain text response information in the response information will be output to the current user.

8. A dialogue device, characterized in that, The device includes: The scene theme extraction module is used to acquire the multimodal data stream of the media content during the playback process, and extract the scene theme describing the current playback scene from the multimodal data stream; The interest model acquisition module is used to acquire the user interest model associated with the current user, wherein the user interest model includes at least one interest topic generated based on the current user's historical interaction data. The target matching score determination module is used to match and calculate the scene theme with at least one interest theme in the user interest model to obtain the target matching score; The response information generation module is used to generate response information based on the multimodal data stream and the user interest model in response to the condition that the target matching score meets the requirement. The response information output module is used to output the response information to the current user.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a dialogue control program stored in the memory to implement the dialogue method according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the dialogue method according to any one of claims 1-7.

Images