Context-aware intelligent interaction system, method and device

Abstract

The application discloses a context-aware intelligent interaction system, method and device, and relates to the technical field of artificial intelligence. The system comprises: a signal acquisition module, which is used for collecting user voice instructions and streaming data of a currently played video in real time; a real-time analysis module, which is used for performing scene recognition on video key frames, recognizing voice instructions, and generating scene description labels and initial interaction requests; an intention understanding and context fusion module, which is used for fusing the above information and user historical behavior data, judging intention correlation, and generating enhanced query prompt words; a task scheduling module, which is used for sending the prompt words and related video data to a local or cloud task processing module for processing; and an interaction control module, which is used for converting processing results into voice or graphical interaction information, or executing device control instructions. The application realizes deep context-aware interaction around viewing content, and significantly improves the naturalness, accuracy and user experience of interaction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to context-aware intelligent interaction systems, methods, and devices. Background Technology

[0002] With the popularization of digital television and streaming media services, set-top boxes have evolved from traditional signal receiving and decoding devices into home entertainment centers that integrate video-on-demand and application ecosystems. Users' demands for intelligent and personalized interactive experiences are growing daily.

[0003] Currently, most smart set-top boxes on the market support basic voice interaction functions, such as searching for programs, adjusting volume, or launching applications after being woken up by keywords. However, such interactions have significant limitations: firstly, the interaction mode is mechanical and fragmented, and the system can only understand preset simple commands or keywords, unable to understand the complex intentions and contextual relationships contained in the user's natural language; secondly, the interaction process is completely disconnected from the user's viewing behavior (for example, when the user asks a question related to the content being viewed, the existing system cannot automatically associate it with the actors on the screen, but can only perform an isolated web search), resulting in an inconsistent and unintelligent interactive experience.

[0004] While existing solutions have proposed adding AI functionality to traditional set-top boxes through external assistant devices to enhance their ability to handle complex intentions, these solutions either lack deep integration with video content or are not sufficiently intelligent, failing to fundamentally solve the core problem of personalized AI interaction based on real-time playback content.

[0005] Therefore, existing technologies lack a solution that can deeply understand users' natural language commands and perform context-aware and intelligent interaction in real time by combining the currently playing video content. Summary of the Invention

[0006] This invention provides a context-aware intelligent interaction system, method, and device. By analyzing video streams in real time and integrating them with user commands, AI interaction can closely revolve around the viewed content, achieving natural, coherent, and personalized intelligent interaction. This solves the problem of disconnect between the interaction provided by existing smart terminals (such as set-top boxes) and the viewed content.

[0007] This invention is achieved through the following technical solution:

[0008] A first aspect of the present invention provides a context-aware intelligent interaction system, comprising:

[0009] The signal acquisition module is used to acquire voice commands and streaming data of the currently playing video in real time; the streaming data includes video keyframes.

[0010] The real-time analysis module is used to perform target recognition processing on the video keyframes, speech recognition processing on the audio stream, generate structured scene description tags based on the target recognition results, and generate initial interaction requests based on the speech recognition results; the target recognition processing includes the recognition of text, objects, faces, and scene types in the image.

[0011] The intent understanding and context fusion module is used to perform intent parsing and context fusion based on the initial interaction request, the scene description tag, user historical behavior data, and the current application state, and generate enhanced query prompts containing scene information; the intent parsing includes: determining whether the initial interaction request is related to the content of the currently playing video; the user historical behavior data includes historical viewing data;

[0012] The task scheduling module is used to send the enhanced query prompts and related video stream data segments to the task processing module for processing, and to receive the processing results fed back by the task processing module.

[0013] An interactive control module is used to convert the processing results into interactive information and / or generate device control commands based on the processing results.

[0014] Furthermore, the task processing module includes a local processing module and a cloud-based AI collaboration module;

[0015] The task scheduling module is also used to: dynamically decide whether to send the enhanced query suggestion and related video stream data segments to the local processing module or the cloud AI collaboration module based on at least one of the factors of the complexity of the enhanced query suggestion, the real-time requirements of the task, and the privacy level of the task.

[0016] Furthermore, the local processing module is a local micro-model execution module or a rule engine, used to execute tasks with a complexity lower than the first threshold, tasks with a real-time requirement higher than the second threshold, and / or privacy tasks.

[0017] Furthermore, the task scheduling module also includes:

[0018] The multimodal large model interface module is used to call the generative artificial intelligence model to perform deep semantic understanding and content generation tasks based on the enhanced query prompts and related video stream data segments;

[0019] The personalized model adaptation module is used to fine-tune the output of the generative artificial intelligence model based on anonymized user historical behavior data, so as to generate feedback content that conforms to the user's personalized preferences.

[0020] Furthermore, the task scheduling module makes routing decisions based on a multi-dimensional scoring function;

[0021] The input to the scoring function includes at least a content relevance strength factor, a computational complexity factor, a real-time requirement factor, and a privacy sensitivity factor, and the output is a comprehensive score; the content relevance strength factor represents the degree of relevance between the scene description tags explicitly referenced in the enhanced query suggestions and the initial interaction request;

[0022] If the overall score is higher than a preset threshold, the enhanced query suggestions and related video stream data segments will be routed to the cloud AI collaboration module; otherwise, they will be routed to the local processing module.

[0023] Furthermore, the interactive control module includes:

[0024] The multimodal feedback rendering module is used to convert the interactive information into voice broadcast or generate a graphical information layer superimposed on the screen of the currently playing video.

[0025] The device control execution module is used to control the device to perform corresponding operations according to the device control instructions, including: channel switching, on-demand playback, and search.

[0026] A second aspect of the present invention provides a context-aware intelligent interaction method, applied to the context-aware intelligent interaction system described in any one of the first aspects of the present invention, comprising the following steps:

[0027] Responding to user interaction needs, it collects user voice commands and streaming data of the currently playing video in real time;

[0028] The streaming data is analyzed in real time to generate structured scene description tags;

[0029] The voice command is recognized in real time to obtain the initial interaction request;

[0030] Based on the initial interaction request, the scene description tags, user historical behavior data, and the current application status, intent parsing and context fusion are performed to generate enhanced query prompts containing scene information;

[0031] According to the preset routing strategy, the enhanced query prompts and related video stream data segments are sent to the task processing module for processing to obtain the processing results;

[0032] Receive the processing result, convert the processing result into interactive information, and / or generate device control commands based on the processing result.

[0033] Furthermore, the routing strategy is determined based on a multi-dimensional scoring function;

[0034] The input to the scoring function includes at least a content relevance strength factor, a computational complexity factor, a real-time requirement factor, and a privacy sensitivity factor, and the output is a comprehensive score; wherein, the content relevance strength factor represents the relevance of the enhanced query suggestion to the currently playing video content;

[0035] If the overall score is higher than a preset threshold, the enhanced query suggestions and related video stream data segments will be routed to the cloud AI collaboration module; otherwise, they will be routed to the local processing module.

[0036] Furthermore, based on the initial interaction request, the scene description tags, user historical behavior data, and the current application state, intent parsing and context fusion are performed to generate enhanced query suggestions containing scene information, including:

[0037] Determine the relevance between the user command and the currently playing content;

[0038] If the correlation is strong, the element most relevant to the initial interaction request is extracted from the scene description tag and inserted or concatenated into the initial interaction request in natural language form to generate the enhanced query suggestion word;

[0039] If the correlation is weak or irrelevant, the user's historical behavior data and / or the current application status will be used as auxiliary information and fused with the initial interaction request to generate the enhanced query suggestion.

[0040] A third aspect of the present invention provides an electronic device having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the context-aware intelligent interaction method described in any one of the second aspects of the present invention.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] By analyzing video streams in real time and integrating them with user commands, AI interaction can closely revolve around the content being viewed, thus solving the core problem of the disconnect between interaction and content.

[0043] Through intelligent task routing, with limited hardware resources, it takes into account both the powerful capabilities of large cloud models and the advantages of low latency and high privacy of local processing, ensuring the availability of basic functions when the network is not smooth.

[0044] By using users' historical behavior data, the output of generative artificial intelligence models can be fine-tuned to make the interaction more natural and the feedback more accurate and personalized. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0046] Figure 1 This is an overall architecture diagram of a context-aware intelligent interaction system according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of a task scheduling working logic according to an embodiment of the present invention;

[0048] Figure 3 This is a flowchart of a context-aware intelligent interaction method according to an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of a context-aware intelligent interaction method according to an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0051] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to other steps or modules inherent in the device.

[0052] The terminology used in the various embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. The terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0053] Embodiments of this invention provide a context-aware intelligent interaction system. Its core lies in achieving a deep understanding of user commands and accurate responses based on video context through the synergy of local real-time perception and generative artificial intelligence capabilities. This system is applicable to set-top boxes and similar terminal devices with audio and video playback functions, significantly enhancing the intelligence level of home entertainment interaction.

[0054] Please see Figure 1 It illustrates the overall architecture of one embodiment of the system, which mainly includes: a signal acquisition module 101, a real-time analysis module 102, an intent understanding and context fusion module 103, a task scheduling module 104, and an interaction control module 105.

[0055] The signal acquisition module 101 is responsible for synchronously capturing two types of key data: 1) User voice commands: acquired through the device's built-in or external microphone array; 2) Streaming data of the currently playing video: captured through HDMI input or asynchronously extracted directly from the internally decoded video pipeline.

[0056] Streaming data is a multimodal data set that mainly includes periodically extracted video keyframes, accompanying audio streams, metadata, and possibly subtitle streams.

[0057] Periodically extracted video keyframes are used to generate scene description labels, which is the core source of visual context.

[0058] The real-time analysis module 102 performs target recognition processing on video keyframes to obtain target recognition results; and performs speech recognition (Speech To Text, STT) processing on voice commands to obtain speech recognition results.

[0059] The target recognition processing includes recognizing text (subtitles, signs, titles, etc.), objects (cars, buildings, teacups, books, etc.), faces, and scene types (indoor, outdoor, daytime, nighttime, sports field, etc.) in keyframes of the video.

[0060] The target recognition process generates the following recognition results:

[0061] Object labeling: Identifies the names of specific objects appearing in the image;

[0062] Face-related tags: Detect whether the image contains a face. If it does, further analyze the face attributes: age range, gender, and face recognition: whether it is a well-known actor or public figure.

[0063] Scene category tags: Identify the overall environment or category of the scene, such as "indoor living room", "outdoor street", "natural scenery", "studio", "night scene", "sports scene", etc.

[0064] Text labels: If the image contains text information such as subtitles, signs, or titles, optical character recognition can be performed.

[0065] Activity / Action Tags: Identify the behavioral state of the main people or objects in the scene, such as "running", "talking", "driving", "cooking", etc.

[0066] The above recognition results are integrated and output as structured scene description labels, such as: "The scene is an indoor living room scene, containing two people (an adult male and a child) who are talking. There are teacups and books on the table."

[0067] Real-time processing of user voice commands can be based on wake word detection. After detecting the wake word, STT processing is performed on the audio stream following the wake word to convert the voice command into text, detect the command word, and generate an initial interaction request in plain text form.

[0068] During the wake-word detection phase, the system continuously monitors the audio stream from the signal acquisition module and uses a lightweight local voice wake-up model to detect whether it contains a preset wake-word. This process only performs simple audio pattern matching to determine if the user intends to interact; it does not perform full speech-to-text conversion. Only after detecting the wake-word does the system officially enter the interactive state and begin recording subsequent voice commands.

[0069] After successful wake-up, the system enters the voice command recognition stage. The system performs voice recognition processing on a continuous speech segment following the wake-up word (i.e., the user's actual command), converts the speech into text, and generates an initial interaction request.

[0070] By detecting wake words, the system reduces standby power consumption and avoids continuous high-energy-consuming voice recognition; it also prevents accidental triggering and ensures that the system only responds when the user clearly expresses an interaction intention, thus protecting user privacy.

[0071] The design goal of this module is to achieve real-time, lightweight feature extraction and transformation within the limited computing power of the set-top box. A lightweight neural network model can be built into the real-time analysis module 102 to perform target recognition processing and speech recognition processing in parallel; or an existing image detection and speech recognition engine can be invoked.

[0072] Furthermore, the accompanying audio stream can be used to assist in voice command endpoint detection and noise reduction. For example, in noisy playback environments, by comparing the user's voice with the background audio, the accuracy of voice wake-up and recognition can be improved. Additionally, by performing event detection (such as laughter, applause, and explosions) or speech recognition (extracting dialogue text from characters in the show) on the audio stream, it can supplement video keyframe analysis to generate richer scene description labels. For example, detecting an explosion can supplement the label with "action scene".

[0073] Subtitle streams can provide precise textual context. When user commands involve specific lines, names, or locations (e.g., "What does 'Project Alpha' mean in the subtitles?"), the subtitle stream provides the most direct and accurate source of textual references, which can be extracted and integrated into the user command to create enhanced query suggestions. Simultaneously, subtitle text can be used as natural language input, cross-modal correlated with the recognition results of video keyframes, improving the accuracy of understanding higher-level semantics such as plot and character relationships.

[0074] Video metadata mainly includes program guide information, episode titles, cast lists, and category tags. It primarily provides structured knowledge, offering accurate program information for intent understanding. For example, it can directly indicate that "the currently playing video is 'Movie A,' and the main cast includes actor B." Metadata can corroborate or supplement real-time recognition results. For instance, matching a recognized "face" with the "cast list" in the metadata directly yields the character's name.

[0075] The real-time analysis module 102 comprehensively utilizes one or more of the above-mentioned streaming data to provide a more comprehensive and accurate context-aware foundation for subsequent modules.

[0076] The intent understanding and context fusion module 103 receives the initial interaction request and scene description tag from the real-time analysis module, and performs preliminary semantic understanding by combining locally stored user historical behavior data (such as viewing history, viewing preferences, and search history) and the current application state (such as playback state, interface state, and system state). Its core operations include:

[0077] (a) Intent analysis: Determine the relevance of the user's command to the currently playing content. For example, when a user is watching a movie, if the voice command is "Who is he?" and the video keyframes identify a face, it is very likely related to the person in the picture, indicating that the user's command is strongly related to the currently playing content; while "What will the weather be like tomorrow?" is weakly related to or unrelated to the currently playing content.

[0078] The current application state refers to the running state and context information of the device or application at the moment of interaction triggering, and mainly includes at least one of the following:

[0079] Playback status: such as playing, paused, fast forward, rewind, stopped, etc.;

[0080] Content source information: The application (such as "Tencent Video", "iQiyi"), channel (such as "CCTV-1"), or content type (such as "movie", "news live", "episode 3 of TV series") to which the currently playing content belongs;

[0081] Interface status: The user interface that the device is currently in, such as full-screen playback interface, application menu home page, electronic program guide list, search page, settings page, etc.;

[0082] System status: such as network connection status (online / offline), time information (day / night), device volume, screen brightness, etc.

[0083] The current application state is crucial for accurately interpreting user intent and generating context-appropriate responses. For example, when a user issues the instruction "switch to the next one," if the current application state is "playing a movie," the intent is interpreted as "play the next movie"; if the state is "browsing an electronic program guide list," the intent is interpreted as "select the next channel in the list."

[0084] (ii) Context fusion: Based on the relevance judgment results between user instructions and the currently playing content, different fusion strategies are used to generate enhanced query suggestions.

[0085] (1) If it is determined to be strongly related, the elements most relevant to the initial interaction request (such as the identified specific person, object, or scene category) are extracted from the scene description tags of the current moment or related moment (time before and after the current moment) as key context, and inserted or spliced ​​into the initial interaction request in the form of natural language to generate an enhanced query prompt word rich in video context.

[0086] For example, given the initial request "What movies has this person acted in?" and the scene description tag "The person in the picture is actor Zhang San", the system determines that the initial interaction request points to or may involve the currently playing video content, classifying it as strongly relevant. The relevant elements are then merged to generate an enhanced query suggestion: "The person in the current video is actor Zhang San. What other movies has actor Zhang San acted in?". This enhanced query suggestion is then sent to generative artificial intelligence for processing, increasing the amount of information in the instructions and achieving a context-aware interactive experience.

[0087] (2) If the correlation is weak or irrelevant, perform limited fusion or no fusion.

[0088] Limited fusion means that the scene description tags are not used as the core context for fusion. The system only uses the current application state (such as "the user is watching a movie") and / or the user's historical behavior data as auxiliary information to fuse with the initial interaction request.

[0089] If not integrated, the initial interaction request is directly passed (the initial interaction request is directly used as an enhanced query suggestion) or only simple enhancement is performed (personalized modification is only combined with user profile (such as user preference for concise answers).

[0090] For example, if the initial request is "What will the weather be like tomorrow?", the merged message might be: "The user is watching the evening news and is checking tomorrow's weather.", or simply use "Check tomorrow's weather" as the prompt.

[0091] Through the above process, this invention achieves intelligent and conditional context fusion. Video context is deeply injected only when the instruction is highly relevant to the video content, ensuring that the generated prompts are accurate and noise-free for the AI ​​model. When irrelevant, it avoids introducing irrelevant video information, thus guaranteeing processing efficiency and accuracy.

[0092] Specifically, when the intent understanding and context fusion module 103 determines that the initial interaction request is strongly related to the currently playing video content, it executes a context fusion operation under a strong correlation strategy, filtering, associating, and structurally combining information from multiple dimensions. The fused information sources include:

[0093] 1. Core integrated information source (from the real-time analysis module):

[0094] Initial interaction request: The text converted from the user's voice command, such as "Who is he?";

[0095] Strongly relevant scene description tags: Extract tags that are semantically directly related to the initial interaction request from the video keyframe analysis results. For "Who is he?", extract "Face recognition result: Actor A, confidence 85%"; for "What model is this car?", extract "Object recognition result: SUV, brand B feature".

[0096] 2. Supporting integrated information sources (for enhancement and personalization):

[0097] Anonymized user history behavior data: such as the user's past preferences for questions like "actor information" and "car model" (e.g., whether they prefer a brief introduction or detailed parameters);

[0098] Current application status: such as the type of program being played ("documentary"), playback progress ("minute 25"), to supplement the context of the merged prompt words.

[0099] The fusion method follows a structured generation based on a template of "contextual statement + core instructions":

[0100] a. Contextual Statement Section: Based on strongly relevant scene description tags and the current application state, generate an objective statement describing the current screen or context.

[0101] Example template: "[Object / Person: Actor A] has been identified in the current scene, and the scene category is [Scene Category: Indoor Interview]".

[0102] b. Core instruction section: The initial interaction request is either directly embedded or embedded after pronoun resolution.

[0103] Example template: "Based on the above screen, the user asks: [Initial interaction request: Who is he / she?]"

[0104] The final example of enhanced query suggestions is: "The person identified in the current image is actor A, and the scene is an indoor interview. Based on the above image, the user asks: Who is he?"

[0105] When the context is determined to be weakly relevant or irrelevant, the context fusion step primarily or only relies on auxiliary information sources for limited enhancement. When weakly relevant, only the current application state is fused, such as "The user is watching the news. Query: What's the weather like tomorrow?"; or only historical user behavior data is fused, refining the "play music" request to "play a playlist of light music that the user frequently listens to" based on user preferences. When irrelevant, video context fusion is not performed, and the initial request is directly passed.

[0106] The task scheduling module 104 receives enhanced query prompts and associated video stream data segments (such as video clips a few seconds before and after the trigger command, or key video frames determined to be strongly related to the initial interaction request), and sends them to the task processing module. This task processing module employs a generative artificial intelligence model, which can be a lightweight micro-model deployed locally or a cloud-based AI collaborative module.

[0107] The local processing module can also be a lightweight rule engine, which is good at handling operations that are simple to calculate, have high real-time requirements, or involve local privacy, such as simple commands like "volume up" or "switch to the previous channel".

[0108] The cloud-based AI collaboration module is deployed on a remote server, possessing more powerful multimodal data processing capabilities. It excels at handling complex tasks requiring deep understanding, reasoning, creative generation, or massive knowledge retrieval, and provides rich media results. For example, it can handle complex tasks that cannot be processed by local models or rule engines, such as "explaining this plot foreshadowing" or "generating a poem based on this image."

[0109] Finally, the task scheduling module 104 receives the processing result from the task processing module and transmits it to the corresponding interactive control module 105. The interactive control module 105 is used for presenting interactive information and executing control commands. If the returned processing result points to information interaction, it converts the processing result into interactive information and interacts with the user through visualization or audiovisual forms; if the returned processing result points to device control, it generates device control commands based on the processing result and controls the device to perform the corresponding operation.

[0110] Furthermore, the interactive control module includes a multimodal feedback rendering module for interactive presentation and a device control execution module for executing control commands.

[0111] The multimodal feedback rendering module converts interactive information into voice broadcasts or generates graphical information layers (such as character tags and knowledge cards), which are then overlaid on the currently playing video in the form of picture-in-picture, floating windows, sidebars, etc.

[0112] The device control execution module controls the device to perform operations such as channel switching, on-demand playback, and searching based on user commands parsed by AI. Furthermore, by using the audio / video playback device (set-top box) as a smart home hub, it can also control associated smart home devices via IoT protocols.

[0113] In a further embodiment, the task scheduling module 104 is configured to dynamically decide whether to send the task (enhanced query suggestions and related video stream data segments) to the local processing module or the cloud AI collaboration module based on at least one of the factors of the complexity of the enhanced query suggestions, the real-time requirements of the task, and the privacy level of the task.

[0114] The local processing module tends to handle tasks with lower complexity, higher real-time requirements, or privacy concerns. During execution, the complexity is quantified and compared to a preset first threshold to determine whether a task is complex or simple. Based on the task's nature or type, real-time requirements and privacy levels are categorized, and tasks with real-time requirements exceeding a second threshold or privacy levels exceeding a preset level are routed to the local processing module.

[0115] Preferably, such as Figure 2 As shown, the routing decision of the task scheduling module is based on a multi-dimensional scoring function. The input of this scoring function includes at least a content relevance strength factor, a computational complexity factor, a real-time requirement factor, and a privacy sensitivity factor. The output is a comprehensive score, expressed as follows:

[0116]

[0117] (Content Relevance Strength Factor): Represents the relevance of the enhanced query suggestion to the currently playing video content. Its value depends on whether the suggestion explicitly references the scene description tag generated by the real-time analysis module, and the specificity of the reference. If the suggestion does not reference any scene description tag (such as "What's the weather like tomorrow?"), then C is assigned a low value (e.g., 0.1).

[0118] For example, if the prompt uses a general scene label ("A movie is currently playing. Who is the main character?"), then C is assigned a median value (e.g., 0.5); if the prompt uses a specific, unique target label (e.g., "Who is the lady in the red dress in the current scene?"), then C is assigned a high value (e.g., 0.9).

[0119] (Computational complexity factor): Characterizes the estimated computational resources required to process the task, based on the number of model parameters and inference time required for the task.

[0120] Specifically, The quantification of values ​​is based on the semantic complexity of the initial interaction request, the processing time of similar historical tasks, and the model capabilities of the task processing module (local / cloud). For example, simple control commands ("pause", "volume +") or queries that can be solved based on rules are assigned low values; complex tasks that require deep semantic understanding, reasoning, generation, or multimodal fusion ("explain this plot", "write a poem for this scene") are assigned high values.

[0121] (Real-time requirement factor): Characterizes the sensitivity of a task to response latency. It is assigned a value according to the nature or type of the task. For immediate control commands such as "pause" and "mute", a high value is set, and for non-immediate tasks, a low value is set.

[0122] The quantification of the value is estimated based on the semantic complexity of the initial interaction request, the processing time of similar historical tasks, and the model capabilities of the task processing module (local / cloud).

[0123] For example, instructions that directly control the device status ("change channel", "pause") are extremely sensitive to latency and are assigned the highest value; information query tasks ("Who is this?") can tolerate a certain delay and are assigned a medium value; non-real-time content generation tasks ("summarize the plot of this episode") are least sensitive to latency and are assigned the lowest value.

[0124] (Privacy Sensitivity Factor): Characterizes the privacy level of user data or local environment involved in the task. A high value is set for a high privacy level, and a low value is set for a low privacy level.

[0125] Specifically, tasks that do not involve any user personal data and are only related to publicly broadcast content (such as querying actor information) should be assigned the lowest value; tasks that involve device local settings, files, or user anonymization preferences should be assigned a medium value; if the voice command or context explicitly or implicitly involves personally identifiable information, sensitive locations, etc., even after anonymization, a high value should be assigned to prioritize local processing.

[0126] Overall score It can be a weighted summation function, expressed as: ,in , , , These are the weighting coefficients for each factor, which can be adjusted according to the product strategy.

[0127] Set a threshold, when the overall score If the threshold is exceeded, the data is routed to the cloud; otherwise, it is processed locally.

[0128] In a further implementation: the task scheduling module also includes a multimodal large model interface module and a personalized model adaptation module.

[0129] The multimodal large model interface module is used to call the cloud-based AI collaboration module (generative artificial intelligence model), utilize it to perform deep semantic understanding and content generation tasks based on enhanced query suggestions and related video stream data fragments, and receive the returned processing results;

[0130] Personalized model adaptation module: Based on anonymized user historical behavior data, it is used to fine-tune the output of the cloud AI collaboration module to generate feedback content that matches the user's personalized preferences.

[0131] In another implementation, the personalized model adaptation module can be deployed on a cloud server together with the cloud AI collaboration module, and the multimodal large model interface module can directly receive the results after personalized processing by the cloud personalized model adaptation module.

[0132] Personalized fine-tuning includes at least one of the following processing methods:

[0133] (1) Style and expression adaptation: Adjust the language style, level of detail and expression structure of the generated content according to the preferences implied in the user's historical interaction data.

[0134] For example, for users who prefer concise information, the system automatically summarizes and refines lengthy explanations generated by large cloud models; for users who prefer a relaxed, conversational style, the system transforms formal written responses into more friendly, conversational expressions.

[0135] (2) Feedback format adaptation: Based on the user's acceptance of different feedback formats such as voice broadcast and graphic cards in the user's historical interactions and operating habits, structured tags or suggestions are embedded in the generated text content to guide the subsequent interactive control modules (such as the multimodal feedback rendering module) to present in the form of user preferences.

[0136] In one specific implementation, the personalized model adaptation module maintains a dynamically updated user preference profile. This profile is generated by a machine learning model based on fully anonymized interaction logs (such as command type, frequently viewed content categories, and skip / favorite / repeated questioning behavior in response to previous feedback), and includes a series of vectors and weights representing user preferences. When the large cloud model returns the initial results, the adaptation module uses this preference profile as input and reconstructs, reorders, or refines the initial results through a lightweight post-processing model or rule set, thereby achieving personalized output fine-tuning. This process can be performed in the cloud and does not involve the transmission or storage of any personally identifiable information.

[0137] Based on the same inventive concept, embodiments of the present invention also provide a context-aware intelligent interaction method, such as... Figure 3 The interactive process shown includes the following steps:

[0138] S101 responds to user interaction needs by collecting user voice commands and streaming data of the currently playing video in real time.

[0139] S102 performs real-time analysis of streaming data to generate structured scene description tags; and performs real-time recognition of voice commands to obtain initial interaction requests.

[0140] S103. Based on the initial interaction request, scene description tags, user historical behavior data and current application status, perform intent parsing and context fusion to generate enhanced query prompts containing scene information;

[0141] S104, according to the preset routing strategy, the enhanced query prompts and related video stream data segments are sent to the task processing module for processing to obtain the processing results;

[0142] S105, receive the processing result, convert the processing result into interactive information, and / or generate device control instructions based on the processing result.

[0143] The interaction requirements are input by the user via voice or remote control.

[0144] The routing strategy in step S104 can be determined based on a multi-dimensional scoring function. For example... Figure 4 As shown, in step S3, based on the different outputs of the scoring function, the task is identified as a complex / generative task or a simple / control task. For the former, the task is routed to the cloud AI model for processing according to steps S4a-S5a, receiving rich media feedback generated by the cloud; for the latter, the task is routed to the local micro-model or rule engine for processing according to steps S4b-S5b, generating local control commands or structured responses. Finally, in step S6, the results are summarized, and the interactive control module presents the multi-module results or executes device control commands.

[0145] The following example uses a smart set-top box. The scenario is: a user is watching a historical drama, and two characters are having a conversation. A context-aware intelligent interaction system will execute the following interaction flow:

[0146] 1. The user issues a voice command: "Did the general who just spoke actually exist in history?"

[0147] 2. The signal acquisition module simultaneously records the user's voice and extracts key frames from the current video and the previous few seconds of audio;

[0148] 3. The real-time analysis module performs STT on the speech to obtain the text request, and performs face recognition on the keyframes to identify one of the characters and generate the label "Character A, played by actor Y, whose identity in the play is General Li";

[0149] 4. The intent understanding and context fusion module determines that the instruction is strongly related to the content and generates enhanced prompts: "In the current historical drama scene, character A (played by actor Y, who plays General Li in the drama). The user asks: Did this General Li actually exist in history?"

[0150] 5. The task scheduling module evaluates that the task involves complex historical knowledge retrieval and reasoning, and has high scores for computational complexity factor (P) and content relevance factor (C), so it decides to route it to the cloud;

[0151] 6. The multimodal large model interface of the cloud-based AI collaboration module receives prompts and related video clips, calls the large model for understanding and retrieval, and generates an introductory text about the historical figure "General Li". The personalized adaptation module simplifies the text according to the user's preferred concise explanation style;

[0152] 7. The interactive control module receives the text returned from the cloud and broadcasts it in voice form via TTS: "According to research, the prototype of General Li in the drama may be the famous Tang Dynasty general Li XX..." At the same time, a semi-transparent information card is generated in the corner of the screen, displaying a key biographical summary;

[0153] Through the above process, users gain a natural and smooth intelligent interactive experience that is deeply integrated with the content they are watching.

[0154] Embodiments of the present invention also provide an electronic device including a processor and a memory, wherein the number of processors may be one or more. The memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory, thereby realizing the context-aware intelligent interaction method of any of the above embodiments of the present invention.

[0155] The electronic device can be an audio-visual hardware product such as a set-top box, smart TV, or projector.

[0156] The memory may primarily comprise a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0157] Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the context-aware intelligent interaction method of any embodiment of the present invention.

[0158] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0160] Embodiments of the present invention also provide a computer program product that, when run on a computer, causes the computer to execute the context-aware intelligent interaction method of any of the above embodiments of the present invention.

[0161] The above embodiments are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the appended claims.

Claims

1. A context-aware intelligent interaction system, characterized in that, include: The signal acquisition module is used to acquire voice commands and streaming data of the currently playing video in real time; the streaming data includes video keyframes. The real-time analysis module is used to perform target recognition processing on the video keyframes, perform speech recognition processing on the voice commands, generate structured scene description tags based on the target recognition results, and generate initial interaction requests based on the speech recognition results; the target recognition processing includes the recognition of text, objects, faces, and scene types in the image. The intent understanding and context fusion module is used to perform intent parsing and context fusion based on the initial interaction request, the scene description tag, user historical behavior data and the current application state, and generate enhanced query prompts containing scene information; The intent parsing includes: determining whether the initial interaction request is related to the content of the currently playing video; the user's historical behavior data includes historical viewing data; The task scheduling module is used to send the enhanced query prompts and related video stream data segments to the task processing module for processing, and to receive the processing results fed back by the task processing module. An interactive control module is used to convert the processing results into interactive information and / or generate device control commands based on the processing results.

2. The context-aware intelligent interaction system according to claim 1, characterized in that, The task processing module includes a local processing module and a cloud-based AI collaboration module; The task scheduling module is also used to: dynamically decide whether to send the enhanced query suggestion and related video stream data segments to the local processing module or the cloud AI collaboration module based on at least one of the factors of the complexity of the enhanced query suggestion, the real-time requirements of the task, and the privacy level of the task.

3. The context-aware intelligent interaction system according to claim 2, characterized in that, The local processing module is a local micro-model execution module or a rule engine, used to execute tasks whose complexity is lower than the first threshold, tasks whose real-time requirements are higher than the second threshold, and / or privacy tasks.

4. The context-aware intelligent interaction system according to claim 2, characterized in that, The task scheduling module also includes: The multimodal large model interface module is used to call the generative artificial intelligence model to perform deep semantic understanding and content generation tasks based on the enhanced query prompts and related video stream data segments; The personalized model adaptation module is used to fine-tune the output of the generative artificial intelligence model based on anonymized user historical behavior data, so as to generate feedback content that conforms to the user's personalized preferences.

5. The context-aware intelligent interaction system according to claim 2, characterized in that, The task scheduling module makes routing decisions based on a multi-dimensional scoring function; The input to the scoring function includes at least a content relevance strength factor, a computational complexity factor, a real-time requirement factor, and a privacy sensitivity factor, and the output is a comprehensive score; wherein, the content relevance strength factor represents the relevance of the enhanced query suggestion to the currently playing video content; If the overall score is higher than a preset threshold, the enhanced query suggestions and related video stream data segments will be routed to the cloud AI collaboration module; otherwise, they will be routed to the local processing module.

6. The context-aware intelligent interaction system according to claim 1, characterized in that, The interactive control module includes: The multimodal feedback rendering module is used to convert the interactive information into voice broadcast or generate a graphical information layer superimposed on the screen of the currently playing video. The device control execution module is used to control the device to perform corresponding operations according to the device control instructions, including: channel switching, on-demand playback, and search.

7. A context-aware intelligent interaction method, characterized in that, An application to a context-aware intelligent interaction system as described in any one of claims 1-6, comprising the following steps: Responding to user interaction needs, it collects user voice commands and streaming data of the currently playing video in real time; The streaming data is analyzed in real time to generate structured scene description tags; The voice command is recognized in real time to obtain the initial interaction request; Based on the initial interaction request, the scene description tags, user historical behavior data, and the current application status, intent parsing and context fusion are performed to generate enhanced query prompts containing scene information; According to the preset routing strategy, the enhanced query prompts and related video stream data segments are sent to the task processing module for processing to obtain the processing results; Receive the processing result, convert the processing result into interactive information, and / or generate device control commands based on the processing result.

8. The context-aware intelligent interaction method according to claim 7, characterized in that, The routing strategy is determined based on a multi-dimensional scoring function. The input of the scoring function includes at least a content relevance strength factor, a computational complexity factor, a real-time requirement factor, and a privacy sensitivity factor, and the output is a comprehensive score. The content association strength factor represents the degree of relevance between the scene description tags explicitly referenced in the enhanced query suggestion and the initial interaction request; If the overall score is higher than a preset threshold, the enhanced query suggestions and related video stream data segments will be routed to the cloud AI collaboration module; otherwise, they will be routed to the local processing module.

9. The context-aware intelligent interaction method according to claim 7, characterized in that, Based on the initial interaction request, the scene description tags, user historical behavior data, and the current application state, intent parsing and context fusion are performed to generate enhanced query suggestions containing scene information, including: Determine the relevance between the user command and the currently playing content; If the correlation is strong, the element most relevant to the initial interaction request is extracted from the scene description tag and inserted or concatenated into the initial interaction request in natural language form to generate the enhanced query suggestion word; If the correlation is weak or irrelevant, the user's historical behavior data and / or the current application status will be used as auxiliary information and fused with the initial interaction request to generate the enhanced query suggestion.

10. An electronic device having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the context-aware intelligent interaction method according to any one of claims 7-9.

Images