Intelligent interaction method and system of photo frame digital human

Abstract

The invention creatively provides an intelligent interaction method of a photo frame digital person, which comprises the following steps of: receiving and analyzing multi-modal input of a user through a voiceprint filtering technology, and generating voice text data with a voiceprint ID (Identity); the sentiment analysis module is configured to perform multi-dimensional sentiment calculation based on sound intonation features and semantic content and generate a sentiment analysis result containing a sentiment state tag; comprising a multi-level task system agent and a knowledge graph engine, is configured with a memory model generated based on historical interaction data, and splits a complex task into an executable subtask sequence through a target analysis algorithm; based on the sentiment analysis result and the executable subtask sequence, generating a natural language response through a context sensing technology, managing priority ranking of the executable subtasks, and generating adaptive response content according to the sentiment analysis result; a dynamic optimization model based on a reinforcement learning framework component is configured, sentiment analysis parameters and a target analysis algorithm are iteratively updated through user feedback data, and user interaction records are stored.

Description

TECHNICAL FIELD

[0001] The present application relates to a digital person for a photo frame, and in particular to an intelligent interaction method and system for a digital person for a photo frame. BACKGROUND

[0002] The existing digital photo frame technology, although it can display multimedia files such as photos and videos, has significant deficiencies in the interaction function with the user, especially in terms of instant communication and emotional accompaniment, which cannot meet the needs of users. The main defects are as follows: lack of instant interaction capability: the current core function of the digital photo frame is static display, mainly used to play pre-recorded photo or video files. This means that users can only passively enjoy the content and cannot have real-time conversations with the characters or content in the photo frame. For example, when users have questions or emotional needs, the digital photo frame cannot immediately respond and provide personalized suggestions or emotional support. This one-way information transmission cannot achieve deep interaction with the user, especially in terms of personalized accompaniment and emotional connection, and is very limited and lacks intelligent dialogue and personalized services: the existing digital photo frame does not have intelligent language processing and dialogue capabilities. Users cannot interact with the characters in the digital photo frame through voice or text commands, resulting in a single use experience of the product, which is limited to displaying fixed content. The limitations of this interaction method make the digital photo frame unable to provide truly personalized services and unable to dynamically adjust content display according to user needs. SUMMARY

[0003] In view of the deficiencies in the prior art, the purpose of the present application is to provide an intelligent interaction method and system for a digital person for a photo frame, which overcomes the above-mentioned deficiencies in the prior art.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0005] The intelligent interaction method for a digital person for a photo frame, characterized in that it comprises

[0006] An input processing step receives and analyzes the user's multi-modal input through voiceprint filtering technology to generate voice text data with a voiceprint ID;

[0007] An emotional analysis step is configured to perform multi-dimensional emotional analysis based on voice tone characteristics and semantic content through an emotional analysis model to generate emotional analysis results containing emotional state labels;

[0008] Agent thinking and processing steps, deploying multi-level task system agents and domain knowledge graph engines, building a dynamic memory model based on historical interaction data, using the BERT model to extract subject-verb-object triples input by users to generate an intent logic tree, calculating the optimal execution path between knowledge graph entities through the Dijkstra algorithm, and configuring a real-time weight controller to dynamically adjust the subtask sequence when user satisfaction falls below the satisfaction threshold. Task decomposition is achieved using an independent GPU acceleration unit;

[0009] The task execution and dialogue management steps obtain the emotion intensity value and executable subtask sequence based on the sentiment analysis results, and control the response process through the red light strategy: in red light mode, non-urgent tasks are suspended and voice endpoint detection is activated. In green light mode, the fast response channel is called to generate natural language feedback and synchronously execute high-priority subtasks. When the emotion intensity is greater than 0.8, the three-channel collaborative response protocol is triggered, and the subtask execution order is dynamically rescheduled based on the context relevance.

[0010] Feedback and learning steps, build a dynamic optimization model based on reinforcement learning, and adopt a dual-loop learning mechanism: the short-term optimization loop collects user correction instructions in real time and updates the emotion model parameters through the Q-learning algorithm; the long-term optimization loop eliminates low-value data based on the time series decay function and uses the KL divergence to detect sudden changes in user behavior patterns. The optimized parameters are reversely injected into the intent logic tree of the Agent module through the bus, and high-value interaction records are solidified through the SSD storage unit. The reinforcement learning coprocessor is bound to accelerate model iteration.

[0011] Preferably, the voiceprint filtering technology includes:

[0012] Speech feature extraction unit, which uses MFCC algorithm to extract voiceprint feature vector;

[0013] The voiceprint analysis unit uses a Gaussian mixture model to calculate the current voiceprint and match it with the pre-stored voiceprint in the memory model, and obtains the matching degree. Based on the matching degree, it obtains the corresponding user information and retrieves the user's historical interaction records.

[0014] Preferably, the voiceprint filtering technology also includes

[0015] A dynamic threshold adjustment unit, configured to obtain the ambient noise decibel, automatically adjust the voiceprint recognition sensitivity, and filter the voiceprint according to the ambient noise decibel;

[0016] A multi-voiceprint concurrent processing unit is used to obtain multiple groups of voice input at the same time, separate the voice according to the voiceprint characteristics to generate several groups of voiceprints, and sort the voiceprint groups by priority. According to the priority sorting, the voiceprint groups with higher priority are processed first;

[0017] The priority sorting unit respectively acquires the emotional analysis result of the voiceprint group, user information and executable subtask sequence, and calculates the urgency degree through a weight algorithm, and sorts the voiceprint group from high to low according to the urgency degree.

[0018] As preferred, the task execution and dialogue management step includes presetting three types of emotion templates of urgency, happiness and neutrality, and configuring a cross-modal response mapping table, when the emotional intensity value of the emotional analysis result is greater than 0.8, a composite response protocol is triggered; real-time collection of user correction instruction frequency and single interaction duration; updating the emotional analysis model parameters through the Q-learning algorithm.

[0019] As preferred, the target analysis algorithm includes extracting subject-predicate-object triples of user input using a BERT model, and generating an intent logic tree in combination with context; identifying entity correlation through a domain knowledge graph, and calculating an optimal execution path using the Dijkstra algorithm.

[0020] When the user feedback satisfaction is less than 60%, the priority weight of the associated subtask is automatically increased.

[0021] As preferred, the task execution and dialogue management step is provided with a traffic light strategy, which detects the start and end points of user input in real time, and judges whether the user input is complete, if the customer continues to input, the red light state is entered, the context tracking and input monitoring are retained, when the user input is completed, or the user issues an interaction request, the green light state is entered, the task execution and response generation are started, and the content is output based on the context.

[0022] As preferred, the feedback and learning step includes encoding interaction records into time series feature vectors using an LSTM network; dividing user types based on a K-means clustering algorithm, fusing current context and user type labels, and matching a preset content library to generate a recommendation list.

[0023] As preferred, the memory model execution uses a Word2Vec algorithm to convert interaction events into 300-dimensional semantic vectors; identifies behavior pattern offset through cosine similarity; and applies an exponential decay function according to the interaction timestamp to eliminate low-weight memory data.

[0024] The smart interaction system of the photo frame digital person includes

[0025] The input processing module receives and analyzes the multi-modal input of the user through voiceprint filtering technology to generate voice text data with voiceprint ID;

[0026] The emotional analysis module is configured to perform multi-dimensional emotional analysis based on voice tone features and semantic content through an emotional analysis model to generate emotional analysis results containing emotional state labels.

[0027] The agent thinking and processing module deploys a multi-level task system agent and a domain knowledge graph engine, constructs a dynamic memory model based on historical interaction data, extracts subject-predicate-object triples from user input using a BERT model to generate an intent logic tree, calculates the optimal execution path between knowledge graph entities using Dijkstra's algorithm, and configures a real-time weight regulator to dynamically adjust the sub-task sequence when user satisfaction is below a satisfaction threshold, and realizes task decomposition by an independent GPU acceleration unit.

[0028] The task execution and dialogue management module obtains an emotional intensity value and an executable sub-task sequence based on the sentiment analysis result, controls the response process through a red light strategy: suspends non-urgent tasks and activates voice endpoint detection in red light mode, generates natural language feedback in green light mode, and synchronously executes high-priority sub-tasks; when the emotional intensity is greater than 0.8, a three-channel collaborative response protocol is triggered, and the sub-task execution order is dynamically rearranged based on context correlation;

[0029] The feedback and learning module constructs a dynamic optimization model based on reinforcement learning, adopts a double-loop learning mechanism: the short-time optimization loop collects user correction instructions in real time, updates the sentiment model parameters through the Q-learning algorithm, and the long-time optimization loop eliminates low-value data based on a time decay function, detects user behavior pattern mutations using KL divergence, and optimizes the parameters. The optimized parameters are injected back into the intent logic tree of the Agent module through the bus, and high-value interaction records are solidified through the SSD storage unit, and the reinforcement learning coprocessor accelerates model iteration

[0030] The beneficial effects of the present application are: users can interact with the digital person in the photo frame through voice or touch, greatly enhancing the usability and interest of the photo frame; the digital person can generate personalized responses based on user input, emotional state, historical preferences, and other factors, providing deeper emotional companionship and interaction; compared to the one-way display of traditional photo frames, the digital person AI photo frame can establish an emotional connection with the user through sentiment analysis technology and provide personalized emotional support, especially in scenarios such as elderly companionship; through voiceprint filtering technology, different users' inputs can be effectively distinguished, allowing the system to focus on specific user instructions even when multiple people are using it, avoiding invalid input interference; personal information, habits, past conversations, and emotional changes can be remembered for a long time. This means that users can refer to conversations from a long time ago when interacting with the digital person, and the system will continue the conversation based on the content at that time, forming a truly long-term interaction and emotional connection. BRIEF DESCRIPTION OF DRAWINGS

[0031] Fig. 1 is the flowchart of the human-computer dialogue of the present application;

[0032] Fig. 2It is a structure schematic diagram of the digital person AI photo frame information processing of the present application.

[0033] Fig. 3 It is a process flow chart of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0035] It should be noted that when a component is referred to as being "fixed" to another component, it can be directly on the other component or there can be intervening components. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or there can be intervening components. When a component is referred to as being "disposed" on another component, it can be directly disposed on the other component or there can be intervening components. The terms "vertical", "horizontal", "left", "right", and similar terms as used herein are for purposes of description only.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this description, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0037] The embodiments of the present application are further described in detail below with reference to the accompanying drawings:

[0038] As shown in the figure, the present application provides an intelligent interaction method of photo frame digital person, which includes Figs. 1-3

[0039] Input processing step, receiving and analyzing the multi-modal input of the user through voiceprint filtering technology to generate voice text data with voiceprint ID; advanced voiceprint filtering technology can accurately receive and analyze the multi-modal input of the user, and the multi-modal input here includes voice, gestures and other forms. In terms of voice input, through voiceprint recognition technology, it can quickly identify the identity of the speaker and generate voice text data with voice ID; when different members in the family communicate with the photo frame digital person, it can accurately distinguish which member it is according to the voiceprint characteristics, and then provide personalized services according to the use habits and preferences of different members.

[0040] ​The sentiment analysis step is configured to perform multi-dimensional sentiment calculation based on voice tone features and semantic content through a sentiment analysis model to generate a sentiment analysis result containing emotional state labels. The sentiment analysis model is configured based on voice tone features and semantic content. In actual operation, it performs multi-dimensional sentiment calculation on the user's input voice content. From the perspective of voice tone, high-pitched and enthusiastic tone may represent excitement or excitement, and low-pitched and slow tone may mean depression or fatigue; from the semantic content level, the large number of positive words indicate that the user's mood is positive, and vice versa negative words reflect negative emotions; after analysis, a sentiment analysis result containing emotional state labels is generated, such as "happy", "sad", "angry", "calm", etc. Through such sentiment analysis, the photo frame digital person can deeply understand the user's current emotional state, provide an important basis for subsequent response strategies, and make the interaction more personalized and thoughtful.

[0041] The agent thinking and processing step deploys a multi-level task system agent and a domain knowledge graph engine, builds a dynamic memory model based on historical interaction data, uses a BERT model to extract the subject-predicate-object triplets of user input to generate an intent logic tree, calculates the optimal execution path between knowledge graph entities through Dijkstra algorithm, and configures a real-time weight regulator to dynamically adjust the sub-task sequence when the user satisfaction is lower than the satisfaction threshold. The task is decomposed by an independent GPU acceleration unit.

[0042] The multi-level task system agent architecture is deployed (supporting ≥7 layers of task nesting), an improved A* algorithm is used to realize target analysis (the time complexity is optimized to O(nlog n)), and a domain knowledge graph engine (containing 1.2 million entity nodes and 4.5 million relationship edges) is used for semantic path planning. The memory model is built based on a time convolution network (TCN), with a storage capacity of ≥1TB, and can retrieve the user's last 200 interaction records. When analyzing user instructions, the following precise operations are performed: extract the core elements of the instruction (confidence threshold ≥95%) through the BERT model, such as "weekend→time constraint" "family→people characteristics" "scenic spots→geographical range" "delicacies→interest labels", call the associated reasoning module of the knowledge graph, calculate the semantic distance between entities (formula: distance = 1-(number of common relationship edges) / √(total relationship edges)), generate a sub-task sequence based on the dynamic weight formula: sub-task weight = 0.6×semantic relevance + 0.3×historical execution success rate + 0.1×real-time resource occupancy rate; FPGA hardware accelerator realizes task decomposition (processing speed ≥3000 tasks / second), such as decomposing the scenic spot recommendation task into:

[0043] Subtask A: Search for 4A-level scenic spots within a 50km radius (response time < 0.3 seconds); Subtask B: Screen family-friendly facilities (baby stroller rental / wheelchair-accessible paths); Subtask C: Associate Michelin-recommended restaurants (spatial distance < 800m from scenic spot entrance)

[0044] Task execution and dialogue management steps, based on the emotion analysis results, obtain the emotional intensity value and the executable subtask sequence, control the response process through the red light strategy: pause non-urgent tasks and activate voice end-point detection in red light mode, call the fast response channel to generate natural language feedback in green light mode, and execute high-priority subtasks synchronously; trigger a three-channel collaborative response protocol when the emotional intensity > 0.8, dynamically rearrange the subtask execution order in combination with the context association degree; include pre-set emotional templates of three types of emotions: emergency, joy, and neutral, and configure a cross-modal response mapping table, trigger the composite response protocol when the emotional intensity value of the emotion analysis result > 0.8; collect user correction instruction frequency and single interaction duration in real time; update the emotion analysis model parameters through Q-learning algorithm (learning rate η = 0.05 ± 0.01).

[0045] Build a sentiment-driven dual-channel response engine, hardware configuration includes: real-time emotion calculation unit (BERT-base model, emotion recognition accuracy 89.2%); low-latency NLG engine (end-to-end generation delay < 0.25 seconds); priority arbiter (supports dynamic adjustment of 16-level task queue)

[0046] The execution process implements a triple control strategy:

[0047] Emotion-based response:

[0048] Emergency state (emotional intensity > 0.7): trigger fast response mode (response template length ≤ 15 words, insert 3 soothing words); neutral state (0.3 ≤ intensity ≤ 0.7): standard response template (length 20-40 words); happy state (< 0.3): enable extended description mode (additional interesting trivia)

[0049] Emergency task insertion algorithm: when the user's breathing frequency is detected to be > 22 times / minute, the priority of the relevant subtasks is increased by Δ = 0.45

[0050] Resource preloading mechanism: pre-load associated data for subtasks with a completion rate > 80% (memory hit rate improved by 37%)

[0051] Feedback and learning steps, build a closed-loop optimization architecture based on deep reinforcement learning (DRL), including:

[0052] Dual-channel feedback collection:

[0053] Explicit feedback: User actively rate (5-level satisfaction scale) Direct reinforcement learning agent

[0054] Implicit feedback: Through eye tracking (fixation duration > 3 seconds is determined as a point of interest) and behavior modeling (click delay < 0.5 seconds is determined as approval)

[0055] Voiceprint filtering technology includes

[0056] The voiceprint feature extraction unit extracts the voiceprint feature vector using the MFCC algorithm. The MFCC algorithm is based on the perceptual characteristics of the human auditory system, which converts the speech signal to the Mel frequency scale, through a series of complex calculations, including pre-emphasis, framing, windowing, fast Fourier transform, Mel filter bank filtering and discrete cosine transform, etc. Finally, the feature vector that can effectively represent the unique information of the voiceprint is extracted. These feature vectors contain the speaker's timbre, pitch, pronunciation habits and other key information, just like everyone's unique "voiceprint".

[0057] The voiceprint analysis unit matches the current voiceprint with the pre-stored voiceprint in the memory model through Gaussian mixture model, and obtains the matching degree, and obtains the corresponding user information and historical interaction records; With the help of Gaussian mixture model (GMM), voiceprint matching and user information acquisition are realized. In actual operation, this unit compares and calculates the voiceprint feature vector extracted at present with the pre-stored voiceprint in the memory model, and obtains the matching degree through the calculation of probability density and other ways. When the matching degree reaches a certain threshold, the speaker can be accurately identified, and the corresponding user information and historical interaction records can be retrieved. For example, when family member A communicates with the digital person of the photo frame, the voiceprint analysis unit quickly completes the matching and immediately retrieves A's past preferences, commonly used instructions and other interaction records, so that the digital person can provide services that meet A's needs based on these information, greatly improving the individualization and accuracy of interaction.

[0058] The voiceprint filtering technology also includes

[0059] The dynamic threshold adjustment unit is used to obtain the environmental noise decibel, automatically retrieve the voiceprint recognition sensitivity, and filter the voiceprint according to the environmental noise decibel; It can real-time perceive environmental changes, improve the accuracy and adaptability of voiceprint recognition. It can automatically retrieve the voiceprint recognition sensitivity by obtaining the environmental noise decibel, and filter the voiceprint according to the environmental noise condition. In a quiet indoor environment, the system can reduce the voiceprint recognition threshold, improve the sensitivity, and capture more subtle voiceprint features; In a noisy living room party scene, the dynamic threshold adjustment unit will automatically increase the recognition threshold to reduce the interference of environmental noise and avoid misidentification. For example, when the TV volume at home is loud and the surrounding voices are noisy, this unit can accurately filter out irrelevant noise;

[0060] Multi-acoustic-print concurrent processing unit, after obtaining multiple groups of simultaneous voice inputs, separates the voices based on acoustic-print characteristics to generate several acoustic-print groups, prioritizes the acoustic-print groups, and prioritizes the acoustic-print groups with higher priority based on the priority ranking. In the scenario of multiple people interacting with a digital person in a photo frame at the same time, the multi-acoustic-print concurrent processing unit plays a key role. After obtaining multiple groups of simultaneous voice inputs, it separates the voices based on acoustic-print characteristics to generate several acoustic-print groups. This separation process is like "sorting" different sounds mixed together, with each acoustic-print group corresponding to the voice information of a speaker. Subsequently, the unit prioritizes the acoustic-print groups to ensure that important or urgent voice requests are prioritized for processing. For example, when discussing travel plans at home, parents and children may ask the digital person different questions at the same time. The multi-acoustic-print concurrent processing unit can quickly separate the voices based on acoustic-print characteristics and determine that the parents' travel arrangement question may be more urgent, prioritizing the acoustic-print group to efficiently address user needs and avoid information processing chaos.

[0061] The priority dynamic control module deploys a multi-dimensional decision engine based on an improved TOPSIS algorithm and implements a three-level weight quantization strategy.

[0062] Emotional weight factor α: Real-time extraction of acoustic-print characteristics using a Bi-LSTM model (sampling rate 48kHz, extraction of 128-dimensional MFCC coefficients); Emotional intensity calculation formula: α = 0.6 × acoustic-print stress index + 0.4 × semantic urgency identifier (such as containing "urgent" "immediately" and other words); When α > 0.7, trigger red alert mode, priority automatically increased Δpriority = 0.55 × (α - 0.7); User value factor β: Construct user identification matrix: β = 0.3 × monthly interaction frequency (standardized value) + 0.5 × historical payment amount (log transformation) + 0.2 × social relationship weight (direct relative coefficient 1.2); Establish a privileged channel: Special users (β > 0.8) request to jump the queue with a 4.7 times probability increase; Task complexity factor γ: Calculate the task chain length penalty term based on Markov decision process: γ = 1 / (1 + 0.1 × estimated subtask number); Comprehensive priority formula: Priority = 0.55α + 0.35β + 0.1γ + ε (dynamic compensation term); Where ε value is dynamically adjusted according to real-time system load (range ± 0.15).

[0063] For example, when user A (β = 0.92 VIP user) excitedly asks: "Why hasn't my medical insurance reimbursement been received yet?!": Voiceprint analysis shows α = 0.83 (speech rate 5.2 words / second + fundamental frequency fluctuation > 30Hz); task analysis generates γ = 0.78 (needs to call 5 government API interfaces); calculated Priority = 0.55 × 0.83 + 0.35 × 0.92 + 0.1 × 0.78 = 0.851, surpassing the regular consultation request with a priority of 0.72 in the current queue, achieving zero-wait response

[0064] In the dynamic response substep, an emotional response engine based on RoBERTa-wwm is deployed. Eight primary emotion templates (such as joy, sadness, anger, and confusion) are pre-configured, each with over 50 scenario-specific sub-templates. A confidence threshold of >95% is set to ensure template accuracy. When a user is detected as sad (emotion intensity >0.68) and mentions "work pressure," the T_Comfort_Workplace Basic template is invoked. This template is then fed with historical user interaction data (extracting interest tags like "fitness" and "classical music" from the last five conversations) and integrated with environmental sensor data (for example, automatically recommending commuting routes for the 6:30 PM end of get off work period). Response generation uses dual-channel fusion technology: the template placeholder is aligned with the 32-dimensional user portrait vector extracted in real time (similarity weight 0.7), and the voice parameters are adjusted through the NVIDIA Riva engine (the speaking rate is reduced to 2.8 words / second, and the fundamental frequency fluctuation range is compressed to ±5Hz). The output is a context-aware response such as "You mentioned last week that you like night runs by the river. The temperature is suitable tonight at 7:00 PM. Do you need to plan a running route?" The generation delay is strictly controlled within 0.22 seconds.

[0065] In the online learning sub-step, a closed-loop optimization system based on the A3C reinforcement learning framework was constructed, collecting over 1,200 pieces of feedback data daily (including explicit 5-level ratings and implicit conversation retention rates). When the misjudgment rate for a particular emotion category exceeded 18% for three consecutive hours, dynamic parameter calibration was initiated: the BERT-large teacher model (F1 = 0.92) was compressed into the DistilBERT student model through knowledge distillation (inference speed increased by 3.1 times), and the weight of voiceprint features was adjusted from 0.65 to 0.72, and the weight of semantic features from 0.35 to 0.28. A grayscale release mechanism was deployed, importing 5% of user traffic to the new model for A / B testing. Full release was required to achieve a misjudgment rate reduction of ≥ 2.5% and an increase in satisfaction of ≥ 1.8% for 48 consecutive hours. Experimental data showed that the accuracy of emotion recognition in workplace stress scenarios increased from 83.7% to 91.2%.

[0066] The target analysis algorithm adopts the ALBERT-xxlarge model to construct a 128-dimensional intent space. When the user expresses "stimulating", with a confidence of 0.93, the task chain is generated in combination with the movie domain knowledge graph (containing 5 million entity nodes): ① Call the Douban API to get the recent release list (response <0.4 seconds) ② Filter the thriller / suspense type (match the IMDb label system) ③ Sort by user historical rating preferences (weight = 0.6 x type matching degree + 0.3 x actor preference + 0.1 x length adaptation). The dependency relationship detection adopts the Kahn topological sorting algorithm (processing speed 1800 tasks / minute), and when the user feedback "don't show violent scenes", the current thread is terminated immediately, and a content filtering sub-task is inserted (age rating ≥ PG-13 constraint is enabled), and the priority coefficient is automatically down-regulated from 0.75 to 0.32.

[0067] A red-green light strategy is set in the task execution and dialogue management step. The red-green light strategy detects the start and end points of the user input in real time and judges whether the user input is complete. If the user continues to input, the system enters the red light state, retains the context tracking and input monitoring, and when the user input is complete or the user initiates an interaction request, the system enters the green light state, starts the task execution and generates a response based on the context output content. During the user input process, the strategy detects the start and end points of the input in real time and judges whether the input is complete. When the user continues to speak, the system enters the red light state, at which time the context tracking and input monitoring are retained to ensure that the user's subsequent expressions are not missed and to avoid incorrect task execution or response generation when the user has not finished speaking. For example, when the user is describing a travel plan and pauses to think of words, the system will not act rashly. When the user input is complete or the user initiates an interaction request (such as saying "finished"), the system immediately enters the green light state, quickly starts the task execution process, and generates an accurate and coherent response based on the previously retained context information, making the entire interaction process natural and smooth and reducing communication problems caused by incorrect input state judgments.

[0068] The red-green light strategy integrates a WebRTC VAD endpoint detection module (250ms silence threshold) and enables background pre-analysis in the red light state: 60% semantic understanding is completed by caching the last 8 seconds of voice stream (ring buffer capacity 1.2MB), while maintaining context tracking (entity coreference resolution accuracy 89%). When a terminator such as "finished" is detected, the system switches to the green light state within 0.15 seconds and activates 32 processing threads in parallel to perform the complete NLU process (including 14 layers of attention mechanism). The false touch prevention mechanism adopts a three-level verification: BLEU score >0.65 to ensure sentence completeness, emotional fluctuation standard deviation <0.12 to maintain consistency, and entity linking accuracy >87% to ensure coherence. Pressure testing shows that the response accuracy is improved by 41% under the condition of an average user speaking speed of 4.2 words per second.

[0069] The feedback and learning step encodes the interaction records into time series feature vectors using an LSTM network; based on the K-means clustering algorithm, the user types are divided, the current context and user type labels are fused, and the preset content library is matched to generate a recommendation list; a 72-dimensional time series feature engine (sliding window = 24h, step = 1h) is designed to extract behavior fingerprints such as the proportion of morning period video consultation and weekend travel topic shift frequency. Apply LightGBM classifier (AUC = 0.94) to divide users into 8 dynamic groups, and perform incremental clustering (KL divergence threshold 0.28) every week. For example, for the C5 tourism enthusiast group, pre-load POI data within a radius of 200km (including 45,000 scenic spot information), and when the "travel plan" intention is detected, real-time injection of weather warning (rain probability > 35% triggers alternative solutions) and traffic control information (updated every 15 minutes through Gaode API). Implementation data shows that the target user recommendation click rate has increased from 22.7% to 38.4%.

[0070] The memory model uses the Word2Vec algorithm to convert interaction events into 300-dimensional semantic vectors; recognize behavior pattern offset through cosine similarity; apply an exponential decay function according to the interaction timestamp to eliminate low-weight memory data; construct a 256-dimensional behavior encoding space (cosine similarity threshold 0.85), for example, map the "suspense movie" query to the [0.68, -0.12,..., 0.41] feature vector. The memory weight algorithm uses an adaptive forgetting curve W(t) = e^(-λt), where λ = 0.05 x recent 7-day interaction frequency + 0.02 x paid record level. When the similarity between user A (encoding vector A) and user B (encoding vector B) reaches 0.89, cross-profile knowledge transfer is enabled: share Top20 recommended items in the candidate pool (processed by differential privacy, ε = 1.2), and the transfer learning efficiency is improved by 37%. The storage weight of important events (such as manually marked preferences) is permanently increased by 3 times, ensuring a long-term memory retention rate of > 92%.

[0071] The smart interaction system of the photo frame digital person includes

[0072] The input processing model receives and analyzes the user's multi-modal input through voiceprint filtering technology to generate voice text data with voiceprint ID;

[0073] The sentiment analysis model is configured to perform multi-dimensional sentiment calculation based on voice tone features and semantic content through a sentiment analysis model to generate sentiment analysis results containing emotion state labels;

[0074] The agent thinking and processing model deploys a multi-level task system agent and a domain knowledge graph engine, constructs a dynamic memory model based on historical interaction data, uses a BERT model to extract the subject-predicate-object triple of user input to generate an intent logic tree, calculates the optimal execution path between knowledge graph entities through Dijkstra algorithm, and configures a real-time weight regulator to dynamically adjust the sub-task sequence when user satisfaction is lower than the satisfaction threshold. The task decomposition is realized by an independent GPU acceleration unit.

[0075] The task execution and dialogue management model obtains an emotional intensity value and an executable sub-task sequence based on the sentiment analysis result, controls the response process through a traffic light state machine: in the red light mode, non-urgent tasks are suspended and voice endpoint detection is activated; in the green light mode, a natural language feedback is generated by calling a quick response channel, and high-priority sub-tasks are executed synchronously; when the emotional intensity is greater than 0.8, a three-channel collaborative response protocol is triggered, and the sub-task execution order is dynamically rearranged according to the context correlation degree.

[0076] The feedback and learning model constructs a dynamic optimization model based on reinforcement learning, adopts a double-loop learning mechanism: the short-time optimization loop collects user correction instructions in real time, updates the sentiment model parameters through Q-learning algorithm; the long-time optimization loop eliminates low-value data based on a time decay function, detects user behavior pattern mutations using KL divergence, and optimizes the parameters. The optimized parameters are injected back into the intent logic tree of the agent module through the bus, and the high-value interaction records are solidified through the SSD storage unit, and the reinforcement learning coprocessor is bound to accelerate model iteration.

[0077] The above is only a preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the scope of the present application should be considered within the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application should also be considered within the protection scope of the present application.

Claims

1. The intelligent interaction method of the photo frame digital human is characterized by: include Input processing step, receiving and parsing the user's multimodal input through voiceprint filtering technology to generate voice text data with voiceprint ID; The sentiment analysis step is configured to perform multi-dimensional sentiment calculation based on voice intonation features and semantic content through a sentiment analysis model to generate sentiment analysis results containing emotional state labels; Agent thinking and processing steps, deploying multi-level task system agents and domain knowledge graph engines, building a dynamic memory model based on historical interaction data, using the BERT model to extract subject-verb-object triples input by users to generate an intent logic tree, calculating the optimal execution path between knowledge graph entities through the Dijkstra algorithm, and configuring a real-time weight controller to dynamically adjust the subtask sequence when user satisfaction falls below the satisfaction threshold. Task decomposition is achieved using an independent GPU acceleration unit; The task execution and dialogue management steps obtain the emotion intensity value and executable subtask sequence based on the sentiment analysis results, and control the response process through the red light strategy: in red light mode, non-urgent tasks are suspended and voice endpoint detection is activated. In green light mode, the fast response channel is called to generate natural language feedback and simultaneously execute high-priority subtasks. When the emotion intensity is greater than 0.8, the three-channel collaborative response protocol is triggered, and the subtask execution order is dynamically rearranged based on the context relevance; Feedback and learning steps, build a dynamic optimization model based on reinforcement learning, and adopt a dual-loop learning mechanism: the short-term optimization loop collects user correction instructions in real time and updates the emotion model parameters through the Q-learning algorithm; the long-term optimization loop eliminates low-value data based on the time series decay function and uses the KL divergence to detect sudden changes in user behavior patterns. The optimized parameters are reversely injected into the intent logic tree of the Agent module through the bus, and high-value interaction records are solidified through the SSD storage unit. The reinforcement learning coprocessor is bound to accelerate model iteration.

2. The intelligent interaction method of the photo frame digital human according to claim 1, characterized in that: The voiceprint filtering technology includes Speech feature extraction unit, which uses MFCC algorithm to extract voiceprint feature vector; The voiceprint analysis unit uses a Gaussian mixture model to calculate the current voiceprint and match it with the pre-stored voiceprint in the memory model, and obtains the matching degree. Based on the matching degree, it obtains the corresponding user information and retrieves the user's historical interaction records.

3. The intelligent interaction method of the photo frame digital human according to claim 2, characterized in that: The voiceprint filtering technology also includes A dynamic threshold adjustment unit, configured to obtain the ambient noise decibel, automatically adjust the voiceprint recognition sensitivity, and filter the voiceprint according to the ambient noise decibel; A multi-voiceprint concurrent processing unit is used to obtain multiple groups of voice input at the same time, separate the voice according to the voiceprint characteristics to generate several groups of voiceprints, and sort the voiceprint groups by priority. According to the priority sorting, the voiceprint groups with higher priority are processed first; The priority sorting unit obtains the sentiment analysis results, user information and executable subtask sequence of the voiceprint group respectively, calculates the urgency through a weight algorithm, and sorts the voiceprint group from high to low according to the urgency.

4. The intelligent interaction method according to claim 1, characterized in that: The task execution and dialogue management steps include pre-setting three emotion template libraries: emergency, pleasure, and neutral, and configuring a cross-modal response mapping table. When the emotion intensity value of the emotion analysis result is greater than 0.8, a composite response protocol is triggered; the frequency of user correction instructions and the duration of a single interaction are collected in real time; Update sentiment analysis model parameters through Q-learning algorithm.

5. The intelligent interaction method according to claim 1, characterized in that: The target parsing algorithm includes using the BERT model to extract the subject-verb-object triples input by the user and generating an intent logic tree based on the context; Identify entity associations through domain knowledge graphs and use Dijkstra's algorithm to calculate the optimal execution path; When user feedback satisfaction is less than 60%, the priority weight of the associated subtasks is automatically increased.

6. The intelligent interaction method according to claim 2, characterized in that: A traffic light strategy is set in the task execution and dialogue management steps. The traffic light strategy detects the starting and ending points of user input in real time and determines whether the user input is complete. If the customer continues to input, it enters the red light state, retains context tracking and input monitoring. When the user input ends or the user issues an interaction request, it enters the green light state, starts task execution and generates a response, and outputs content based on the context.

7. The intelligent interaction method according to claim 2, characterized in that: The feedback and learning steps include encoding interaction records into time series feature vectors using an LSTM network; dividing user types based on a K-means clustering algorithm, fusing current context and user type labels, and matching a preset content library to generate a recommendation list.

8. The intelligent interaction method according to claim 2, characterized in that: The memory model uses the Word2Vec algorithm to convert interaction events into 300-dimensional semantic vectors; identifies behavior pattern offsets through cosine similarity; and applies an exponential decay function based on interaction timestamps to eliminate low-weight memory data.

9. The intelligent interactive system of the photo frame digital human is characterized by: include The input processing module receives and analyzes the user's multimodal input through voiceprint filtering technology, and generates voice text data with voiceprint ID; The sentiment analysis module is configured to perform multi-dimensional sentiment calculation based on voice intonation features and semantic content through a sentiment analysis model to generate sentiment analysis results containing emotional state labels; The Agent thinking and processing module deploys a multi-level task system agent and a domain knowledge graph engine. It builds a dynamic memory model based on historical interaction data, uses the BERT model to extract the subject-verb-object triples of user input to generate an intent logic tree, calculates the optimal execution path between knowledge graph entities using the Dijkstra algorithm, and configures a real-time weight controller to dynamically adjust the subtask sequence when user satisfaction falls below the satisfaction threshold. Task decomposition is achieved using an independent GPU acceleration unit. The task execution and dialogue management module obtains emotion intensity values ​​and executable subtask sequences based on sentiment analysis results, and controls the response process through a red light strategy: in red light mode, non-urgent tasks are suspended and voice endpoint detection is activated; in green light mode, a fast response channel is called to generate natural language feedback and simultaneously execute high-priority subtasks; When the emotion intensity is greater than 0.8, the three-channel collaborative response protocol is triggered, and the subtask execution order is dynamically rearranged based on the context relevance; The feedback and learning module builds a dynamic optimization model based on reinforcement learning and adopts a dual-loop learning mechanism: the short-term optimization loop collects user correction instructions in real time and updates the emotion model parameters through the Q-learning algorithm; the long-term optimization loop eliminates low-value data based on the time series decay function and uses the KL divergence to detect sudden changes in user behavior patterns. The optimized parameters are reversely injected into the intent logic tree of the Agent module through the bus, and high-value interaction records are solidified through the SSD storage unit. The reinforcement learning coprocessor is bound to accelerate model iteration.

Images