An interactive AI intelligent hardware feedback processing optimization method and system

Abstract

The application discloses an interactive AI intelligent hardware feedback processing optimization method and system, and relates to the technical field of artificial intelligence and intelligent hardware interaction. The application constructs a front-end real-time analysis-middle dynamic arrangement-back-end real-time output architecture, adopts streaming transmission and parallel analysis technology to extract semantic and emotional features of text and audio, and generates a structured semantic-emotional fusion label through weighted fusion. Based on the label, multi-dimensional special effect instructions of vision, hearing and hardware action are dynamically generated, and the instruction complexity is adaptively adjusted according to the hardware performance. Through a synchronous control mechanism, the special effect is aligned with the AI dialogue content. The application breaks through the limitation of traditional pre-defined rule matching, realizes adaptive feedback in any dialogue scene, greatly improves the interactive immersion, and is compatible with multiple hardware forms such as intelligent sound boxes, children learning machines and pet companion robots, and is suitable for multiple fields such as education, companionship, entertainment and home.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and smart hardware interaction technology, and in particular to an interactive AI smart hardware feedback processing optimization method and system. Background Technology

[0002] With the rapid iteration and popularization of artificial intelligence technology, smart hardware with voice interaction capabilities has been widely used in various scenarios such as home, education, and entertainment, including smart speakers, children's learning machines, pet companion robots, and smart home control systems. This type of hardware, through interaction with cloud-based AI servers, can respond to users' voice commands and provide corresponding feedback, becoming an important carrier of human-computer interaction. Early smart hardware AI feedback was presented only in the form of simple text or audio. As users' demands for interactive experiences have continued to increase, the industry has gradually introduced auxiliary feedback forms such as visual effects and simple hardware actions, attempting to enhance the fun and immersion of the interaction.

[0003] Currently, mainstream AI feedback processing technologies for smart hardware generally adopt a predefined rule matching-fixed feedback implementation model. This model requires engineers to build a keyword-effect mapping rule library and template library in advance. During actual interaction, the system identifies predefined keywords in the user's dialogue and calls the corresponding preset effects to provide feedback. For example, the pre-stored "happy" keyword corresponds to displaying a smiley face image, the "question" keyword corresponds to playing a specific prompt sound effect, and the "dance" command corresponds to executing a preset hardware action sequence.

[0004] However, the aforementioned existing technologies have significant defects and shortcomings in practical applications:

[0005] First, its response capabilities are limited, failing to cover diverse interaction scenarios. Due to the limited capacity of the rule base and template library, the system can only handle semantic scenarios within a predefined scope. For undefined situations such as niche scenario descriptions, personalized dialogue content, and impromptu expressions, it can only provide generalized and untargeted basic feedback, and may even result in feedback content that is disconnected from the semantics of the dialogue, severely reducing the user's interactive experience.

[0006] Second, the interaction methods are simplistic and lack immersion. Most existing technologies employ single-dimensional feedback or simple multi-dimensional overlay methods, failing to achieve deep synergy between visual, auditory, and hardware actions. For example, visual effects and audio content are out of sync, and hardware actions do not match the emotional tone of dialogue, failing to create a unified interactive experience and making it difficult to meet users' high-experience demands in scenarios such as children's education, emotional companionship, and immersive entertainment.

[0007] Third, poor hardware compatibility limits the scope of application. The technical architecture of traditional feedback processing solutions is deeply tied to the performance of specific hardware, and the processing engine adopts a unified complex logic design without considering the performance differences between different hardware. This results in a solution often only being compatible with specific types or performance levels of smart hardware. When running on low-performance hardware, it will experience problems such as stuttering and latency, while on high-performance hardware, it cannot fully utilize its hardware advantages, which greatly limits the large-scale application of the technology.

[0008] The root cause of the above problems lies in the limitations of the logic of existing technologies. Traditional solutions are based on predefined matching and lack real-time dynamic processing capabilities, making it impossible to flexibly adjust the feedback format according to the actual dialogue content. At the same time, a real-time processing architecture for the entire chain has not been built. The parsing, orchestration, and output stages are independent of each other and have delays, making it difficult to achieve synchronous collaboration of multi-dimensional special effects. In addition, the design of the processing engine does not introduce lightweight and adaptive mechanisms, and cannot dynamically adjust the processing complexity according to hardware performance.

[0009] Therefore, there is an urgent need to develop an AI feedback processing optimization technology that can break through the limitations of predefined arrangement, achieve real-time dynamic linkage of multi-dimensional special effects, and be compatible with multiple types of smart hardware, so as to solve the above-mentioned problems of existing technologies and improve the interactive experience and application value of smart hardware. Summary of the Invention

[0010] To address this, the present invention provides an interactive AI smart hardware feedback processing optimization method and system, which solves the technical problems of existing smart hardware AI feedback relying on predefined rules, having limited ability to cope with undefined scenarios, poor multi-dimensional feedback coordination and insufficient immersion, and being unable to meet the high experience requirements of multiple scenarios.

[0011] To achieve the above objectives, the present invention provides the following technical solution: an interactive AI smart hardware feedback processing optimization method, comprising the following steps:

[0012] S1. The front-end real-time parsing layer receives the AI ​​dialogue return content output by the AI ​​server. The AI ​​dialogue return content includes text data and audio stream data. The text data and audio stream data are transmitted in real time through the streaming transmission submodule and a unique time sequence identifier is added to each of them.

[0013] S2. The segmented text data and framed audio data with the time sequence identifier are parsed synchronously through the parallel parsing submodule to obtain text parsing results including semantic and emotional features and audio parsing results including acoustic features, respectively.

[0014] S3. The semantic sentiment extraction unit performs temporal alignment of the text parsing result and the audio parsing result based on the temporal identifier, and then performs weighted fusion of the aligned text parsing result and the audio parsing result to generate a semantic-sentiment fusion tag including core scene, dominant emotion, sentence function, key entity and temporal interval.

[0015] S4. The AI ​​orchestration engine submodule of the intermediate dynamic orchestration layer generates visual effect instructions, auditory effect instructions, and hardware action instructions that match the semantic and emotional content of the dialogue based on the semantic-emotion fusion tags.

[0016] S5. The complexity adaptation submodule dynamically adjusts the complexity of the visual effects instructions, the auditory effects instructions, and the hardware action instructions based on the comprehensive performance parameters of the smart hardware to obtain the final multi-dimensional effects instruction set adapted to the target hardware's operating capabilities.

[0017] S6. The backend real-time output layer receives the final multi-dimensional special effects instruction set, executes the visual special effects instruction, the auditory special effects instruction and the hardware action instruction through the visual channel, the auditory special effects instruction and the motion channel respectively, and aligns the output timing of the special effects with the timing of the main content of the AI ​​dialogue through a synchronization control mechanism.

[0018] As a preferred scheme for optimizing the feedback processing of interactive AI smart hardware, in step S1, the streaming submodule establishes a real-time data transmission channel between the AI ​​server and the smart hardware using the HTTP / 2 streaming protocol or the WebSocket protocol; the mathematical model for text data segmentation and audio data framing in the streaming submodule is as follows:

[0019] Text data segmentation: The continuous text stream is traversed, and when a character is detected to belong to the sentence segmentation node set, it is segmented to obtain multiple text segments. Each text segment generates a unique time sequence identifier. The final text output set is:

[0020]

[0021] In the formula, The final output is a collection of text data with time-series identifiers. The total number of text segments. For the first Each text segment The text segment number, with a range of values. , For the first A unique temporal identifier for each text segment;

[0022] Audio data framing: The continuous audio stream is discretized and split according to a preset audio frame granularity constant. The total number of frames is the floor function of the total audio duration and the audio frame granularity constant. Each audio frame generates a unique timing identifier. The final audio output set is as follows:

[0023]

[0024] In the formula, This is the final output set of audio data with timing identifiers. This represents the total number of audio frames. For the first Frame audio, This refers to the audio frame number, with a value range of... , For the first A unique timing identifier for a frame of audio;

[0025] Timing alignment mapping: Establishing a timeline mapping relationship between text segments and corresponding audio frames, that is, each text segment corresponds to all audio frames within its pronunciation time interval, and the mapping relationship is as follows:

[0026]

[0027] In the formula, This is a time-alignment mapping function between text segments and audio frames. For the first The pronunciation start timestamp corresponding to each text fragment. For the first The pronunciation end timestamp corresponding to each text fragment.

[0028] As a preferred scheme for optimizing feedback processing of interactive AI smart hardware, the set of sentence segmentation nodes includes commas, periods, exclamation marks, and question marks; the temporal identifier of the text segment is the timestamp of the start of generation of the text segment on the global timeline; the temporal identifier of the audio frame is the timestamp of the start of the audio frame.

[0029] As a preferred scheme for optimizing the feedback processing of interactive AI smart hardware, in step S2, the parallel parsing submodule includes a text parsing unit and an audio parsing unit;

[0030] The text parsing unit integrates an NLP model to perform real-time parsing of each text data sentence by sentence and phrase by phrase, extracting scene keywords, emotion categories, sentence types, and key entities; the NLP model is a trimmed and optimized version of TinyBERT or MobileBERT.

[0031] The audio parsing unit uses an audio feature extraction algorithm to extract features from each frame of audio data, analyze tone changes, speech rate, and detect audio pauses.

[0032] As a preferred scheme for optimizing feedback processing in interactive AI smart hardware, the mathematical model of the weighted fusion algorithm for the semantic sentiment extraction unit in step S3 is as follows:

[0033] Weighted fusion calculation: For the dominant sentiment and sentence function dimensions, a linear weighted summation is used to calculate the fused score matrix. The calculation formula is as follows:

[0034]

[0035] In the formula, For the first The fusion score matrix of each parsed segment. For the first The temporal interval of each parsed segment The decision weights for the text parsing results. The decision weights for the audio parsing results satisfy the following conditions: , Time interval The output vector of the inner text parsing unit. Time interval The output vector of the internal audio parsing unit;

[0036] Final tag decoding: The tag with the highest fusion score is extracted using the index function corresponding to the maximum value, which is used as the dominant sentiment and sentence function. The calculation formula is as follows:

[0037]

[0038]

[0039] In the formula, To determine the dominant emotion in the final analysis, For a pre-defined set of dominant emotions, In the time series interval Internal Corresponding Emotions The fusion score; For the finalized statement functionality, For a pre-defined set of statement functions, In the time series interval Internal corresponding statement function The fusion score;

[0040] The core scenario and key entities directly inherit the item with the highest confidence level from the text parsing results. The calculation formula is as follows:

[0041]

[0042]

[0043] In the formula, For the final determined core scenario, For a pre-defined set of core scenarios, For text parsing units in time intervals Internal corresponding core scenarios Confidence score; The key entity to be extracted in the end. This is a key entity extraction function used to extract key entities from text parsing results;

[0044] Final output: Reassemble all calculated labels with time intervals to output the final semantic-sentiment fusion label. The calculation formula is as follows:

[0045]

[0046] In the formula, For the first The semantic-sentiment fusion label of each parsed segment is a quintuple that includes the core scene, dominant emotion, sentence function, key entity, and time interval.

[0047] As a preferred solution for optimizing the feedback processing of interactive AI smart hardware, in step S4, the AI ​​orchestration engine sub-module includes a visual effects orchestration unit, an auditory effects orchestration unit, and a hardware motion orchestration unit.

[0048] The visual effects arrangement unit calls the local lightweight scene resource library or edge computing node to generate scene images based on core scene keywords, generates facial expressions and punctuation effects based on dominant emotions and sentence functions, and adjusts the complexity of the effects according to hardware display capability parameters.

[0049] The auditory effects arrangement unit generates ambient sounds based on core scene keywords, generates tone auxiliary sounds based on dominant emotions and sentence functions, and determines the playback timing of each audio based on the time interval.

[0050] The hardware motion orchestration unit matches corresponding body movements based on the dominant emotion, sentence function, and core scene, adjusts the amplitude and frequency of the movements according to the intensity of the emotion, and determines the timing of the action execution based on the key nodes of the sentence.

[0051] As a preferred solution for optimizing feedback processing in interactive AI smart hardware, the mathematical model for scene generation in the visual effects choreography unit is as follows:

[0052] Scene generation function: Generates the corresponding scene image based on the core scene and hardware display capabilities. The calculation formula is as follows:

[0053]

[0054] In the formula, For the generated scene, Generate functions for the scene. This is a feature vector representing the hardware display capabilities, including screen resolution. Does it support dynamic video playback? Does it have projection functionality? ;

[0055] Complexity Adaptive Adjustment: Based on screen resolution and hardware performance, a complexity pruning function is used to perform tiered pruning of scene visuals and basic effects, outputting a set of visual effect instructions with strict temporal boundaries. The calculation formula is as follows:

[0056]

[0057] In the formula, Time interval The visual effects within ultimately execute the instruction set. This is a complexity trimming function. The basic effects collection, consisting of facial expression effects Punctuation effects composition;

[0058] The segmentation logic is as follows:

[0059]

[0060] In the formula, The preset low-resolution threshold, This is the preset high-resolution threshold.

[0061] As a preferred solution for optimizing feedback processing in interactive AI smart hardware, the maximum capacity limit of the local lightweight scene resource library... Frame rate constraint range for dynamic video The target frame rate is 15-30fps; if the hardware supports dynamic video playback, the target frame rate will be adaptively adjusted according to the overall hardware rendering performance index, calculated as follows:

[0062]

[0063] In the formula, The target frame rate for dynamic video. This is the floor function. This is a comprehensive hardware rendering performance index.

[0064] As a preferred solution for optimizing the feedback processing of interactive AI smart hardware, in step S5, the complexity adaptation submodule presets a hardware performance level classification standard, which divides the hardware into three levels: entry-level, intermediate-level, and high-end-level based on the processor clock speed and memory size of the smart hardware.

[0065] For entry-level hardware, hardware action commands are disabled to reduce the complexity of visual and auditory effects.

[0066] For advanced hardware, retain basic visual dynamic effects, auditory ambient sounds, and voice auxiliary sounds; if it has a simple body structure, retain basic motion commands.

[0067] For high-end hardware, retain all multi-dimensional special effects instructions and enable the most complex special effects parameters;

[0068] In step S6, the back-end real-time output layer uses the NTP time synchronization protocol through the synchronization control submodule to calibrate the execution time of the visual channel, auditory channel and motion channel; the auditory channel mixes the main audio stream with ambient sound and tone auxiliary sound in real time according to a preset volume ratio before outputting.

[0069] The present invention also provides an interactive AI smart hardware feedback processing optimization system, which adopts the above-mentioned interactive AI smart hardware feedback processing optimization method, including a front-end real-time parsing layer, an intermediate dynamic orchestration layer and a back-end real-time output layer.

[0070] The front-end real-time parsing layer receives AI dialogue return content from the AI ​​server. The AI ​​dialogue return content includes text data and audio stream data. The text data and audio stream data are transmitted in real time through a streaming transmission submodule, and a unique time sequence identifier is added to each. The parallel parsing submodule synchronously parses the segmented text data and framed audio data with the time sequence identifier, respectively obtaining text parsing results including semantic and emotional features and audio parsing results including acoustic features. The semantic and emotional extraction unit performs time sequence alignment of the text parsing results and the audio parsing results based on the time sequence identifier, and then performs weighted fusion of the aligned text parsing results and audio parsing results to generate semantic-emotional fusion tags including core scene, dominant emotion, sentence function, key entity and time sequence interval.

[0071] The intermediate dynamic orchestration layer is used by the AI ​​orchestration engine submodule to generate visual effect instructions, auditory effect instructions, and hardware action instructions that match the semantic and emotional content of the dialogue, respectively, based on the semantic-emotion fusion tags. The complexity adaptation submodule dynamically adjusts the complexity of the visual effect instructions, auditory effect instructions, and hardware action instructions based on the comprehensive performance parameters of the smart hardware, to obtain the final multi-dimensional effect instruction set adapted to the target hardware's operating capabilities.

[0072] The backend real-time output layer module is used to receive the final multi-dimensional special effects instruction set, execute the visual special effects instructions, the auditory special effects instructions and the hardware action instructions through the visual channel, the auditory special effects instructions and the motion channel respectively, and use a synchronization control mechanism to align the output timing of the special effects with the timing of the main content of the AI ​​dialogue.

[0073] The present invention has the following advantages:

[0074] This invention adopts an architecture of streaming transmission, parallel parsing, dynamic orchestration, and synchronous output. It abandons the traditional predefined rule matching mode and realizes that the dialogue content is the basis for orchestration. It can adapt to any dialogue scenario and completely solves the problem that existing technologies cannot cope with undefined semantic scenarios. It significantly improves the flexibility and pertinence of smart hardware feedback.

[0075] This invention breaks through the limitations of traditional single feedback or simple superposition, enabling special effects to be highly matched with the semantic and emotional content of dialogue, providing users with an immersive, multi-sensory interactive experience. It is particularly suitable for scenarios with high requirements for interactive experience, such as children's education, emotional companionship, and immersive entertainment.

[0076] This invention can dynamically adjust the complexity of special effects according to hardware performance, and is compatible with various smart hardware forms such as entry-level smart speakers, advanced children's learning machines, and high-end pet companion robots. It not only ensures the smooth operation of low-performance hardware, but also gives full play to the advantages of high-performance hardware, greatly expanding the application scope of the technology and reducing cross-hardware development costs.

[0077] This invention enables parallel parsing and weighted fusion of text and audio, comprehensively capturing the core semantics and emotional tendencies of dialogue. Compared with single-modal parsing methods, it significantly improves the accuracy of semantic emotion recognition and provides a reliable basis for the arrangement of multi-dimensional special effects.

[0078] This invention achieves accurate alignment between multi-dimensional special effects and the main content of AI dialogue by using a unified time sequence identifier and a simplified NTP time synchronization protocol, effectively avoiding desynchronization issues and ensuring the smoothness and consistency of the interaction. Attached Figure Description

[0079] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0080] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0081] Figure 1 This is a flowchart of the interactive AI smart hardware feedback processing optimization method provided in this embodiment of the invention;

[0082] Figure 2 This is a technical roadmap for the interactive AI smart hardware feedback processing optimization method provided in the embodiments of the present invention;

[0083] Figure 3 This is a schematic diagram of the interactive AI smart hardware feedback processing optimization system architecture provided in this embodiment of the invention. Detailed Implementation

[0084] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] Example 1

[0086] See Figure 1 and Figure 2 Embodiment 1 of the present invention provides an interactive AI smart hardware feedback processing optimization method, comprising the following steps:

[0087] S1. The front-end real-time parsing layer receives the AI ​​dialogue return content output by the AI ​​server. The AI ​​dialogue return content includes text data and audio stream data. The text data and audio stream data are transmitted in real time through the streaming transmission submodule and a unique time sequence identifier is added to each of them.

[0088] This architecture, employing front-end receiving and processing instead of centralized cloud processing, reduces end-to-end latency. The cloud is only responsible for generating dialogue content, while effects parsing and orchestration are handled on the hardware, avoiding network latency associated with sending effects commands to and from the cloud. Text contains explicit semantic, entity, and scene information, while audio includes emotional information such as tone and speed. The combination of bimodal data enables effects orchestration. Unlike traditional methods of receiving and processing complete content, streaming allows for simultaneous reception and processing, compressing overall latency from hundreds of milliseconds to tens of milliseconds, meeting the real-time requirements of human dialogue. A unique timing identifier establishes a unified global time base for text, audio, and all subsequently generated effects, resolving the asynchrony issues caused by asynchronous multimodal data.

[0089] S2. The segmented text data and framed audio data with the time sequence identifier are parsed synchronously through the parallel parsing submodule to obtain text parsing results including semantic and emotional features and audio parsing results including acoustic features, respectively.

[0090] This system leverages the multi-core computing power of smart hardware processors to simultaneously launch text parsing and audio parsing threads, rather than processing them serially, thus reducing overall parsing latency. Breaking the continuous data stream into independent small blocks allows for streaming processing—parsing as it's generated—and ensures that the failure of a single small block doesn't affect the overall process, improving system robustness. Text parsing extracts semantic and emotional features, providing content for subsequent special effects arrangement. It also supplements emotional details that text cannot convey; for example, the same word "good" can express doubt with a rising tone and affirmation with a falling tone, requiring differentiation through audio features.

[0091] S3. The semantic sentiment extraction unit performs temporal alignment on the text parsing results and the audio parsing results based on the temporal identifier, and then performs weighted fusion on the aligned text parsing results and the audio parsing results to generate semantic-sentiment fusion tags including core scene, dominant emotion, sentence function, key entity and temporal interval.

[0092] Temporal alignment is a prerequisite for multimodal fusion. Text segmentation and audio framing are performed independently, and there may be a time difference between text and audio corresponding to the same semantic content. Temporal identifiers bind the parsing results within the same time interval to ensure that the fused features are those of the same content. Information from different modalities contributes differently to different tasks. Text has a higher accuracy in recognizing semantics, entities, and scenes, while audio is more sensitive to recognizing emotions and tone. Therefore, different weights are assigned to achieve complementary advantages. Unstructured dialogue content is converted into a standardized data format that machines can directly understand, serving as the sole input for special effects choreography, thus achieving content-driven choreography rather than rule-driven choreography.

[0093] S4. The AI ​​orchestration engine submodule of the intermediate dynamic orchestration layer generates visual effect instructions, auditory effect instructions, and hardware action instructions that match the semantic and emotional content of the dialogue based on the semantic-emotion fusion tags.

[0094] The dynamic orchestration process abandons the traditional fixed keyword-effect mapping, transforming effect generation into a dynamic mapping process of semantic tags and effect parameters. As long as the corresponding semantic tag can be parsed, the corresponding effect can be generated, thus supporting any undefined dialogue scenario. Visual, auditory, and action effects are not generated in isolation, but based on the same semantic-emotional tag, ensuring consistency in the emotions and content expressed. For example, excitement corresponds simultaneously to a twinkling star effect, a light sound effect, and rapid body movements. Instructions contain only resource indexes, playback parameters, and timing information, resulting in minimal data volume and extremely fast transmission and processing speeds, avoiding the enormous computational consumption associated with real-time effect resource generation.

[0095] S5. The complexity adaptation submodule dynamically adjusts the complexity of the visual effects instructions, the auditory effects instructions, and the hardware action instructions based on the comprehensive performance parameters of the smart hardware, to obtain the final multi-dimensional effects instruction set adapted to the target hardware's operating capabilities.

[0096] One of the challenges in cross-hardware adaptation is the difference in hardware performance. Different types of smart hardware vary greatly in terms of processor computing power, memory size, storage space, display capabilities, and physical structure. The same set of special effects instructions can produce drastically different results on different hardware. After the special effects instructions are generated, they are trimmed or enhanced based on the hardware performance parameters to maximize the utilization of hardware performance while ensuring smooth operation. Tiered adaptation avoids the tedious work of developing a separate adaptation version for each piece of hardware, enabling a single codebase to run on multiple hardware devices, significantly reducing development and maintenance costs.

[0097] S6. The backend real-time output layer receives the final multi-dimensional special effects instruction set, executes the visual special effects instruction, the auditory special effects instruction and the hardware action instruction through the visual channel, the auditory special effects instruction and the motion channel respectively, and aligns the output timing of the special effects with the timing of the main content of the AI ​​dialogue through a synchronization control mechanism.

[0098] In this system, visual, auditory, and motion effects are executed by independent hardware modules (display screen, speakers, and motors). Parallel processing improves efficiency, and a failure in one channel does not affect others. Different channels execute at different speeds; for example, visual rendering may take a few milliseconds, while motor motion initiation may take tens of milliseconds. Without synchronized control, issues such as audio-visual asynchrony and delayed motion will occur. Special effects serve as supplementary expressions of dialogue and must precisely match the rhythm and content of the dialogue to enhance the experience; otherwise, they will create a sense of disharmony.

[0099] In one possible embodiment, in step S1, the streaming submodule establishes a real-time data transmission channel between the AI ​​server and the smart hardware using the HTTP / 2 streaming protocol or the WebSocket protocol; the mathematical model for text data segmentation and audio data framing in the streaming submodule is as follows:

[0100] Text data segmentation: The continuous text stream is traversed, and when a character is detected to belong to the sentence segmentation node set, it is segmented to obtain multiple text segments. Each text segment generates a unique time sequence identifier. The final text output set is:

[0101]

[0102] In the formula, The final output is a collection of text data with time-series identifiers. The total number of text segments. For the first Each text segment The text segment number, with a range of values. , For the first Each text segment is a unique temporal identifier. Natural language uses sentences as the basic semantic unit, and punctuation marks such as commas and periods mark semantic pauses. Segmenting by these nodes ensures that each segment contains complete semantic information, facilitating parsing and arrangement. Each segment is an independent processing unit, and the system can process these segments one by one to achieve streaming parsing. The temporal identifier is bound to the text segment, ensuring that each text segment has a unique timestamp. All processing based on that segment (parsing, arrangement, output) inherits this timestamp, achieving end-to-end temporal consistency.

[0103] Audio data framing: The continuous audio stream is discretized and split according to a preset audio frame granularity constant. The total number of frames is the floor function of the total audio duration and the audio frame granularity constant. Each audio frame generates a unique timing identifier. The final audio output set is as follows:

[0104]

[0105] In the formula, This is the final output set of audio data with timing identifiers. This represents the total number of audio frames. For the first Frame audio, This refers to the audio frame number, with a value range of... , For the first A frame is a unique temporal identifier for audio. While speech signals are non-stationary signals that vary over time, their spectral characteristics remain largely unchanged over short periods, allowing them to be processed as stationary signals. Rounding up ensures that all audio data is included in the frame, preventing the loss of audio segments shorter than a full frame at the end. Audio is a continuous time series, and the temporal identifier for each frame corresponds to its start time in the audio stream, providing a basis for aligning audio with text segments.

[0106] Timing alignment mapping: Establishing a timeline mapping relationship between text segments and corresponding audio frames, that is, each text segment corresponds to all audio frames within its pronunciation time interval, and the mapping relationship is as follows:

[0107]

[0108] In the formula, This is a time-alignment mapping function between text segments and audio frames. For the first The pronunciation start timestamp corresponding to each text fragment. For the first Each text segment corresponds to a pronunciation end timestamp. The text and audio generation processes are asynchronous; the AI ​​server generates text and audio independently. Text generation is typically faster than audio generation, so the audio frame corresponding to the same text segment may arrive later. The mapping function establishes a one-to-many relationship between text segments and audio frames. After text parsing is complete, the system can wait for all corresponding audio frames to arrive before performing fusion processing, ensuring that the fused features are those of the same content. The time interval is typically obtained by the AI ​​server providing the estimated pronunciation duration of the text synchronously during text generation, or by the hardware calculating it in real-time using speech endpoint detection technology.

[0109] In one possible embodiment, the preset audio frame granularity constant is set to 20ms; the sentence segment node set includes commas, periods, exclamation marks, and question marks; the time sequence identifier of the text segment is the start generation timestamp of the text segment on the global timeline; and the time sequence identifier of the audio frame is the start timestamp of the audio frame.

[0110] Specifically, the selection of audio frame granularity is based on meeting the short-term stationarity requirement of speech signals, enabling accurate extraction of speech features. Smaller granularity results in higher accuracy but also greater computational cost. Comma, period, exclamation mark, and question mark are chosen as sentence breakpoints; these are the most commonly used semantic pause punctuation marks in natural language, capable of dividing the text stream into appropriately long and semantically complete segments. A unified start timestamp is used as the timing identifier, simplifying the timing calculation logic; all processing times can be offset based on this start timestamp.

[0111] In one possible embodiment, in step S2, the parallel parsing submodule includes a text parsing unit and an audio parsing unit;

[0112] The text parsing unit integrates an NLP model to perform real-time parsing of each text data sentence by sentence and phrase by phrase, extracting scene keywords, emotion categories, sentence types, and key entities; the NLP model is a trimmed and optimized version of TinyBERT or MobileBERT.

[0113] Specifically, lightweight BERT models TinyBERT and MobileBERT, designed specifically for mobile and embedded devices, were selected. Through techniques such as model distillation, parameter pruning, and quantization, the model parameters were compressed from hundreds of millions to millions while maintaining high accuracy, enabling real-time operation on low-power processors in smart hardware. A sentence may contain multiple semantic segments, such as "I saw a cute little squirrel at the beach," which includes the "beach" scene and the "squirrel" entity. Phrasal parsing is required to extract complete information. Scene keywords are used to generate corresponding visual scenes and ambient sounds; emotion categories are used to generate corresponding facial expressions, tone sounds, and body movements; sentence types are used to generate corresponding punctuation effects and tone sounds; and key entities are used to generate corresponding entity effects and targeted body movements.

[0114] The audio parsing unit employs audio feature extraction algorithms to extract features from each frame of audio data, analyzing intonation changes, speech rate, and detecting audio pauses. Commonly used audio feature extraction algorithms include Mel-frequency cepstral coefficients (MFCC), Mel-frequency spectrum, and linear prediction coefficients (LPC). These algorithms can convert time-domain audio signals into frequency-domain feature vectors, facilitating computer processing. By extracting the fundamental frequency (F0) feature of the audio, a higher fundamental frequency indicates rising pitch (question, excitement), while a lower fundamental frequency indicates falling pitch (affirmation, sadness). Speech rate is determined by calculating the number of speech frames or syllables per unit time; a fast speech rate usually indicates excitement or urgency, while a slow speech rate usually indicates gentleness or sadness. Pauses are determined by detecting time periods when the audio energy is below a threshold; these pauses represent the optimal time to insert intonation accents and adjust body movements.

[0115] In one possible embodiment, the mathematical model of the weighted fusion algorithm of the semantic sentiment extraction unit in step S3 is as follows:

[0116] Weighted fusion calculation: For the dominant sentiment and sentence function dimensions, a linear weighted summation is used to calculate the fused score matrix. The calculation formula is as follows:

[0117]

[0118] In the formula, For the first The fusion score matrix of each parsed segment. For the first The temporal interval of each parsed segment The decision weights for the text parsing results. The decision weights for the audio parsing results satisfy the following conditions: , Time interval The output vector of the inner text parsing unit. Time interval The output vector of the internal audio parsing unit. Linear weighted summation is the simplest and most effective decision-level fusion method, with minimal computational cost, suitable for real-time operation on hardware, and its performance meets the needs of most scenarios. The constraint that the weights sum to 1 ensures that the score range after fusion is consistent with the score range of the individual modalities, facilitating label decoding. Only the dominant emotion and sentence function are fused; core scene and key entity information exist only in the text, which cannot be provided by audio, therefore no fusion is needed.

[0119] Final tag decoding: The tag with the highest fusion score is extracted using the index function corresponding to the maximum value, which is used as the dominant sentiment and sentence function. The calculation formula is as follows:

[0120]

[0121]

[0122] In the formula, To determine the dominant emotion in the final analysis, For a pre-defined set of dominant emotions, In the time series interval Internal Corresponding Emotions The fusion score; For the finalized statement functionality, For a pre-defined set of statement functions, In the time series interval Internal corresponding statement function The fusion score. The function selects the category with the highest probability as the output, which aligns with human decision-making logic. A pre-defined label set limits the types of emotions and sentence functions the system can recognize, ensuring that special effects choreography can find corresponding resources while reducing model complexity. The score matrix is ​​split dimensionally, separating the fused score matrix into emotion scores and sentence function scores, which are then decoded separately, improving decoding accuracy.

[0123] The core scenario and key entities directly inherit the item with the highest confidence level from the text parsing results. The calculation formula is as follows:

[0124]

[0125]

[0126] In the formula, For the final determined core scenario, For a pre-defined set of core scenarios, For text parsing units in time intervals Internal corresponding core scenarios Confidence score; The key entity to be extracted in the end. This is a key entity extraction function used to extract key entities from the text parsing results. The core scene directly inherits from the text parsing results; the audio signal does not contain scene information and cannot identify the specific scene from the audio. The key entity extraction function extracts specific entities such as names of people, places, and objects from the text parsing results. These entities are crucial for generating targeted special effects; for example, when "squirrel" is mentioned, the hardware can mimic squirrel movements.

[0127] Final output: Reassemble all calculated labels with time intervals to output the final semantic-sentiment fusion label. The calculation formula is as follows:

[0128]

[0129] In the formula, For the first The semantic-sentiment fusion tags for each parsed segment are quintuples comprising the core scene, dominant emotion, sentence function, key entity, and time interval. This quintuple structure contains all the information needed for special effects choreography, with the time interval specifying the effect's execution time and the other four elements defining the effect's content. This structured output facilitates rapid reading and processing by the AI ​​choreography engine, eliminating the need to re-parse unstructured text content.

[0130] In one possible embodiment, in step S4, the AI ​​orchestration engine submodule includes a visual effects orchestration unit, an auditory effects orchestration unit, and a hardware motion orchestration unit.

[0131] The visual effects orchestration unit generates scene visuals by calling a local lightweight scene resource library or edge computing nodes based on core scene keywords. It generates facial expressions and punctuation effects based on the dominant emotion and sentence function, and adjusts the complexity of the effects according to hardware display capabilities. The local resource library stores commonly used basic scene elements, ensuring fast access and no network latency. Edge computing nodes store less frequently used complex scene resources; when the corresponding resource is not available locally, it is retrieved from the edge node in real time, saving local storage space and avoiding the high latency of the cloud. Facial expressions and punctuation effects intuitively express the emotions and tone of the dialogue, enhancing the fun of the interaction; for example, a rotating question mark corresponds to a question, and twinkling stars correspond to excitement. Adjusting the complexity based on hardware display capabilities ensures that the effects display correctly on different hardware, considering variations in screen resolution, dynamic video support, and projection support.

[0132] The auditory effects arrangement unit generates ambient sounds based on core scene keywords, generates auxiliary tone sounds based on dominant emotions and sentence functions, and determines the playback timing of each audio segment according to time intervals. Ambient sounds enhance the sense of immersion in the scene; for example, the sound of waves in a beach scene or birdsong in a rainforest scene makes the user feel as if they are actually there. Auxiliary tone sounds reinforce the emotions and tone of the dialogue; for example, a ding sound when asking a question or a light, cheerful sound effect when happy, compensating for the shortcomings of pure voice expression. Ambient sounds need to start and end synchronously with the dialogue content, and auxiliary tone sounds need to be played at key points in the sentence (such as pauses or keywords) to achieve the best effect.

[0133] The hardware motion orchestration unit matches corresponding body movements based on the dominant emotion, sentence function, and core scenario, adjusts the amplitude and frequency of the movements according to the intensity of the emotion, and determines the timing of the action execution based on key nodes in the sentence. The matching of actions with semantic emotion is based on research findings in human behavior and psychology. For example, tilting the head when questioning, jumping when happy, and swaying slowly when gentle are all common human body language that users can naturally understand. The amplitude and frequency of the movements are positively correlated with the intensity of the emotion; the stronger the emotion, the larger the amplitude and the higher the frequency of the movements. For example, wagging the tail rapidly when extremely excited and wagging the tail slowly when slightly happy allows for a more nuanced expression of emotion. Executing corresponding actions when keywords appear, such as executing a jump action when jumping is mentioned, ensures that actions and content accurately correspond, enhancing the interactive experience.

[0134] In one possible embodiment, the mathematical model for scene generation by the visual effects choreography unit is as follows:

[0135] Scene generation function: Generates the corresponding scene image based on the core scene and hardware display capabilities. The calculation formula is as follows:

[0136]

[0137] In the formula, For the generated scene, Generate functions for the scene. This is a feature vector representing the hardware display capabilities, including screen resolution. Does it support dynamic video playback? Does it have projection functionality? The scene generation function is an abstract interface function whose internal implementation can be flexibly adjusted according to hardware capabilities and resource availability. For example, it can call video resources on hardware that supports dynamic video and static image resources on hardware that does not support it. The hardware display capability feature vector quantifies the hardware's display capabilities and serves as an input parameter to the function, ensuring that the generated scene matches the hardware capabilities.

[0138] Complexity Adaptive Adjustment: Based on screen resolution and hardware performance, a complexity pruning function is used to perform tiered pruning of scene visuals and basic effects, outputting a set of visual effect instructions with strict temporal boundaries. The calculation formula is as follows:

[0139]

[0140] In the formula, Time interval The visual effects within ultimately execute the instruction set. This is a complexity trimming function. The basic effects collection, consisting of facial expression effects Punctuation effects Composition. The complexity pruning function reduces the computational load and resource consumption of special effects without changing their expressive meaning. The basic effect set contains all possible facial and punctuation effects; the pruning function selects appropriate effects from this set for output based on hardware capabilities. Strict timing boundaries ensure that the execution time of visual effects is completely consistent with the timing interval of the dialogue content.

[0141] The segmentation logic is as follows:

[0142]

[0143] In the formula, The preset low-resolution threshold, This is a preset high-resolution threshold. The three-level tiered adaptation logic covers all hardware types from entry-level to high-end, representing the optimal solution for balancing experience and performance. Screen resolution is the primary criterion, as it is the most significant factor affecting visual effects and computational load; higher resolution allows for more complex effects. Low-resolution screens cannot clearly display dynamic effects and may even cause stuttering, thus forcing the use of static effects.

[0144] In one possible embodiment, the maximum capacity limit of the local lightweight scene resource library Frame rate constraint range for dynamic video The target frame rate is 15-30fps; if the hardware supports dynamic video playback, the target frame rate will be adaptively adjusted according to the overall hardware rendering performance index, calculated as follows:

[0145]

[0146] In the formula, The target frame rate for dynamic video. This is the floor function. This is a comprehensive hardware rendering performance index.

[0147] Specifically, considering that entry-level smart hardware typically only has a few hundred MB to a few GB of storage space, a 100 MB resource library won't occupy too much storage space while still storing enough commonly used scene elements. 15fps is the minimum frame rate at which the human eye can perceive continuous animation; below 15fps, noticeable stuttering occurs. 30fps is the standard frame rate for smooth animation; above 30fps, the human eye can hardly distinguish the difference and it increases unnecessary computation. The hardware rendering performance index is linearly mapped to the 15-30fps range; better performance results in a higher frame rate, maximizing visual effects while ensuring smoothness. A rounding function ensures the frame rate is an integer, meeting the requirements of video encoding and display.

[0148] In one possible embodiment, in step S5, the complexity adaptation submodule presets a hardware performance level classification standard, and classifies the hardware into three levels: entry-level, intermediate-level, and high-end-level according to the processor clock speed and memory size of the smart hardware.

[0149] For entry-level hardware, hardware action commands are disabled to reduce the complexity of visual and auditory effects. Entry-level hardware typically lacks limb structure and has a processor clock speed of ≤1GHz ​​and memory of ≤1GB, making it unable to handle multi-dimensional effects simultaneously. Therefore, hardware action commands are disabled, retaining only the most basic visual and auditory effects to ensure smooth system operation.

[0150] For advanced hardware, basic visual dynamic effects, auditory ambient sounds, and voice prompts are retained. If the device has simple body structures, basic motion commands are also retained. Advanced hardware has a processor clock speed between 1-2 GHz and 1-2 GB of memory, capable of handling moderately complex effects. Therefore, basic multi-dimensional effects are retained to provide a better interactive experience.

[0151] For high-end hardware, all multi-dimensional special effects instructions are retained, and the most complex special effects parameters are enabled. High-end hardware has a processor clock speed of >2GHz, memory of >2GB, and typically features rich limb structures and high-definition display screens, enabling smooth operation of the most complex special effects. Therefore, enabling all functions provides the best interactive experience.

[0152] In one possible embodiment, in step S6, the back-end real-time output layer uses the NTP time synchronization protocol through the synchronization control submodule to calibrate the execution time of the visual channel, auditory channel and motion channel; the auditory channel mixes the main audio stream with ambient sound and tone auxiliary sound in real time according to a preset volume ratio before outputting.

[0153] Specifically, while the standard NTP protocol achieves millisecond-level accuracy, it involves significant computation. This invention employs a simplified version, calibrating only the relative time of each channel, with accuracy controlled within 10ms, sufficient for interactive needs while substantially reducing computational load. A hardware audio mixing module proportionally superimposes multiple audio streams into a single output stream, avoiding the latency and sound quality loss associated with software mixing. The main audio stream, being the core content of the dialogue, needs to be clearly distinguishable and therefore accounts for the largest proportion; ambient sounds and auxiliary voice prompts are supplementary content, and their volume must not overpower the main audio, thus their proportion is lower.

[0154] The application scenarios of this invention are as follows:

[0155] I. Children's Educational Scenarios:

[0156] Applicable hardware includes children's learning machines, early education robots, smart reading pens, and AR picture book readers.

[0157] Typical applications are as follows:

[0158] Immersive picture book reading experience: When AI tells the story of "Adventures in the Rainforest", the system analyzes the story content in real time, dynamically generates dynamic scenes of the rainforest and ambient sounds of birdsong and flowing water, and controls the robot to make corresponding actions such as "imitating monkey climbing" and "squirrel jumping", realizing an immersive reading experience that integrates "sound-image-action".

[0159] Interactive Q&A: When a child asks "Why do volcanoes erupt?", the AI ​​answers the question while simultaneously displaying a dynamic diagram of a volcanic eruption on the screen, playing sound effects of lava flow, and the robot makes "simulated eruption" body movements, transforming abstract knowledge into a multi-sensory experience.

[0160] Personalized early childhood education: The feedback format is adjusted according to the child's learning mood. When the child shows confusion, the explanation speed is automatically reduced, accompanied by soothing sound effects and gentle movements. When the child answers a question correctly, excited facial expressions, cheerful sound effects, and robot clapping are triggered to enhance the child's learning enthusiasm.

[0161] II. Scenarios of Emotional Companionship:

[0162] Applicable hardware includes pet companion robots, humanoid companion robots, smart speakers with screens, and elderly companion terminals.

[0163] Typical applications are as follows:

[0164] Emotional perception and soothing: When a user expresses "I'm in a bad mood today", the system recognizes the sad emotion through a combination of text semantics and voice tone, automatically switches to a gentle voice, plays soothing background music, displays healing images on the screen, and controls the robot to make soothing actions such as "slowly shaking its head" and "gently rubbing its hand".

[0165] Daily interactive companionship: When chatting with users, corresponding special effects are generated in real time based on the content of the conversation. For example, if a user says "I saw a rainbow today", a rainbow effect will be displayed on the screen immediately and a lighthearted sound effect will be played. When a user tells a joke, the robot will make a "laughing" expression and gesture in sync to enhance the realism of the interaction.

[0166] Intelligent schedule reminders: When reminding users to take medicine or exercise, the feedback format is adjusted according to the reminder content. For example, the reminder to take medicine uses a gentle tone and a "heart" effect, while the reminder to exercise uses an energetic sound effect and a "cheering" gesture, avoiding the resistance caused by harsh voice reminders.

[0167] III. Family Entertainment Scenarios:

[0168] Applicable hardware includes smart speakers, home theater control systems, AI game controllers, and desktop companion robots.

[0169] Typical applications are as follows:

[0170] Interactive puzzle / board game: When playing a puzzle game, the system provides hints and special effects step by step according to the content of the puzzle. For example, when guessing an animal, the system will show partial features of the animal in sequence, make the corresponding sounds, and the robot will imitate the actions, which will enhance the fun of the game.

[0171] Music Interactive Experience: When playing music, corresponding visual effects are generated in real time according to the style and rhythm of the song (such as particle effects for rock style and ink painting effects for classical style), while controlling the robot to sway to the rhythm, and the LED lights change color and brightness in sync.

[0172] Family role-playing: Supports users to play role-playing games with AI, such as "space exploration" and "fairy tale kingdom". The system dynamically generates scene images, environmental sound effects and robot actions according to the development of the plot, creating an immersive family entertainment experience.

[0173] IV. Smart Home Control Scenarios:

[0174] Applicable hardware includes smart home control screens, smart speakers, smart switch panels, and whole-house smart hosts.

[0175] Typical applications are as follows:

[0176] Intuitive device control: When a user says "turn on the living room warm light", the central control screen simultaneously displays a dynamic effect of the warm light turning on, and the light gradually brightens; when adjusting the air conditioner temperature, the screen displays an animation of temperature changes and the current temperature value, while playing a "beep" confirmation sound effect, realizing a closed loop of "voice-visual-feedback".

[0177] Scene mode linkage: When the "Homecoming Mode" is triggered, the system turns on the lights, curtains, and air conditioner in sequence. The central control screen simultaneously displays the on-state animation of each device, plays a warm welcome sound effect, and the robot makes a "Welcome Home" gesture, creating a unified scene experience.

[0178] Abnormal status alerts: When abnormalities such as gas leaks or open doors and windows are detected, in addition to voice alarms, the screen displays a flashing red warning effect, the robot quickly moves to the user's side and makes a warning action, and the multi-channel alerts ensure that the user is informed in a timely manner.

[0179] V. Commercial Service Scenarios:

[0180] Applicable hardware includes intelligent shopping guide robots, exhibition hall explanation robots, hotel service robots, and bank lobby robots.

[0181] Typical applications are as follows:

[0182] Dynamic product explanation: When the sales robot introduces a product, it generates corresponding dynamic demonstration screens and functional effects in real time based on the product's characteristics. For example, when introducing a robot vacuum cleaner, the screen displays an animation of its cleaning path, and the robot makes "simulated cleaning" actions, allowing customers to understand the product's functions more intuitively.

[0183] Personalized exhibition hall explanations: The content and format of the explanations are adjusted according to the audience's interests. When the audience shows great interest in a certain exhibit, the explanation time is automatically extended to show more detailed effects and historical background, while corresponding body language is used to enhance the appeal of the explanation.

[0184] Guided service experience: When the hotel robot guides guests to their rooms, the screen displays a route navigation animation and plays clear voice instructions. At the same time, it uses body language to indicate directions to prevent guests from getting lost. When they arrive at their rooms, the robot makes a "please come in" gesture to enhance the service experience.

[0185] It should be noted that the method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0186] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0187] Example 2

[0188] See Figure 3 Embodiment 2 of the present invention also provides an interactive AI smart hardware feedback processing optimization system, which adopts the interactive AI smart hardware feedback processing optimization method of the above embodiments, including a front-end real-time parsing layer, an intermediate dynamic orchestration layer and a back-end real-time output layer.

[0189] The front-end real-time parsing layer receives AI dialogue return content from the AI ​​server. The AI ​​dialogue return content includes text data and audio stream data. The text data and audio stream data are transmitted in real time through a streaming transmission submodule, and a unique time sequence identifier is added to each. The parallel parsing submodule synchronously parses the segmented text data and framed audio data with the time sequence identifier, respectively obtaining text parsing results including semantic and emotional features and audio parsing results including acoustic features. The semantic and emotional extraction unit performs time sequence alignment of the text parsing results and the audio parsing results based on the time sequence identifier, and then performs weighted fusion of the aligned text parsing results and audio parsing results to generate semantic-emotional fusion tags including core scene, dominant emotion, sentence function, key entity and time sequence interval.

[0190] The intermediate dynamic orchestration layer is used by the AI ​​orchestration engine submodule to generate visual effect instructions, auditory effect instructions, and hardware action instructions that match the semantic and emotional content of the dialogue, respectively, based on the semantic-emotion fusion tags. The complexity adaptation submodule dynamically adjusts the complexity of the visual effect instructions, auditory effect instructions, and hardware action instructions based on the comprehensive performance parameters of the smart hardware, to obtain the final multi-dimensional effect instruction set adapted to the target hardware's operating capabilities.

[0191] The backend real-time output layer module is used to receive the final multi-dimensional special effects instruction set, execute the visual special effects instructions, the auditory special effects instructions and the hardware action instructions through the visual channel, the auditory special effects instructions and the motion channel respectively, and use a synchronization control mechanism to align the output timing of the special effects with the timing of the main content of the AI ​​dialogue.

[0192] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0193] Example 3

[0194] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for an interactive AI smart hardware feedback processing optimization method. The program code includes instructions for executing the interactive AI smart hardware feedback processing optimization method of Embodiment 1 or any possible implementation thereof.

[0195] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0196] Example 4

[0197] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0198] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the interactive AI smart hardware feedback processing optimization method of Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0199] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0200] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0201] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0202] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. An interactive AI smart hardware feedback processing optimization method, characterized in that, Includes the following steps: S1. The front-end real-time parsing layer receives the AI ​​dialogue return content output by the AI ​​server. The AI ​​dialogue return content includes text data and audio stream data. The text data and audio stream data are transmitted in real time through the streaming transmission submodule and a unique time sequence identifier is added to each of them. S2. The segmented text data and framed audio data with the time sequence identifier are parsed synchronously through the parallel parsing submodule to obtain text parsing results including semantic and emotional features and audio parsing results including acoustic features, respectively. S3. The semantic sentiment extraction unit performs temporal alignment of the text parsing result and the audio parsing result based on the temporal identifier, and then performs weighted fusion of the aligned text parsing result and the audio parsing result to generate a semantic-sentiment fusion tag including core scene, dominant emotion, sentence function, key entity and temporal interval. S4. The AI ​​orchestration engine submodule of the intermediate dynamic orchestration layer generates visual effect instructions, auditory effect instructions, and hardware action instructions that match the semantic and emotional content of the dialogue based on the semantic-emotion fusion tags. S5. The complexity adaptation submodule dynamically adjusts the complexity of the visual effects instructions, the auditory effects instructions, and the hardware action instructions based on the comprehensive performance parameters of the smart hardware to obtain the final multi-dimensional effects instruction set adapted to the target hardware's operating capabilities. S6. The backend real-time output layer receives the final multi-dimensional special effects instruction set, executes the visual special effects instruction, the auditory special effects instruction and the hardware action instruction through the visual channel, the auditory special effects instruction and the motion channel respectively, and aligns the output timing of the special effects with the timing of the main content of the AI ​​dialogue through a synchronization control mechanism.

2. The interactive AI smart hardware feedback processing optimization method according to claim 1, characterized in that, In step S1, the streaming submodule establishes a real-time data transmission channel between the AI ​​server and the smart hardware using the HTTP / 2 streaming protocol or the WebSocket protocol; the mathematical model for text data segmentation and audio data framing in the streaming submodule is as follows: Text data segmentation: The continuous text stream is traversed, and when a character is detected to belong to the sentence segmentation node set, it is segmented to obtain multiple text segments. Each text segment generates a unique time sequence identifier. The final text output set is: In the formula, The final output is a collection of text data with time-series identifiers. The total number of text segments. For the first Each text segment The text segment number, with a range of values. , For the first A unique temporal identifier for each text segment; Audio data framing: The continuous audio stream is discretized and split according to a preset audio frame granularity constant. The total number of frames is the floor function of the total audio duration and the audio frame granularity constant. Each audio frame generates a unique timing identifier. The final audio output set is as follows: In the formula, This is the final output set of audio data with timing identifiers. This represents the total number of audio frames. For the first Frame audio, This refers to the audio frame number, with a value range of... , For the first A unique timing identifier for a frame of audio; Timing alignment mapping: Establishing a timeline mapping relationship between text segments and corresponding audio frames, that is, each text segment corresponds to all audio frames within its pronunciation time interval, and the mapping relationship is as follows: In the formula, This is a time-alignment mapping function between text segments and audio frames. For the first The pronunciation start timestamps corresponding to each text fragment. For the first The timestamp indicating the end of pronunciation for each text segment.

3. The interactive AI smart hardware feedback processing optimization method according to claim 2, characterized in that, The set of sentence break nodes includes commas, periods, exclamation marks, and question marks; the temporal identifier of the text segment is the timestamp at which the text segment was generated on the global timeline; the temporal identifier of the audio frame is the timestamp at which the audio frame started.

4. The interactive AI smart hardware feedback processing optimization method according to claim 1, characterized in that, In step S2, the parallel parsing submodule includes a text parsing unit and an audio parsing unit; The text parsing unit integrates an NLP model to perform real-time parsing of each text data sentence by sentence and phrase by phrase, extracting scene keywords, emotion categories, sentence types, and key entities; the NLP model is a trimmed and optimized version of TinyBERT or MobileBERT. The audio parsing unit uses an audio feature extraction algorithm to extract features from each frame of audio data, analyze tone changes, speech rate, and detect audio pauses.

5. The interactive AI smart hardware feedback processing optimization method according to claim 1, characterized in that, In step S3, the mathematical model of the weighted fusion algorithm for the semantic sentiment extraction unit is as follows: Weighted fusion calculation: For the dominant sentiment and sentence function dimensions, a linear weighted summation is used to calculate the fused score matrix. The calculation formula is as follows: In the formula, For the first The fusion score matrix of each parsed segment. For the first The temporal interval of each parsed segment The decision weights for the text parsing results. The decision weights for the audio parsing results satisfy the following conditions: , Time interval The output vector of the inner text parsing unit. Time interval The output vector of the internal audio parsing unit; Final tag decoding: The tag with the highest fusion score is extracted using the index function corresponding to the maximum value, which is used as the dominant sentiment and sentence function. The calculation formula is as follows: In the formula, To determine the dominant emotion in the final analysis, For a pre-defined set of dominant emotions, In the time series interval Internal Corresponding Emotions The fusion score; For the finalized statement functionality, For a pre-defined set of statement functions, In the time series interval Internal corresponding statement function The fusion score; The core scenario and key entities directly inherit the item with the highest confidence level from the text parsing results. The calculation formula is as follows: In the formula, For the finalized core scenario, For a pre-defined set of core scenarios, For text parsing units in time intervals Internal corresponding core scenarios Confidence score; The key entity to be extracted in the end. This is a key entity extraction function used to extract key entities from text parsing results; Final output: Reassemble all calculated labels with time intervals to output the final semantic-sentiment fusion label. The calculation formula is as follows: In the formula, For the first The semantic-sentiment fusion label of each parsed segment is a quintuple that includes the core scene, dominant emotion, sentence function, key entity, and time interval.

6. The interactive AI smart hardware feedback processing optimization method according to claim 5, characterized in that, In step S4, the AI ​​choreography engine submodule includes a visual effects choreography unit, an auditory effects choreography unit, and a hardware motion choreography unit. The visual effects arrangement unit calls the local lightweight scene resource library or edge computing node to generate scene images based on core scene keywords, generates facial expressions and punctuation effects based on dominant emotions and sentence functions, and adjusts the complexity of the effects according to hardware display capability parameters. The auditory effects arrangement unit generates ambient sounds based on core scene keywords, generates tone auxiliary sounds based on dominant emotions and sentence functions, and determines the playback timing of each audio based on the time interval. The hardware motion orchestration unit matches corresponding body movements based on the dominant emotion, sentence function, and core scene, adjusts the amplitude and frequency of the movements according to the intensity of the emotion, and determines the timing of the action execution based on the key nodes of the sentence.

7. The interactive AI smart hardware feedback processing optimization method according to claim 6, characterized in that, The mathematical model for generating scene visuals in the visual effects choreography unit is as follows: Scene generation function: Generates the corresponding scene image based on the core scene and hardware display capabilities. The calculation formula is as follows: In the formula, For the generated scene, Generate functions for the scene. This is a feature vector representing the hardware display capabilities, including screen resolution. Does it support dynamic video playback? Does it have projection functionality? ; Complexity Adaptive Adjustment: Based on screen resolution and hardware performance, a complexity pruning function is used to perform tiered pruning of scene visuals and basic effects, outputting a visual effects instruction set with strict temporal boundaries. The calculation formula is as follows: In the formula, Time interval The visual effects within ultimately execute the instruction set. This is a complexity trimming function. The basic effects collection, consisting of facial expression effects Punctuation effects composition; The segmentation logic is as follows: In the formula, The preset low-resolution threshold, This is the preset high-resolution threshold.

8. The interactive AI smart hardware feedback processing optimization method according to claim 7, characterized in that, Maximum capacity limit of the local lightweight scenario resource library Frame rate constraint range for dynamic video 15-30fps; If the hardware supports dynamic video playback, the target frame rate will be adaptively adjusted according to the overall hardware rendering performance index, calculated as follows: In the formula, The target frame rate for dynamic video. This is the floor function. This is a comprehensive hardware rendering performance index.

9. The interactive AI smart hardware feedback processing optimization method according to claim 1, characterized in that, In step S5, the complexity adaptation submodule presets a hardware performance level classification standard, which divides the hardware into three levels: entry-level, intermediate-level, and high-end-level based on the processor clock speed and memory size of the smart hardware. For entry-level hardware, hardware action commands are disabled to reduce the complexity of visual and auditory effects. For advanced hardware, retain basic visual dynamic effects, auditory ambient sounds, and voice auxiliary sounds; if it has a simple body structure, retain basic motion commands. For high-end hardware, retain all multi-dimensional special effects instructions and enable the most complex special effects parameters; In step S6, the back-end real-time output layer uses the NTP time synchronization protocol through the synchronization control submodule to calibrate the execution time of the visual channel, auditory channel and motion channel; the auditory channel mixes the main audio stream with ambient sound and tone auxiliary sound in real time according to a preset volume ratio before outputting.

10. An interactive AI smart hardware feedback processing optimization system, employing the interactive AI smart hardware feedback processing optimization method according to any one of claims 1 to 9, characterized in that, It includes a front-end real-time parsing layer, a middle dynamic orchestration layer, and a back-end real-time output layer; The front-end real-time parsing layer is used to receive the AI ​​dialogue return content output by the AI ​​server. The AI ​​dialogue return content includes text data and audio stream data. The text data and audio stream data are transmitted in real time through the streaming transmission submodule and a unique time sequence identifier is added to each of them. The parallel parsing submodule synchronously parses the segmented text data and framed audio data with the time sequence identifier, and obtains text parsing results including semantic and emotional features and audio parsing results including acoustic features, respectively. The semantic sentiment extraction unit performs temporal alignment of the text parsing results and the audio parsing results based on the temporal identifier, and then performs weighted fusion of the aligned text parsing results and audio parsing results to generate semantic-sentiment fusion tags including core scene, dominant emotion, sentence function, key entity and temporal interval. The intermediate dynamic orchestration layer is used by the AI ​​orchestration engine submodule to generate visual effect instructions, auditory effect instructions, and hardware action instructions that match the semantic and emotional content of the dialogue, respectively, based on the semantic-emotion fusion tags. The complexity adaptation submodule dynamically adjusts the complexity of the visual effect instructions, auditory effect instructions, and hardware action instructions based on the comprehensive performance parameters of the smart hardware, to obtain the final multi-dimensional effect instruction set adapted to the target hardware's operating capabilities. The backend real-time output layer module is used to receive the final multi-dimensional special effects instruction set, execute the visual special effects instructions, the auditory special effects instructions and the hardware action instructions through the visual channel, the auditory special effects instructions and the motion channel respectively, and use a synchronization control mechanism to align the output timing of the special effects with the timing of the main content of the AI ​​dialogue.

Images