A unified streaming processing method, system, device, medium and program product of multi-modal AI interactive content
By using a unified streaming processing method, the output of the multimodal AI interaction system is transmitted in a single ordered stream, which solves the problems of inconsistent modal outputs and fragmented inference, realizes efficient and scalable multimodal AI interaction, and improves user experience and system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING DIANFU TECHNOLOGY CO LTD
- Filing Date
- 2025-12-18
- Publication Date
- 2026-06-09
AI Technical Summary
Existing multimodal AI interaction systems suffer from inconsistent modal output formats, fragmented inference processes, complex integration, and a lack of unified quality and performance metrics, resulting in high parsing costs, poor interactive experiences, and insufficient scalability.
A unified streaming processing method is adopted. By receiving image and text requests, a context is constructed, image features are extracted, image parsing and intent recognition are performed, multi-stage reasoning is carried out and content fragments are generated in a streaming manner, content fragments are buffered and published, recommendation triggering and event management are performed, a standardized message fragmentation format is defined, and the orderly streaming transmission of different types of output is realized.
It reduces parsing costs, improves interactive experience and scalability, enhances user experience and system performance, supports flexible expansion of modalities and fault self-healing, provides robustness and observability, and ensures accurate restoration and security of interactive behavior.
Smart Images
Figure CN121705057B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of AI (intelligent) interaction technology, and in particular to a unified streaming processing method, system, device, medium and program product for multimodal AI interactive content. Background Technology
[0002] With the integration of large models with multimodal retrieval, video generation / recommendation, and other capabilities, multimodal AI (intelligent) interaction systems have emerged. However, current multimodal intelligent interaction systems still have the following problems during interaction:
[0003] 1. The inconsistent output formats of different modalities require the front-end or caller to write multiple sets of parsing logic.
[0004] 2. The reasoning process is separated from the final answer, making it impossible to expose the model's thinking progress and uncertainties in a timely manner during the generation process.
[0005] 3. Video recommendation, structured metadata, and event notifications (such as stream start, end, error, and interruption) are usually implemented through independent interfaces or asynchronous messages, which are complex to integrate and prone to losing timing relationships.
[0006] 4. The lack of unified quality and performance metrics makes it difficult to perform fine-grained optimizations for aspects such as first token latency and recommended trigger timing.
[0007] Therefore, there is an urgent need to invent a universal method that can transmit different types of output in a single ordered stream to reduce parsing costs, improve interactive experience and scalability. Summary of the Invention
[0008] In view of this, embodiments of the present invention provide a unified streaming processing method, system, device, medium, and program product for multimodal AI interactive content, which at least partially solves the problems existing in the prior art.
[0009] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.
[0010] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0011] In a first aspect, the present invention provides a unified streaming processing method for multimodal AI interactive content, comprising:
[0012] Receive a request from the user containing images and text, and construct a request context;
[0013] Extract image features based on the request context;
[0014] Based on the image features, perform image parsing and output image description text;
[0015] Intent recognition is performed based on the image description text and the text in the user's request, and the intent is output.
[0016] Based on the request context, perform multi-stage reasoning and generate thought content fragments in a streaming manner;
[0017] Buffer and publish the thought content fragments based on the aforementioned thought content fragments;
[0018] Generate text content fragments based on multi-stage reasoning;
[0019] Buffer and publish the text content fragment based on the aforementioned text content fragment;
[0020] Recommendations are triggered based on the image description text and intent;
[0021] Based on the recommendation, a recommended content fragment is triggered, and the recommended content fragment contains metadata;
[0022] Insert the recommended content fragment into the recommended position, buffer it, and publish it;
[0023] It also includes process event management, which publishes event notifications when the process starts, any step fails, or the process ends.
[0024] As a further improvement of the present invention, the step of buffering and publishing the thought content fragments based on the thought content fragments includes: generating thought content fragments through a text flow based on the thought content fragments generated by incremental reasoning streaming; aggregating and segmenting the generated thought content fragments; buffering the aggregated and segmented thought content fragments, and generating and publishing thought content fragments based on the merging rules of concatenation within the same stage and immediate dequeueing across stages; inputting the published thought content fragments into the publish-subscribe queue and distributing the thought content fragments.
[0025] Furthermore, the step of buffering and publishing the text content fragment based on the text content fragment includes: generating a text content fragment through a text stream based on the text content fragment generated by incremental inference streaming; buffering and throttling the generated text content fragment; buffering and publishing the buffered and throttled text content fragment again; and inputting the published text content fragment into a publish-subscribe queue and distributing the text content fragment.
[0026] Furthermore, the step of inserting the recommended content fragment into the recommended position, buffering and publishing it includes: inserting the recommended content fragment into the recommended position; buffering the recommended content fragment inserted into the recommended position and publishing the recommended content fragment; inputting the published recommended content fragment into the publish-subscribe queue and distributing the recommended content fragment.
[0027] Furthermore, the step of publishing event notifications when starting, when any step fails, and when ending includes: marking the event when starting, when any step fails, and when ending; buffering and publishing the marked event; inputting the published event into the publish-subscribe queue and sending the event notification.
[0028] Furthermore, the recommendation triggering based on the image description text and intent includes: after the first complete text content fragment is published, the recommendation triggering decision matrix calculates the relevance, and the recommendation is triggered based on the relevance; the recommendation triggering decision matrix includes: combining the image description text, intent, and current inference stage into multi-dimensional conditions, and triggering the recommendation only when the relevance score exceeds the configured threshold and the frequency control is not violated;
[0029] The method of triggering recommended content segments based on the recommendation includes: generating video recommended segments if the recommendation is triggered, wherein the video recommended segments include a title, cover, URL, trigger tag and relevance score.
[0030] Furthermore, the buffering and throttling includes a ternary strategy based on a character cumulative threshold, semantic boundary markers, and a maximum time window to perform granular control on the text content segmentation.
[0031] Furthermore, the unified streaming processing method also includes: assigning a strictly incremental sequence number to each published segment; rendering the segments sequentially on the client based on the sequence number; allowing requests for re-issuance or marking downgrades if a sequence number gap is detected; performing a "sequence coherence judgment" before publishing the segments, and issuing STREAM_ERROR or STREAM_TERMINATED if an exception is found.
[0032] Furthermore, the unified streaming processing method also includes: performing policy filtering on sensitive words, illegal tags and / or invalid video resources before the release segment is formed, and replacing them with safe placeholders or triggering EVENT downgrade notifications when necessary.
[0033] Furthermore, the unified streaming processing method also includes in-situ metric injection; the in-situ metric injection includes writing metric records at the moment of buffer dequeue or recommendation trigger completion, the metric records include: first toke delay, recommendation offset, and buffer residency.
[0034] Furthermore, the unified streaming processing method also includes retaining a minimum persistent record, which includes: sequence number, type, source, and digest hash.
[0035] Secondly, the present invention also provides a unified streaming processing system for multimodal AI interactive content, comprising:
[0036] The dialogue access device is configured to receive requests from users that include images and text, and to construct a request context;
[0037] The agent scheduling device includes an agent orchestration module and an intent recognition module;
[0038] The Agent orchestration module is configured to: extract image features based on the request context; request the LLM image understanding module to perform image parsing and output image description text based on the image features; request the LLM inference engine to perform multi-stage inference and stream content fragments based on the request context; request the LLM inference engine to generate text content fragments based on multi-stage inference; trigger recommendations based on the image description text and intent; request the content recommendation engine to recommend content fragments based on the recommendation trigger, wherein the recommended content fragments contain metadata; and manage process events.
[0039] The intent recognition module is configured to: recognize intent based on the image description text and the text in the request sent by the user, and output the intent;
[0040] The stream generation and processing apparatus is configured to: buffer and publish the thought content fragment based on the thought content fragment; buffer and publish the text content fragment based on the text content fragment; insert the recommended content fragment into a recommended position, buffer and publish it; and publish event notifications when starting, any step fails, and when ending.
[0041] Furthermore, the stream generation and processing device includes a text stream generator, an event processing chain, and a buffer manager; the event processing chain is a pluggable event processing chain.
[0042] The text stream generator is configured to: generate thought content fragments from incremental reasoning streams; and generate text content fragments from incremental reasoning streams.
[0043] The event processing chain is configured to: aggregate and segment based on generated thought content fragments; buffer and throttle based on generated text content fragments; insert recommended content fragments into recommended positions; and mark events when starting, any step fails, or ends.
[0044] The buffer manager is configured to: buffer aggregated and segmented thought content fragments; generate thought content fragments and publish them based on the merging rules of concatenation within the same stage and immediate dequeueing across stages; buffer and publish the buffered and throttled text content fragments; buffer and publish recommended content fragments inserted at recommended positions; and buffer and publish tagged events.
[0045] The unified streaming processing system also includes a publish-subscribe queue module, configured to: input a published thought content fragment and send the thought content fragment; input a published text content fragment and send the text content fragment; input a published recommended content fragment and send the recommended content fragment; input a published event and send the event notification.
[0046] Thirdly, the present invention provides a computer device, the device comprising: a processor and a memory;
[0047] The memory is used to store one or more program instructions;
[0048] The processor is configured to run one or more program instructions to perform the steps of a unified streaming method for multimodal AI interactive content as described above.
[0049] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described unified streaming processing method for multimodal AI interactive content.
[0050] Fifthly, the present invention also provides a computer program product, the computer program product including computer program instructions, which, when executed by a processor, implement the steps of the unified streaming processing method for multimodal AI interactive content as described above.
[0051] The unified streaming processing method for multimodal AI interactive content of this invention defines a standardized message fragmentation format, which dynamically outputs four types of messages—thinking content, text output, recommended content, and flow control events—and their extensible subtypes to the calling end in chronological and logical order. It is a general method that can transmit different types of output in a single ordered stream, which can reduce parsing costs, improve interactive experience, and enhance scalability. Attached Figure Description
[0052] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.
[0053] Figure 1 A flowchart of a unified streaming processing method for multimodal AI interactive content provided in an embodiment of the present invention;
[0054] Figure 2This is an interactive flowchart of a unified streaming processing method for multimodal AI interactive content provided in an embodiment of the present invention.
[0055] Figure 3 This is a schematic diagram of the structure of a unified streaming system for multimodal AI interactive content provided in an embodiment of the present invention; Detailed Implementation
[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0058] Figure 1 The diagram shows a flowchart of a unified streaming processing method for multimodal AI interactive content provided by an embodiment of the present invention. Figure 2 The diagram illustrates an interactive flowchart of a unified streaming method for multimodal AI interactive content provided by an embodiment of the present invention.
[0059] Combination Figure 1 , 2 As shown in this embodiment, a unified streaming processing method for multimodal AI interactive content includes:
[0060] Step S1: Receive the request sent by the user, which includes images and text, and construct the request context.
[0061] More specifically, the client receives a request sent by the user: the request contains the image to be analyzed and the text "Please help me analyze the image content". The client calls the dialogue API interface, i.e. the dialogue access device. The dialogue API interface performs access verification: the interface layer verifies the request structure, authentication token, and the legality of image references. After the access verification is passed, the request context (including image references) is constructed.
[0062] Furthermore, policies can be loaded according to needs, such as loading whether to enable the display of thought-provoking content and the upper limit of recommendation frequency from the configuration center / user preferences.
[0063] Here, the multimodal request sequence includes: image analysis, reasoning, content (video) recommendation + events.
[0064] Step S2: Extract image features based on the request context.
[0065] More specifically, the Agent orchestration module extracts image features based on the request context. These features include extracting the image's subject, scene, color, texture, or OCR, which can be selected as needed.
[0066] Step S3: Perform image parsing based on image features and output image description text;
[0067] More specifically, the image features extracted by the Agent orchestration module are used to request the LLM image understanding module to parse the image and output image description text; the image description text typically includes tags and semantic fragments.
[0068] Step S4: Perform intent recognition based on the image description text and the text in the user's request, and output the intent;
[0069] More specifically, the intent recognition module uses the image description text fed back to the Agent orchestration module by the LLM image understanding module, such as based on tags, and combines it with the text in the user's request (such as "Please help me analyze the image content") to perform intent recognition and classification, and output high-level intent (such as "Description of healthy eating scenario").
[0070] Step S5: Perform multi-stage reasoning based on the request context and generate thought content fragments in a streaming manner;
[0071] More specifically, the Agent orchestration module requests the LLM inference engine to perform multi-stage inference (including image semantics) based on the request context and generates thought content fragments in a streaming manner. Here, the LLM inference engine adopts an incremental inference approach, which generates staged thoughts based on the context; when the accumulated tokens are greater than or equal to the threshold or exceed the time window T, thought content is constructed and thought content fragments are generated in a streaming manner.
[0072] Step S6: Buffer and publish the thought content fragments.
[0073] More specifically, this includes: generating thought content fragments based on incremental inference streaming generated by the LLM inference engine, and generating thought content fragments through a text stream generator; the event processing chain aggregating and segmenting thought content fragments generated by the text stream generator; the cache manager buffering the aggregated and segmented thought content fragments, and the buffer can generate thought content fragments and publish them according to the merging rules of concatenation within the same stage and immediate dequeueing across stages; and inputting the published thought content fragments into the publish-subscribe queue and distributing the thought content fragments.
[0074] Step S7: Generate text content fragments based on multi-stage reasoning.
[0075] More specifically, the request is to have the LLM inference engine output a structured description and suggestions after converging inference. Here, the LLM inference engine uses incremental inference, generating text content fragments in a streaming manner.
[0076] Step S8: Buffer and publish the text content fragments;
[0077] More specifically, this includes: generating text content fragments based on incremental inference streaming, and then generating text content fragments through a text stream generator; the event processing chain buffers and throttles the text content fragments generated by the text stream generator. This buffering and throttling includes: a ternary strategy based on a character accumulation threshold, semantic boundary markers, and a maximum time window to granularly control the text content fragmentation. Here, an adaptive buffering and throttling algorithm is used to granularly control the text content fragmentation, balancing real-time performance and rendering stability.
[0078] The buffer manager buffers and publishes the buffered and throttled text content fragments; it then inputs the published text content fragments into the publish-subscribe queue and distributes the text content fragments.
[0079] Step S9: Trigger recommendations based on image description text and intent.
[0080] More specifically, after the first complete text content fragment is published, the recommendation trigger decision matrix calculates the relevance and triggers recommendations based on the relevance. The recommendation trigger decision matrix includes: multi-dimensional conditions composed of image description text (tag), intent, and current inference stage, and recommendations are triggered only when the relevance score exceeds the configured threshold and frequency control is not violated. Among them, the recommended content fragments based on the recommendation trigger include: if the recommendation is triggered, a video recommendation fragment is generated, which includes title, cover, URL, trigger tag and relevance score.
[0081] The aforementioned recommendation trigger decision matrix incorporates image labels, user intent classification, the current inference stage, and may also include historical interaction features to form multi-dimensional conditions; recommendation intent is generated only when the "relevance score" exceeds the configured threshold and frequency control is not violated.
[0082] Step S10: Trigger recommended content fragments based on recommendations; the recommended content fragments contain metadata.
[0083] More specifically, the recommendation engine here triggers recommended content snippets based on recommendations, metadata such as cover images, etc.
[0084] Step S11: Insert the recommended content fragment into the recommended position, buffer it, and publish it.
[0085] More specifically, the event handling chain inserts recommended content fragments into recommended positions, and the buffer manager...
[0086] Buffer the recommended content fragments to be inserted at the recommended position, and publish the recommended content fragments; input the published recommended content fragments into the publish-subscribe queue, and then distribute the recommended content fragments.
[0087] Step S12: This also includes process event management, issuing event notifications when starting, any step fails, or ends.
[0088] More specifically, the event handling chain marks events when they start, when any step fails, and when they end; the buffer manager buffers and publishes the marked events; the published events are entered into the publish-subscribe queue, and event notifications are sent.
[0089] In the above process event management, STREAM_START can be sent at the start of the flow; if any step fails (such as an image parsing error), STREAM_ERROR (including the error code and scope of impact) can be sent. STREAM_END can be sent upon successful completion (end).
[0090] Here, the failure branch is made explicit. For example, the error is packaged as an EVENT type and includes a structured error code, stage, scope of impact, and suggested action. The front end is programmable.
[0091] The unified streaming processing method described above also includes: assigning strictly incrementing sequence numbers to each published segment; rendering the segments sequentially on the client based on the sequence numbers; allowing requests for re-sending or marking downgrades if a gap in the sequence numbers is detected; performing a "sequence continuity check" before publishing the segments, and issuing STREAM_ERROR or STREAM_TERMINATED if an exception is found.
[0092] This means that a sequence consistency protocol is set up here. Client behaviors also include: rendering sequentially based on sequence number; providing a "stop reasoning" interaction for thought content; offering "play now / view later" options for video recommendations; and updating the state machine driven by events.
[0093] In addition, as a preferred embodiment, the above-mentioned unified streaming processing method also includes an embedded security filtering node: performing policy filtering on sensitive words, illegal tags and / or invalid video resources before the release chunk is formed, and replacing them with security placeholders or triggering EVENT downgrade notifications when necessary.
[0094] Preferably, the above-mentioned unified streaming processing method further includes in-situ metric injection; in-situ metric injection includes writing metric records at the moment of buffer dequeue or recommendation trigger completion, the metric records include: first toke delay, recommendation offset, and buffer residency; wherein, MetricsCollector writes metric records at the moment of buffer dequeue or recommendation trigger completion in an "observer" manner, without blocking the main process.
[0095] Specifically, the metrics recorded may include the first thought content, the first text, the first REC_VIDEO, the total streaming time, and the average segment size. These metrics are recorded for performance analysis.
[0096] Preferably, the aforementioned unified streaming processing method further includes retaining a minimal persistent record, which includes: sequence number, type, source, and digest hash. Here, by retaining the minimal persistent record, the server enables replay / reconstruction support, allowing subsequent auditing or training data distillation to accurately restore the interaction.
[0097] Figure 3 This is a schematic diagram of the structure of a unified streaming system for multimodal AI interactive content provided in an embodiment of the present invention.
[0098] Combination Figure 3 As shown in the figure, this embodiment provides a unified streaming processing system for multimodal AI interactive content, including:
[0099] The dialogue access device is configured to receive requests from users that include images and text, and to construct a request context;
[0100] The agent scheduling device includes an agent orchestration module and an intent recognition module;
[0101] The Agent orchestration module is configured to: extract image features based on the request context; request the LLM image understanding module to parse the image and output image description text based on the image features; request the LLM inference engine to perform multi-stage inference and generate thought content fragments in a streaming manner based on the request context; request the LLM inference engine to generate text content fragments based on multi-stage inference; trigger recommendations based on image description text and intent; request the content recommendation engine to recommend content fragments based on the recommendation trigger, and the recommended content fragments contain metadata; and manage process events.
[0102] The intent recognition module is configured to: recognize intent based on the image description text and the text in the request sent by the user, and output the intent;
[0103] The stream generation and processing device is configured to: buffer and publish thought content fragments based on thought content fragments; buffer and publish text content fragments based on text content fragments; insert recommended content fragments into recommended positions, buffer and publish them; and publish event notifications when starting, any step fails, and when ending.
[0104] More specifically, the stream generation and processing apparatus includes a text stream generator, an event handling chain, and a buffer manager; the event handling chain is a pluggable event handling chain.
[0105] The text stream generator is configured to: generate thought content fragments from incremental reasoning streams; and generate text content fragments from incremental reasoning streams.
[0106] The event handling chain is configured as follows: aggregate and segment based on generated thought content fragments; buffer and throttle based on generated text content fragments; insert recommended content fragments into recommended positions; and mark events when starting, at any step fails, or at the end.
[0107] The aforementioned event processing chain is a hierarchical event processing chain: processors are registered according to the stages of "startup → incremental inference → text generation → recommendation insertion → closing", which can support parallel / delayed injection and dynamic short-circuiting.
[0108] The buffer manager is configured to: buffer aggregated and segmented thought content fragments; generate and publish thought content fragments based on the merging rules of concatenation within the same stage and immediate dequeueing across stages; buffer and publish the buffered and throttled text content fragments; buffer and publish recommended content fragments inserted in recommended positions; and buffer and publish tagged events.
[0109] The unified streaming system also includes a publish-subscribe queue module, which is configured to send published thought content snippets as input; send published text content snippets as input; send published recommended content snippets as input; and send published event notifications as input.
[0110] This invention provides a unified streaming processing method for multimodal AI interactive content. By defining a standardized MessageStreamChunk message sharding format, and using pluggable event handling chains, buffering and aggregation management, Redis streaming publish / subscribe, intent and recommendation triggering components on the server side, it dynamically outputs four types of messages—text output, thought content, recommended content, and flow control events—and their extensible subtypes to the calling end in chronological and logical order. This is a general method that can transmit different types of output in a single ordered stream, reducing parsing costs, improving interactive experience, and enhancing scalability.
[0111] The unified streaming processing method for multimodal AI interactive content of the present invention also has the following advantages:
[0112] 1. Improved development efficiency: The unified Chunk protocol eliminates the costs of parsing and maintaining multiple protocols; adding new modalities only requires adding rendering strategies and processors, reducing iteration friction.
[0113] 2. Enhanced User Experience: Phased incremental output of thought-provoking content shortens the psychological waiting time; video recommendations and text-based presentations enhance information relevance and engagement.
[0114] 3. Performance and resource optimization: Buffering and throttling algorithms reduce fragmented messages and redraw times; adaptive frequency reduction in weak network conditions improves smoothness and mobile power consumption.
[0115] 4. Scalability and module decoupling: The event processing chain and the recommendation decision matrix are loosely coupled, supporting on-demand plugging and unplugging and canary activation of new capabilities without affecting the main line.
[0116] 5. Robustness and fault self-healing: Event-based errors and degradation strategies (STREAM_ERROR / TERMINATED / degradation EVENT) enable clients to quickly identify and execute alternative paths without additional polling.
[0117] 6. Convenient observation and optimization: In-situ metrics (first token delay, recommended offset, buffer dwell) form a closed-loop feedback, supporting continuous optimization.
[0118] 7. Data Replay and Audit: Standardized sequences, source tags, and compliance markers enable precise reconstruction of interactive behaviors, meeting the needs of quality analysis and compliance documentation.
[0119] 8. Security and Compliance: Embedded filters intercept non-compliant content in advance, reducing the risk of violations spreading, and presenting the handling results transparently in the form of events.
[0120] 9. Improved Recommendation Relevance: The decision matrix utilizes multi-dimensional features (image labels + intent + stage) to improve the accuracy of recommendation triggering and avoid excessive interference.
[0121] 10. Cost control: Unified flow reduces the number of additional recommendation or event interface calls, and reduces network and backend connection overhead.
[0122] In addition, embodiments of the present invention also provide a computer device, the device comprising: a processor and a memory; the memory being used to store one or more program instructions; the processor being used to execute one or more program instructions to perform the steps of a unified streaming processing method for multimodal AI interactive content as described above.
[0123] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a unified streaming processing method for multimodal AI interactive content as described above.
[0124] In addition, embodiments of the present invention also provide a computer program product, which includes computer program instructions that, when executed by a processor, implement the steps of a unified streaming processing method for multimodal AI interactive content as described above.
[0125] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods. The storage medium can be memory, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory. Those skilled in the art will recognize that the functions described in the above examples can be implemented using a combination of hardware and software. When applied software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers. Although the invention has been described in detail above with general description and specific embodiments, modifications or improvements can be made to it, which will be apparent to those skilled in the art. Therefore, such modifications or improvements made without departing from the spirit of the invention are all within the scope of protection claimed by the invention.
[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, or alterations made by those skilled in the art using the disclosed technical content shall fall within the protection scope of the present invention.
Claims
1. A unified streaming processing method for multimodal AI interactive content, characterized in that, include: Receive a request from the user containing images and text, and construct a request context; Extract image features based on the request context; Based on the image features, perform image parsing and output image description text; Intent recognition is performed based on the image description text and the text in the user's request, and the intent is output. Based on the request context, perform multi-stage reasoning and generate thought content fragments in a streaming manner; Buffer and publish the thought content fragments based on the aforementioned thought content fragments; Generate text content fragments based on multi-stage reasoning; Buffer and publish the text content fragment based on the aforementioned text content fragment; Recommendations are triggered based on the image description text and intent; Based on the recommendation, a recommended content fragment is triggered, and the recommended content fragment contains metadata; Insert the recommended content fragment into the recommended position, buffer it, and publish it; It also includes process event management, which publishes event notifications when the process starts, any step fails, or the process ends; The process of buffering and publishing thought content fragments based on the thought content fragments includes: generating thought content fragments through a text stream based on thought content fragments generated by incremental reasoning; aggregating and segmenting the generated thought content fragments; buffering the aggregated and segmented thought content fragments, generating thought content fragments based on the merging rules of concatenation within the same stage and immediate dequeueing across stages, and publishing the published thought content fragments; inputting the published thought content fragments into the publish-subscribe queue and distributing the thought content fragments. The process of buffering and publishing text content fragments based on the text content fragments includes: generating text content fragments through a text stream based on text content fragments generated by incremental inference streaming; buffering and throttling the generated text content fragments; buffering and publishing the buffered and throttled text content fragments again; and inputting the published text content fragments into a publish-subscribe queue and distributing the text content fragments. The step of inserting the recommended content fragment into the recommended position, buffering and publishing includes: inserting the recommended content fragment into the recommended position; buffering the recommended content fragment inserted into the recommended position and publishing the recommended content fragment; inputting the published recommended content fragment into the publish-subscribe queue and distributing the recommended content fragment. The step of publishing event notifications when starting, at any failed step, or at the end includes: marking the event when starting, at any failed step, or at the end; buffering and publishing the marked event; inputting the published event into the publish-subscribe queue and sending the event notification; The unified streaming processing method further includes: assigning a strictly incrementing sequence number to each published fragment; and rendering it sequentially on the client based on the sequence number.
2. The unified streaming processing method for multimodal AI interactive content according to claim 1, characterized in that, The recommendation triggering based on the image description text and intent includes: after the first complete text content fragment is published, the recommendation triggering decision matrix calculates the relevance, and the recommendation triggering is performed based on the relevance; the recommendation triggering decision matrix includes: combining the image description text, intent, and current inference stage into multi-dimensional conditions, and only triggering the recommendation when the relevance score exceeds the configured threshold and does not violate the frequency control. The method of triggering recommended content segments based on the recommendation includes: generating video recommended segments if the recommendation is triggered, wherein the video recommended segments include a title, cover, URL, trigger tag and relevance score.
3. The unified streaming processing method for multimodal AI interactive content according to claim 2, characterized in that, The buffering and throttling include: The text content is segmented using a ternary strategy based on character cumulative threshold, semantic boundary marker, and maximum time window.
4. The unified streaming processing method for multimodal AI interactive content according to claim 1, characterized in that, The unified streaming processing method further includes: if a sequence number gap is detected, a request for resending or a downgrade flag is allowed; a "sequence coherence judgment" is performed before publishing the segment, and if an exception is found, a STREAM_ERROR or STREAM_TERMINATED is issued; And / or, the unified streaming processing method further includes: performing policy filtering on sensitive words, illegal tags and / or invalid video resources before the published segment is formed, or replacing them with security placeholders or triggering EVENT downgrade notifications; And / or, the unified streaming processing method further includes in-situ metric injection; the in-situ metric injection includes: writing metric records at the moment of buffer dequeue or recommendation trigger completion, the metric records including: first toke delay, recommendation offset, and buffer residency; And / or, the unified streaming method further includes retaining a minimal persistent record, which includes: sequence number, type, source, and digest hash.
5. A unified streaming processing system for multimodal AI interactive content, characterized in that, include: The dialogue access device is configured to receive requests from users that include images and text, and to construct a request context; The agent scheduling device includes an agent orchestration module and an intent recognition module; The Agent orchestration module is configured to extract image features based on the request context; Based on the image features, the LLM image understanding module is requested to parse the image and output image description text; based on the request context, the LLM inference engine is requested to perform multi-stage inference and generate thought content fragments in a streaming manner; the LLM inference engine is requested to generate text content fragments based on multi-stage inference; and recommendation is triggered based on the image description text and intent. Based on the recommendation trigger request, the content recommendation engine recommends content fragments, which contain metadata; and performs process event management. The intent recognition module is configured to: recognize intent based on the image description text and the text in the request sent by the user, and output the intent; The stream generation and processing device is configured to: buffer and publish the thought content fragment based on the thought content fragment; buffer and publish the text content fragment based on the text content fragment; insert the recommended content fragment into the recommended position, buffer and publish it; and publish event notifications when starting, any step fails, and when ending. The stream generation and processing device includes a text stream generator, an event processing chain, and a buffer manager; the event processing chain is a pluggable event processing chain. The text stream generator is configured to: generate thought content fragments from incremental reasoning streams; and generate text content fragments from incremental reasoning streams. The event processing chain is configured to: aggregate and segment based on generated thought content fragments; buffer and throttle based on generated text content fragments; insert recommended content fragments into recommended positions; and mark events when starting, any step fails, or ends. The buffer manager is configured to: buffer aggregated and segmented thought content fragments; generate thought content fragments and publish them based on the merging rules of concatenation within the same stage and immediate dequeueing across stages; buffer and publish the buffered and throttled text content fragments; buffer and publish recommended content fragments inserted at recommended positions; and buffer and publish tagged events. The unified streaming processing system also includes a publish-subscribe queue module, configured to: input a published thought content fragment and send the thought content fragment; input a published text content fragment and send the text content fragment; input a published recommended content fragment and send the recommended content fragment; input a published event and send the event notification. The unified streaming processing system also includes a published fragment sequential rendering module, which is configured to assign a strictly incremental sequence number to each published fragment; The serial numbers are rendered sequentially on the client side.
6. A computer device, characterized in that, The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of a unified streaming method for multimodal AI interactive content as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a unified streaming method for multimodal AI interactive content as described in any one of claims 1 to 4.
8. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed by a processor, implement the steps of a unified streaming method for multimodal AI interactive content as described in any one of claims 1 to 4.