Information processing systems, methods, apparatus and equipment

Abstract

This application discloses an information processing system, method, apparatus, and device. The system generates a text block stream via a server; assigns a block identifier to each text block; generates an audio-visual synchronization display instruction based on the block identifier, the text block, and corresponding audio data; sends the audio-visual synchronization display instruction to a client; receives the audio-visual synchronization display instruction through the client; and synchronously displays the audio data and the text block according to the audio-visual synchronization display instruction. This processing method achieves streaming audio-visual synchronization based on the server's "intercepting of stream-generated text blocks and packaging the text blocks into instructions with audio-visual synchronization signaling." Therefore, it can effectively reduce the output latency of multimodal streaming interaction, lower the performance requirements of the client, and improve the accuracy of audio-visual synchronization.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to information processing systems, methods and apparatus, and electronic devices. Background Technology

[0002] In modern intelligent interaction, streaming output has become a standard feature for large model applications (LLM) in order to improve the naturalness and efficiency of human-computer interaction. Users no longer need to wait for AI to generate complete, lengthy articles, but can see and hear the AI's "thinking" and "expression" process in real time, just like having a conversation with a person.

[0003] On multimodal devices with screens (voice playback and text display), an ideal streaming interactive experience is one where, as the voice plays a word or sentence, the corresponding text on the screen is simultaneously highlighted, achieving streaming synchronous highlighting of voice playback and screen text. Currently, a typical solution for achieving streaming synchronous highlighting of voice playback and screen text involves the server (cloud) providing the terminal with a complete text generated by a large language model and a corresponding complete audio segment. The terminal then performs synchronous matching processing of voice progress and text position based on the complete text and audio, such as complex audio allocation or timestamp matching, to achieve audio-visual synchronization.

[0004] However, in implementing this application, the applicant found that the existing solutions have at least the following problems: 1) The cloud provides the terminal with a complete text and a complete audio segment, so the latency of the streaming output is high, which affects the user experience; 2) The terminal needs complex audio analysis and text processing capabilities, which not only places high demands on the terminal performance, but also makes it difficult to align accurately. Summary of the Invention

[0005] This application provides an information processing system to address the problems of high streaming output latency, high terminal performance requirements, and low synchronization accuracy in existing technologies. This application also provides an information processing method and apparatus, as well as electronic equipment.

[0006] This application provides an information processing system, including: The server is used to generate a text block stream; assign block identifiers to the text blocks; generate an audio-visual synchronization display instruction based on the block identifier, the text block, and the corresponding audio data; and send the audio-visual synchronization display instruction to the client. The client is used to receive the audio-visual synchronization display instruction and to synchronously display the voice data and the text block according to the audio-visual synchronization display instruction.

[0007] This application also provides an information processing method, including: Generate a stream of text blocks; Assign a block identifier to the text block; Based on the block identifier, the text block, and the voice data corresponding to the text block, generate an audio-visual synchronized display instruction; Send the audio-visual synchronization display command to the device.

[0008] Optionally, generating an audio-visual synchronized display instruction based on the block identifier, the text block, and the voice data corresponding to the text block includes: Generate a text block display instruction based on the block identifier and the text block; Based on the block identifier and the voice data corresponding to the text block, a voice broadcast command is generated.

[0009] Optionally, the generation of the text block stream includes: The AI ​​agent invokes at least one tool based on the required information; Invoke the large language model based on at least one piece of content returned by the at least one tool; The large language model generates a comprehensive content text block stream based on at least one of the contents.

[0010] Optional, also includes: Determine the target tool from the at least one tool; Obtain the interface template information of the target tool; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, and the voice data corresponding to the text block includes: Based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool, an audio-visual synchronized display instruction is generated.

[0011] Optional, also includes: The first text synchronous display mode for the requirement processing task is set, and the target interface template information of the requirement processing task is set as the interface template information of the target tool; the first text synchronous display mode includes synchronous or asynchronous modes. Before generating the audio-visual synchronization display instruction, the following is also included: Obtain the first text synchronization display mode and the target interface template information; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool includes: If the first text synchronization display mode is synchronized, then a text block display instruction is generated based on the block identifier, the target interface template information, and the text block; Based on the block identifier and the voice data corresponding to the text block, a voice broadcast command is generated.

[0012] Optionally, determining the target tool from the at least one tool includes: Obtain the second text synchronization display mode of the tool; From the tools whose second text synchronization display mode is synchronized, identify the target tool; The first text synchronization display mode for setting up the demand processing task includes: If a tool exists with a second text synchronization display mode set to synchronization, then the first text synchronization display mode will be set to synchronization.

[0013] Optionally, determining the target tool from the at least one tool includes: The target tool is determined from the at least one tool based on the priority of the tools.

[0014] Optionally, determining the target tool from the at least one tool includes: Obtain the second text synchronization display mode of the tool; From the tools whose second text synchronization display mode is synchronized, identify the target tool.

[0015] Optionally, obtaining the second text synchronization display mode of the tool includes: Obtain the interface instructions returned by the tool, the interface instructions including a second text synchronization display mode.

[0016] Optionally, the interface instructions may also include tool task identifiers; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool includes: Based on the tool task identifier, the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool, an audio-visual synchronized display instruction is generated.

[0017] Optional, also includes: Delete the interface command.

[0018] Optional, also includes: Detect whether the device displays an interface on a dedicated screen other than the at least one tool; If the detection result is yes, then the first text synchronous display mode will be set to asynchronous.

[0019] Optional, also includes: If any of the at least one tools is in conflict, then information regarding the tool conflict is sent to the device.

[0020] Optional, also includes: Get text display properties; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, and the voice data corresponding to the text block includes: Based on the block identifier, the text display attribute, the text block, and the voice data corresponding to the text block, an audio-visual synchronized display instruction is generated.

[0021] Optionally, obtaining the text display attributes includes: Obtain the text display attributes corresponding to the text block.

[0022] Optionally, assigning a block identifier to the text block includes: Get the allocated block identifier; Based on the already assigned block identifier, determine the block identifier assigned to the text block.

[0023] Optional, also includes: Obtain the task identifier for processing the text block stream; Based on the block identifier, the text block, and the corresponding voice data, a synchronized audio-visual display instruction is generated, including: Based on the task identifier, the block identifier, the text block, and the voice data corresponding to the text block, an audio-visual synchronized display instruction is generated.

[0024] This application also provides an information processing method, including: Receive audio-visual synchronization display instruction; the audio-visual synchronization display instruction includes a text block, voice data corresponding to the text block, and a block identifier, wherein the text block belongs to a text block stream; According to the audio-visual synchronization display instruction, the voice data and the text block are displayed synchronously.

[0025] Optionally, the text block belongs to a comprehensive content text block stream generated by the artificial intelligence agent based on demand information; The audio-visual synchronization display command includes the interface template information of the target tool; Displaying the text block includes: Based on the interface template information, the text block is displayed in the interface of the target tool.

[0026] This application also provides an information processing apparatus, including: The text block stream generation unit is used to generate text block streams; A block identifier allocation unit is used to assign a block identifier to the text block; The instruction generation unit is used to generate an audio-visual synchronized display instruction based on the block identifier, the text block, and the voice data corresponding to the text block; The instruction sending unit is used to send the audio-visual synchronization display instruction to the device.

[0027] This application also provides an information processing apparatus, including: The instruction receiving unit is used to receive an audio-visual synchronization display instruction; the audio-visual synchronization display instruction includes a text block, voice data corresponding to the text block, and a block identifier, wherein the text block belongs to a text block stream; The instruction execution unit is used to synchronously display the voice data and the text block according to the audio-visual synchronization display instruction.

[0028] This application provides an electronic device, including: Processor; and A memory for storing a program for implementing the method described in any of the preceding methods, wherein the device is powered on and the program of the method is executed by the processor.

[0029] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the various methods described above.

[0030] This application also provides a computer program product including instructions that, when run on a computer, cause the computer to perform the various methods described above.

[0031] Compared with the prior art, this application has the following advantages: The information processing system provided in this application embodiment generates a text block stream on the server side; assigns a block identifier to each text block; generates an audio-visual synchronization display instruction based on the block identifier, the text block, and the corresponding audio data; sends the audio-visual synchronization display instruction to the client; receives the audio-visual synchronization display instruction through the client; and synchronously displays the audio data and the text block according to the audio-visual synchronization display instruction. This processing method enables real-time interception of stream-generated text content (text blocks) on the server side and dynamically "packages" it into an audio-visual synchronization display instruction containing a block identifier (audio-visual synchronization identifier, synchronization signaling), achieving streaming synchronization based on server-side "interception-packaging". This approach handles the complex audio-visual synchronization logic on the server side, greatly simplifying client-side implementation. Because the synchronization signaling is generated in the cloud and bound to text content blocks, its synchronization accuracy is far higher than client-side parsing. This provides the client with more accurate signaling needed for precise synchronization, achieving near-zero error and enabling a true streaming experience. Users don't need to wait; content is presented instantly. This reduces streaming output latency and improves audio-visual synchronization accuracy, providing users with an immersive and smooth reading and listening experience in scenarios such as long text reading and Q&A. Since this approach simplifies client-side logic, allowing it to focus solely on rendering and playback without needing to worry about complex synchronization state machines, it effectively reduces client-side implementation complexity and performance overhead. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of device interaction of an embodiment of the information processing system provided in this application; Figure 2 This is a flowchart illustrating an embodiment of the information processing method provided in this application; Figure 3 This is a schematic flowchart illustrating a specific embodiment of the information processing method provided in this application; Figure 4 This is another specific flowchart illustrating an embodiment of the information processing method provided in this application; Figure 5 This is another specific flowchart illustrating an embodiment of the information processing method provided in this application. Detailed Implementation

[0033] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0034] This application provides information processing systems, methods, and apparatuses, as well as electronic devices. The various solutions are described in detail below in each embodiment.

[0035] First Embodiment Please refer to Figure 1 This is a schematic diagram of device interaction for the information processing system of this application. In this embodiment, the information processing system includes a server and a client. The server is used to generate a text block stream; assign block identifiers to the text blocks; generate an audio-visual synchronization display instruction based on the block identifiers, the text blocks, and the corresponding audio data; and send the audio-visual synchronization display instruction to the client. Correspondingly, the client is used to receive the audio-visual synchronization display instruction; and synchronously display the audio data and the text blocks according to the audio-visual synchronization display instruction.

[0036] In information processing systems, the server provides the client with content generated in a streaming manner (a stream of text blocks). The client then displays each content block synchronously in both on-screen text and audio playback formats. When generating text content, the server does not wait for all the text to be generated before returning; instead, it pushes the text blocks to the client in real-time, word by word or sentence by sentence, in a "stream" format, such as "The capital of China is," "Beijing," "A city with a long history."

[0037] During the generation of the text block stream, the server can intercept each text block in real time. After intercepting a text block, it obtains the corresponding speech data. In practice, a text-to-speech (TTS) service can be used to generate the speech data corresponding to the text block.

[0038] After the server intercepts a text block and obtains the corresponding audio data, it performs audio-visual synchronization processing on the text block and its corresponding audio data, thus generating an audio-visual synchronization display instruction. Both the client and server adhere to the "Multimodal Data Synchronization Display Protocol," a predefined communication agreement between the server and client. The audio-visual synchronization display instruction includes not only the text block and its corresponding audio data but also a unique block identifier assigned to the text block. This block identifier is used to synchronize text highlighting and audio playback. The server provides globally unique and ordered synchronization points for each text block within the same macro-synchronization task. This processing method implements an "interception-packaging" mechanism, dynamically converting the generated raw, stateless text block stream into an audio-visual synchronization display instruction rich in synchronization signaling (block identifier) ​​that conforms to the "Multimodal Data Synchronization Display Protocol."

[0039] In practice, the block identifier assigned to a text block can be either a timestamp or a sequence number. In one example, the block identifier is a sequence number; the server specifically obtains the assigned block identifiers and determines the block identifier assigned to the text block based on these assigned block identifiers. This approach not only avoids assigning the same block identifier to different text blocks but also reduces the amount of data in the block identifier; therefore, it can effectively improve the accuracy and efficiency of streaming audio-visual synchronization.

[0040] The client is a multimodal device with screen display and voice playback capabilities, such as smartphones, tablets, and other smart communication devices, or smart wearable devices like AI glasses. The client parses audio-visual synchronization display instructions according to the "Multimodal Data Synchronization Display Protocol." Based on the block identifiers included in the instructions, on a multimodal client device with a screen, when the voice broadcasts a word or sentence, the corresponding text on the screen is simultaneously highlighted, achieving streaming synchronous highlighting of voice broadcast and screen text—that is, streaming audio-visual synchronization.

[0041] In one example, the audio-visual synchronization command can be a standalone command that includes a block identifier, a text block, and audio data. The client parses these three pieces of information from the standalone command, plays the audio data corresponding to each text block in the order they were generated, and highlights the text blocks synchronously.

[0042] In another example, the audio-visual synchronization display instructions include text block display instructions and voice playback instructions. Generating the audio-visual synchronization display instructions based on the block identifier, the text block, and the corresponding voice data can be achieved as follows: Generate a text block display instruction based on the block identifier and the text block; generate a voice playback instruction based on the block identifier and the corresponding voice data. The server sends the text block display instructions and voice playback instructions to the client. For example, a text block display instruction (e.g., Render) includes a block identifier (e.g., a sequence number) and the current text block content ("Beijing,"); a voice playback instruction (e.g., Speak) includes a block identifier and the voice data of the current text block. This processing method implements an "interception-packaging" mechanism, dynamically converting the generated raw, stateless text block stream into text block display instructions and voice playback instructions rich in synchronization signaling (block identifiers) that conform to the "multimodal data synchronization display protocol." Both instructions carry block identifiers, forming a strong synchronization relationship. The client parses text display instructions and voice broadcast instructions according to the "Multimodal Data Synchronization Display Protocol." Based on the block identifiers included in the instructions, it retrieves the text display instructions and voice broadcast instructions corresponding to the same text block and executes these two instructions. On multimodal client devices with screens, when a word or sentence is broadcast, the corresponding text on the screen is simultaneously highlighted, achieving streaming synchronous highlighting of voice broadcast and screen text, i.e., streaming audio-visual synchronization.

[0043] This processing method not only allows for the playback of audio data corresponding to each text block in the order of their generation, based on the block identifier, and the simultaneous highlighting of the text blocks, but also enables the client to directly execute text block display commands and audio playback commands. This avoids the client having to parse and generate separate audio-visual synchronization display commands from a single command. Therefore, it effectively improves the client's synchronization efficiency, further reduces streaming output latency, and provides users with an immersive and smooth reading and listening experience in scenarios such as long text reading and Q&A.

[0044] In one example, the server is also used to obtain text display attributes; specifically, it is used to generate text block display instructions based on the block identifier, the text display attributes, and the text block. This approach allows for server-side control of text block display attributes, such as highlight color, animation effects, and font color, without requiring modifications to the client-side application logic; therefore, it effectively improves the flexibility of adjusting text display methods.

[0045] In practice, obtaining text display attributes can be achieved by retrieving the text display attributes corresponding to the text block. This approach allows for control over the display attributes of text blocks with different content types.

[0046] In one example, the server is also used to obtain the task identifier for processing the text block stream; and specifically, to generate audio-visual synchronization display instructions based on the task identifier, the block identifier, the text block, and the corresponding audio data. The task identifier indicates that a set of audio-visual synchronization display instructions (instruction stream) corresponding to the text block stream belongs to the same macro-synchronization task, providing a globally unique and ordered synchronization marker for each text block. This set of (task identifier, block identifier) ​​together constitutes a simple, unambiguous, and uninterpretable synchronization signaling. It completely eliminates the need for the client to perform timestamp comparison, content matching, or progress monitoring. The implementation difficulty of the client is reduced from O(N) (requiring the processing of N word matching) to O(1).

[0047] Information processing systems include, but are not limited to: large language model processing systems, search systems, recommendation systems, question-answering systems, etc. For example, a large language model (LLM) processing system, where the server deploys a large language model (LLM), which generates content in a streaming manner based on prompts (i.e., outputs a stream of text blocks). A search system, where the server searches based on search terms and generates search results in a streaming manner (i.e., outputs a stream of search result text blocks). A recommendation system, where the server generates recommended content in a streaming manner (i.e., outputs a stream of recommended content text blocks). A question-answering system, where the server generates answers to questions in a streaming manner (i.e., outputs a stream of answer content text blocks).

[0048] In one example, the information processing system is an AI agent system, with the AI ​​agent deployed on the server side. An AI agent is a system that autonomously executes tasks by designing workflows using available tools. It understands needs, plans goals, and executes tasks through a large language model, possessing the ability to autonomously understand, perceive, plan, remember, and use tools. It can automatically complete complex tasks, and its core characteristics are proactive thinking and cross-tool operation capabilities, distinguishing it from traditional AI systems that rely on preset instructions. The system typically consists of perception, planning, memory, and tool usage modules, enabling automated processing of complex tasks such as cross-application operations and data analysis. For example, it can understand a user's intent based on a single command and execute the retrieval and cancellation of a renewal function.

[0049] The server-side AI agent acquires demand information; based on the demand information, it invokes at least one tool; based on at least one piece of content returned by the at least one tool, it invokes a large language model; the large language model generates a comprehensive content text block stream based on the at least one piece of content. The demand information is the input data for the AI ​​agent, which processes the demand information and outputs the processing result.

[0050] In practice, users can input their needs into the client, and the client submits a service request to the server. The AI ​​agent on the server calls relevant tools based on the user's input to obtain the tool's response to the user's needs (service content). The service content returned by these tools is then submitted to the large language model, which generates comprehensive service content and returns it to the client. The client plays the audio and simultaneously highlights (e.g., displays) the corresponding text blocks.

[0051] For example, a user might say to the AI ​​glasses (client-side), "Help me plan a trip to Hangzhou next weekend. My budget is 5000 yuan. I'll be traveling by high-speed rail, staying near West Lake. Please check the weather and recommend some authentic, highly-rated restaurants." The AI ​​glasses then treat this information as a request and send it to the AI ​​agent's server. The AI ​​agent will open a browser to check the availability of high-speed rail tickets using a train ticket purchasing tool (tool 1); it will access an online travel platform (tool 2) to filter hotels that match the user's budget and location; it will call a weather API (tool 3) to tell the user whether it will be sunny or rainy in Hangzhou this weekend; it will go to a local services website (tool 4) to find the top-rated Hangzhou-style restaurants; finally, it will send the service information returned by these tools to a large language model, which will generate a well-organized itinerary and send it to the AI ​​glasses. The AI ​​glasses will then display the itinerary for the user to view.

[0052] In practice, the client can also automatically sense the surrounding environment information and submit it to the server. The server's AI agent calls at least one tool based on this environmental information; and calls a large language model based on at least one piece of content returned by the at least one tool; the large language model generates a comprehensive content text block stream based on the at least one piece of content.

[0053] In one example, the AI ​​agent determines a target tool from the at least one set of tools; obtains the target tool's interface template information; and generates a synchronized audio-visual display instruction based on the block identifier, the text block, the corresponding voice data, and the target tool's interface template information. In a specific implementation, a text block display instruction can be generated based on the block identifier, the target tool's interface template information, and the text block. The AI ​​agent invokes multiple tools and needs to determine the target tool, using its interface as the client's interface for displaying comprehensive content. The interface template information can be the interface template itself, or a Uniform Resource Locator (URI) for the interface template, etc. This approach allows the client to display a text block of comprehensive content on the interface of one of the multiple tools invoked by the AI ​​agent (the target tool) based on the interface template information in the synchronized audio-visual display instruction. For example, in the above example, the AI ​​glasses display a travel itinerary on the interface of a lifestyle service website, but not on a weather interface; therefore, it effectively improves the user experience.

[0054] The target tool can be a tool with a higher display priority among multiple tools. In specific implementation, the display priority of the tools can be obtained; based on the display priority of the tools, the target tool can be determined from the at least one tool. Furthermore, the target tool can also be a tool pre-defined according to actual needs, and so on.

[0055] In one example, determining the target tool from the at least one tool can be achieved as follows: obtaining the second text synchronization display mode of the tool; determining the target tool from tools whose second text synchronization display mode is synchronized. Among multiple tools invoked by the AI ​​agent, if the second text synchronization display mode of all or some tools is synchronized, the target tool can be selected based on the display priority of these tools.

[0056] In practice, the tool can return interface instructions (such as Render) to the agent, including a second text-synchronized display mode. The agent retrieves the tool's second text-synchronized display mode from the interface instructions returned by the tool. After obtaining the various information included in the interface instructions, the AI ​​agent can delete the interface instructions. Alternatively, the AI ​​agent can pre-obtain the second text-synchronized display modes of various available tools.

[0057] In specific implementation, the interface instructions may also include at least one of the following information: interface template information and tool task identifier. Based on the block identifier, the text block, the corresponding voice data, and the interface template information of the target tool, an audio-visual synchronized display instruction can be generated. This can be achieved in the following way: Based on the tool task identifier, the block identifier, the text block, the corresponding voice data, and the interface template information of the target tool, an audio-visual synchronized display instruction can be generated. For example, a text block display instruction (such as Render) includes: tool task identifier, block identifier (such as serial number), interface template URI, and current text block content ("Beijing,"); a voice broadcast instruction (such as Speak) includes: task identifier, block identifier, and voice data of the current text block. Using this processing method, the AI ​​agent can know which tool and which interface to highlight the text block corresponding to the voice.

[0058] In one example, the AI ​​agent sets a first text-synchronized display mode for a demand processing task, sets the target interface template information of the demand processing task as the interface template information of the target tool, and obtains the first text-synchronized display mode and the target interface template information before generating the audio-visual synchronized display instruction. If the first text-synchronized display mode is synchronized, a text block display instruction is generated based on the block identifier, the target interface template information, and the text block. A voice broadcast instruction is generated based on the block identifier and the voice data corresponding to the text block.

[0059] The demand processing task refers to the task by which the AI ​​agent processes specific demand information. The first text synchronization display mode includes synchronous or asynchronous modes. When the first text synchronization display mode is synchronous, the client uses a synchronized audio-visual display method, highlighting the corresponding text block synchronously while playing audio. When the first text synchronization display mode is asynchronous, the client does not need to highlight the corresponding text block synchronously while playing audio.

[0060] In one example, determining the target tool from the at least one tool can be achieved as follows: obtaining the second text synchronization display mode of the tool; determining the target tool from tools whose second text synchronization display mode is synchronized. Setting the first text synchronization display mode for the requirement processing task can be achieved as follows: if there is a tool whose second text synchronization display mode is synchronized, then setting the first text synchronization display mode to synchronized.

[0061] This approach allows for the maintenance of a dedicated state machine within the task session. The AI ​​agent temporarily stores the "synchronization intent" (first text synchronization display mode) determined in the first phase as an active state (first text synchronization display mode is synchronized) and metadata (target interface template information). This state machine becomes the core bridge connecting two asynchronous, heterogeneous event streams. When the large model content stream arrives, a content processor does not simply forward the data but acts as a converter: it reads the highlight state (first text synchronization display mode) and determines whether the AI ​​agent is in "highlight mode." If so, it performs dynamic packaging, "upgrading" the stateless text block into structured instruction pairs carrying synchronization signaling. This completely decouples the "triggering of synchronization intent" from the "carrying of synchronization content" (separation of triggering and processing), achieving "decoupled, state machine-based streaming data transformation." This approach allows the AI ​​agent to perform stateful and intelligent transformation of the data stream at different stages and in response to different events, representing a flexible and scalable stream processing paradigm.

[0062] In one example, the AI ​​agent detects whether the client is displaying an interface on a dedicated screen other than the at least one tool; if the detection result is yes, the second text synchronization display mode is set to desynchronized. This approach shifts the resolution of interaction conflicts from "post-event remediation" to "pre-event decision-making," ensuring that the display of the client's dedicated screen is not affected, fundamentally eliminating the possibility of client UI chaos.

[0063] In practical implementation, during the tool execution result orchestration phase, the AI ​​agent introduces a protocol preprocessor. Instead of passively waiting for a signal, it performs the following processes: 1. Active scanning and identification: Parallel scanning of all instruction sets returned by the tools to identify all potential protocol requests attempting to seize the "audio-visual synchronization" channel (such as weather APIs, train ticket purchasing tools, online travel platforms, etc.); 2. Priority arbitration: Based on predefined priorities, a "sole winner" arbitration is conducted among multiple protocol requests, ensuring that at any given time, only one interaction intent can dominate audio-visual synchronization; 3. Screen resource conflict detection: Furthermore, the AI ​​agent checks for independent application UI instructions incompatible with the winning protocol (such as a full-screen weather card). If such a "hard conflict" exists, the AI ​​agent actively suppresses all highlighted protocols to ensure that the core application process is not interrupted.

[0064] In one example, the AI ​​agent detects at least one tool that is running in conflict; if the at least one tool is running in conflict, it sends information providing tool running conflict information to the client.

[0065] As can be seen from the above embodiments, the information processing system provided in this application generates a text block stream on the server side; assigns a block identifier to each text block; generates an audio-visual synchronization display instruction based on the block identifier, the text block, and the corresponding voice data; sends the audio-visual synchronization display instruction to the client; receives the audio-visual synchronization display instruction through the client; and synchronously displays the voice data and the text block according to the audio-visual synchronization display instruction. This processing method enables real-time interception of stream-generated text content (text blocks) on the server side and dynamically "packages" it into an audio-visual synchronization display instruction containing a block identifier (audio-visual synchronization identifier, synchronization signaling), achieving streaming synchronization based on server-side "interception-packaging". This approach handles the complex audio-visual synchronization logic on the server side, greatly simplifying client-side implementation. Because the synchronization signaling is generated in the cloud and bound to text content blocks, its synchronization accuracy is far higher than client-side parsing. This provides the client with more accurate signaling needed for precise synchronization, achieving near-zero error and enabling a true streaming experience. Users don't need to wait; content is presented instantly. This reduces streaming output latency and improves audio-visual synchronization accuracy, providing users with an immersive and smooth reading and listening experience in scenarios such as long text reading and Q&A. Since this approach simplifies client-side logic, allowing it to focus solely on rendering and playback without needing to worry about complex synchronization state machines, it effectively reduces client-side implementation complexity and performance overhead.

[0066] Second Embodiment In the above embodiments, an information processing system is provided. Correspondingly, this application also provides an information processing method for a server. This method corresponds to the embodiments of the above system. Since the method embodiments are basically similar to the system embodiments, they are described simply, and relevant details can be found in the descriptions of the system embodiments. The method embodiments described below are merely illustrative.

[0067] Please refer to Figure 2 This is a flowchart illustrating the information processing method of this application. This application also provides an information processing method, which may include the following steps: Step S201: Generate a text block stream.

[0068] Step S203: Assign a block identifier to the text block.

[0069] In one example, step S203 can be implemented as follows: obtain the allocated block identifier; determine the block identifier assigned to the text block based on the allocated block identifier.

[0070] Step S205: Generate an audio-visual synchronized display instruction based on the block identifier, the text block, and the voice data corresponding to the text block.

[0071] In one example, step S205 can be implemented as follows: generate a text block display instruction based on the block identifier and the text block; generate a voice broadcast instruction based on the block identifier and the voice data corresponding to the text block.

[0072] Step S207: Send the audio-visual synchronization display command to the device.

[0073] In one example, step S201 can be implemented as follows: the AI ​​agent invokes at least one tool based on the demand information; the AI ​​agent invokes a large language model based on at least one piece of content returned by the at least one tool; the large language model generates a comprehensive content text block stream based on the at least one piece of content.

[0074] Please refer to Figure 3 This is a schematic diagram illustrating the specific flow of the information processing method of this application. In one example, step S201 may include the following sub-steps: Step S2011: The AI ​​agent invokes at least one tool based on the required information.

[0075] Step S2013: Based on at least one piece of content returned by the at least one tool, invoke the large language model; the large language model generates a comprehensive content text block stream based on the at least one piece of content.

[0076] Accordingly, the method provided in this application embodiment may further include the following steps: Step S301: Determine the target tool from the at least one tool; Step S303: Obtain the interface template information of the target tool; Step S205 can be implemented as follows: Generate an audio-visual synchronized display instruction based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool.

[0077] Please refer to Figure 4 This is a schematic diagram illustrating the specific flow of the information processing method of this application. In one example, the method provided in this embodiment may further include the following steps: Step S401: Set the first text synchronization display mode for the requirement processing task, and set the target interface template information of the requirement processing task to the interface template information of the target tool; the first text synchronization display mode includes synchronization or non-synchronization.

[0078] Step S403: Obtain the first text synchronization display mode and the target interface template information.

[0079] Accordingly, step S205 may include the following sub-steps: Step S2051: If the first text synchronous display mode is synchronous, then generate a text block display instruction based on the block identifier, the target interface template information, and the text block; Step S2053: Generate a voice broadcast command based on the block identifier and the voice data corresponding to the text block.

[0080] Please refer to Figure 5 This is a schematic diagram of the specific flow of the information processing method of this application. In one example, step S301 can be implemented as follows: obtain the second text synchronization display mode of the tool; determine the target tool from the tools whose second text synchronization display mode is synchronized; step S401 can be implemented as follows: if there is a tool whose second text synchronization display mode is synchronized, then set the first text synchronization display mode to synchronized, and set the target interface template information of the task to be processed as the interface template information of the target tool.

[0081] In one example, step S301 can be implemented as follows: determining the target tool from the at least one tool can be implemented as follows: determining the target tool from the at least one tool according to the priority of the tool.

[0082] In one example, step S301 can be implemented as follows: obtain the second text synchronization display mode of the tool; determine the target tool from the tools whose second text synchronization display mode is synchronized.

[0083] In one example, obtaining the second text synchronization display mode of the tool can be achieved by obtaining the interface instructions returned by the tool, the interface instructions including the second text synchronization display mode.

[0084] In one example, the interface instructions also include a tool task identifier; step S205 can be implemented as follows: generate an audio-visual synchronized display instruction based on the tool task identifier, the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool.

[0085] In one example, the method provided in this application embodiment may further include the following step: deleting the interface instructions.

[0086] In one example, the method provided in this application embodiment may further include the following steps: detecting whether the device displays an interface of a dedicated screen other than the at least one tool; if the detection result is yes, then setting the first text synchronization display mode to asynchronous.

[0087] In one example, the method provided in this application embodiment may further include the following steps: if there is a conflicting tool among the at least one tool, then send information about the tool conflict to the device.

[0088] In one example, the method provided in this application embodiment may further include the following steps: obtaining text display attributes; step S205 may be implemented as follows: generating an audio-visual synchronized display instruction based on the block identifier, the text display attributes, the text block, and the voice data corresponding to the text block. In specific implementation, a text block display instruction may be generated based on the block identifier, the text display attributes, and the text block.

[0089] In one example, obtaining the text display attributes can be achieved by obtaining the text display attributes corresponding to the text block.

[0090] In one example, the method provided in this application embodiment may further include the following steps: obtaining a task identifier for processing the text block stream; step S205 may be implemented in the following manner: generating an audio-visual synchronized display instruction based on the task identifier, the block identifier, the text block, and the voice data corresponding to the text block.

[0091] Third Embodiment In the above embodiments, an information processing method is provided. Correspondingly, this application also provides an information processing apparatus. This apparatus corresponds to the embodiments of the method described above. Since the apparatus embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant details can be found in the description of the method embodiments. The apparatus embodiments described below are merely illustrative.

[0092] This application also provides an information processing apparatus, comprising: a text block stream generation unit, a block identifier allocation unit, an instruction generation unit, and an instruction sending unit. The text block stream generation unit is used to generate a text block stream; the block identifier allocation unit is used to allocate block identifiers to the text blocks; the instruction generation unit is used to generate an audio-visual synchronized display instruction based on the block identifiers, the text blocks, and corresponding voice data; and the instruction sending unit is used to send the audio-visual synchronized display instruction to a device.

[0093] In one example, the instruction generation unit is specifically used to generate a text block display instruction based on the block identifier and the text block; and to generate a voice broadcast instruction based on the block identifier and the voice data corresponding to the text block.

[0094] In one example, the text block stream generation unit is specifically used for the AI ​​agent to invoke at least one tool based on demand information; to invoke a large language model based on at least one piece of content returned by the at least one tool; and for the large language model to generate a comprehensive content text block stream based on the at least one piece of content.

[0095] In one example, the apparatus provided in this application embodiment may further include: a target tool determination unit, configured to determine a target tool from the at least one tool; an interface template information acquisition unit, configured to acquire interface template information of the target tool; and an instruction generation unit, specifically configured to generate an audio-visual synchronized display instruction based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool.

[0096] In one example, the apparatus provided in this application embodiment may further include: a state setting unit, configured to set a first text synchronization display mode for a demand processing task, and set the target interface template information of the demand processing task as the interface template information of the target tool; the first text synchronization display mode includes synchronization or desynchronization; a state acquisition unit, configured to acquire the first text synchronization display mode and the target interface template information; and an instruction generation unit, specifically configured to, if the first text synchronization display mode is synchronized, generate a text block display instruction based on the block identifier, the target interface template information, and the text block; and generate a voice broadcast instruction based on the block identifier and the voice data corresponding to the text block.

[0097] In one example, the target tool determination unit is specifically used to obtain the second text synchronization display mode of the tool; determine the target tool from the tools whose second text synchronization display mode is synchronized; and the state setting unit is specifically used to set the first text synchronization display mode to synchronize if there is a tool whose second text synchronization display mode is synchronized.

[0098] In one example, the target tool determination unit is specifically configured to determine a target tool from the at least one tool based on the priority of the tools.

[0099] In one example, the target tool determination unit is specifically used to obtain the second text synchronization display mode of the tool; and determine the target tool from the tools whose second text synchronization display mode is synchronized.

[0100] In one example, obtaining the second text synchronization display mode of the tool includes: obtaining the interface instructions returned by the tool, the interface instructions including the second text synchronization display mode.

[0101] In one example, the interface instructions further include a tool task identifier; the instruction generation unit is specifically used to generate audio-visual synchronized display instructions based on the tool task identifier, the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool.

[0102] In one example, the apparatus provided in this application embodiment may further include: deleting the interface instruction.

[0103] In one example, the apparatus provided in this application embodiment may further include: detecting whether the device displays an interface of a dedicated screen other than the at least one tool; if the detection result is yes, then setting the first text synchronization display mode to asynchronous.

[0104] In one example, the apparatus provided in this application embodiment may further include: if there is a conflicting tool among the at least one tool, then sending information about the tool conflict to the apparatus.

[0105] In one example, the apparatus provided in this application embodiment may further include: acquiring text display attributes; and an instruction generation unit, specifically configured to generate an audio-visual synchronized display instruction based on the block identifier, the text display attributes, the text block, and the voice data corresponding to the text block.

[0106] In one example, obtaining the text display attributes includes: obtaining the text display attributes corresponding to the text block.

[0107] In one example, assigning a block identifier to the text block includes: obtaining an already assigned block identifier; and determining a block identifier to be assigned to the text block based on the already assigned block identifier.

[0108] In one example, the apparatus provided in this application embodiment may further include: acquiring a task identifier for processing the text block stream; and an instruction generation unit, specifically configured to generate an audio-visual synchronized display instruction based on the task identifier, the block identifier, the text block, and the voice data corresponding to the text block.

[0109] Fourth embodiment In the above embodiments, an information processing system is provided. Correspondingly, this application also provides an information processing method for a client. This method corresponds to the embodiments of the above system. Since the method embodiments are basically similar to the system embodiments, they are described simply, and relevant details can be found in the descriptions of the system embodiments. The method embodiments described below are merely illustrative.

[0110] This application also provides an information processing method, which may include the following steps: Step 1: Receive audio-visual synchronization display instruction; the audio-visual synchronization display instruction includes a text block, the corresponding audio data and block identifier, and the text block belongs to a text block stream.

[0111] Step 2: According to the audio-visual synchronization display instruction, simultaneously display the audio data and the text block.

[0112] In one example, the text block belongs to a comprehensive content text block stream generated by the AI ​​agent based on demand information; the audio-visual synchronization display instruction includes the interface template information of the target tool; displaying the text block includes: displaying the text block in the interface of the target tool according to the interface template information.

[0113] Fifth embodiment In the above embodiments, an information processing method is provided. Correspondingly, this application also provides an information processing apparatus. This apparatus corresponds to the embodiments of the method described above. Since the apparatus embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant details can be found in the description of the method embodiments. The apparatus embodiments described below are merely illustrative.

[0114] This application also provides an information processing apparatus, comprising: an instruction receiving unit for receiving an audio-visual synchronization display instruction; the audio-visual synchronization display instruction includes a text block, voice data corresponding to the text block, and a block identifier, wherein the text block belongs to a text block stream; and an instruction execution unit for synchronously displaying the voice data and the text block according to the audio-visual synchronization display instruction.

[0115] In one example, the text block belongs to a comprehensive content text block stream generated by the AI ​​agent based on demand information; the audio-visual synchronization display instruction includes the interface template information of the target tool; displaying the text block includes: displaying the text block in the interface of the target tool according to the interface template information.

[0116] Sixth Embodiment In the above embodiments, a method is provided. Correspondingly, this application also provides an electronic device. This device corresponds to the embodiment of the method described above. Since the device embodiment is basically similar to the method embodiment, it is described simply, and relevant details can be found in the description of the method embodiment. The device embodiment described below is merely illustrative.

[0117] The electronic device of this embodiment includes: a memory and a processor; the memory is used to store a program for implementing the information processing method, and the device is powered on and runs the program of the information processing method through the processor.

[0118] In practice, the electronic device can be any device capable of implementing the information processing method.

[0119] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0120] In specific implementations, the electronic device may further include one or more of the following components: a power supply component, an input / output (I / O) interface, and a communication component. The power supply component provides power to various components of the electronic device. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device. The I / O interface provides an interface between the processor 503 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. The communication component is configured to facilitate wired or wireless communication between the electronic device and user devices (such as smartphones, tablets, etc.).

[0121] Seventh Embodiment This application also provides a computer-readable storage medium. Since the embodiments of the computer-readable storage medium are substantially similar to the method embodiments, the description is relatively simple; relevant details can be found in the description of the method embodiments. The computer-readable storage medium embodiments described below are merely illustrative.

[0122] In this embodiment, a non-transitory computer-readable storage medium including instructions is provided, such as a memory including instructions, which can be executed by a processor of an electronic device to complete the information processing method provided by this disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0123] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0124] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0125] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0126] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0127] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0128] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. An information processing system, characterized in that, include: The server is used to generate a text block stream; assign block identifiers to the text blocks; generate an audio-visual synchronization display instruction based on the block identifier, the text block, and the corresponding audio data; and send the audio-visual synchronization display instruction to the client. The client is used to receive the audio-visual synchronization display instruction and to synchronously display the voice data and the text block according to the audio-visual synchronization display instruction.

2. An information processing method, characterized in that, include: Generate a stream of text blocks; Assign a block identifier to the text block; Based on the block identifier, the text block, and the voice data corresponding to the text block, generate an audio-visual synchronized display instruction; Send the audio-visual synchronization display command to the device.

3. The method according to claim 2, characterized in that, The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, and the voice data corresponding to the text block includes: Generate a text block display instruction based on the block identifier and the text block; Based on the block identifier and the voice data corresponding to the text block, a voice broadcast command is generated.

4. The method according to claim 2, characterized in that, The generated text block stream includes: The AI ​​agent invokes at least one tool based on the required information; Invoke the large language model based on at least one piece of content returned by the at least one tool; The large language model generates a comprehensive content text block stream based on at least one of the contents.

5. The method according to claim 4, characterized in that, Also includes: Determine the target tool from the at least one tool; Obtain the interface template information of the target tool; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, and the voice data corresponding to the text block includes: Based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool, an audio-visual synchronized display instruction is generated.

6. The method according to claim 5, characterized in that, Also includes: Set the first text synchronization display mode for the requirement processing task, and set the target interface template information of the requirement processing task as the interface template information of the target tool. The first text synchronization display mode includes synchronous or asynchronous modes; Before generating the audio-visual synchronization display instruction, the following is also included: Obtain the first text synchronization display mode and the target interface template information; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, the voice data corresponding to the text block, and the interface template information of the target tool includes: If the first text synchronization display mode is synchronized, then a text block display instruction is generated based on the block identifier, the target interface template information, and the text block; Based on the block identifier and the voice data corresponding to the text block, a voice broadcast command is generated.

7. The method according to claim 6, characterized in that, The step of determining the target tool from the at least one tool includes: Obtain the second text synchronization display mode of the tool; From the tools whose second text synchronization display mode is synchronized, identify the target tool; The first text synchronization display mode for setting up the demand processing task includes: If a tool exists with a second text synchronization display mode set to synchronization, then the first text synchronization display mode will be set to synchronization.

8. The method according to claim 6, characterized in that, Also includes: Detect whether the device displays an interface on a dedicated screen other than the at least one tool; If the detection result is yes, then the first text synchronous display mode will be set to asynchronous.

9. The method according to claim 2, characterized in that, Also includes: Get text display properties; The step of generating audio-visual synchronized display instructions based on the block identifier, the text block, and the voice data corresponding to the text block includes: Based on the block identifier, the text display attribute, the text block, and the voice data corresponding to the text block, an audio-visual synchronized display instruction is generated.

10. An information processing method, characterized in that, include: Receive audio-visual synchronization display instruction; the audio-visual synchronization display instruction includes a text block, voice data corresponding to the text block, and a block identifier, wherein the text block belongs to a text block stream; According to the audio-visual synchronization display instruction, the voice data and the text block are displayed synchronously.

11. The method according to claim 10, characterized in that, The text block belongs to a comprehensive content text block stream generated by the artificial intelligence entity based on the demand information; The audio-visual synchronization display command includes the interface template information of the target tool; Displaying the text block includes: Based on the interface template information, the text block is displayed in the interface of the target tool.

12. An information processing device, characterized in that, include: The text block stream generation unit is used to generate text block streams; A block identifier allocation unit is used to assign a block identifier to the text block; The instruction generation unit is used to generate an audio-visual synchronized display instruction based on the block identifier, the text block, and the voice data corresponding to the text block; The instruction sending unit is used to send the audio-visual synchronization display instruction to the device.

13. An information processing device, characterized in that, include: The instruction receiving unit is used to receive an audio-visual synchronization display instruction; the audio-visual synchronization display instruction includes a text block, voice data corresponding to the text block, and a block identifier, wherein the text block belongs to a text block stream; The instruction execution unit is used to synchronously display the voice data and the text block according to the audio-visual synchronization display instruction.

14. An electronic device, characterized in that, include: processor; as well as A memory for storing a program for implementing the information processing method according to any one of claims 2 to 11, wherein the device is powered on and the program of the method is executed by the processor.

Images