Model streaming output method and electronic device

Abstract

The application provides a model streaming output method and an electronic device, applied to the field of artificial intelligence technology. The method comprises the following steps: in response to a plurality of models receiving a target question sent by a user end, creating a corresponding event loop for each model in an asynchronous thread; executing an asynchronous streaming output task of the corresponding model based on the event loop in the asynchronous thread to generate streaming data for the target question; writing the streaming data output by each asynchronous thread into a message queue, each streaming data having different first priorities based on the importance of executing the asynchronous streaming output task of the corresponding model; in response to the streaming data in the message queue reaching a preset capacity threshold, determining target streaming data with a first priority lower than a preset priority threshold from the message queue, and merging and processing at least two word units in the target streaming data; acquiring the word units from the message queue after the merging and processing, and returning to the user end in a synchronous streaming output manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a model streaming output method and an electronic device. BACKGROUND

[0002] With the wide deployment of large language models in practical applications, more and more scenarios need to call multiple models simultaneously to handle complex tasks to improve response quality and processing efficiency. However, when multiple models concurrently execute streaming output tasks, due to the differences in processing speed and output frequency of each model, the overall streaming output is prone to instability, affecting the receiving experience of the user end and the overall performance of the system. SUMMARY

[0003] In view of the above problems, the present application provides a model streaming output method and an electronic device.

[0004] According to a first aspect of the present application, a model streaming output method is provided, comprising: in response to multiple models receiving a target question sent by a user end, creating a corresponding event loop in an asynchronous thread for each model; executing an asynchronous streaming output task of the corresponding model based on the event loop in the asynchronous thread, and generating streaming data for the target question; writing the streaming data output by each asynchronous thread into a message queue, wherein each streaming data has a different first priority based on the importance of the corresponding model executing the asynchronous streaming output task; in response to the streaming data in the message queue reaching a preset capacity threshold, determining target streaming data with a first priority lower than a preset priority threshold from the message queue, and merging and processing at least two word units in the target streaming data; sequentially obtaining the word units from the message queue after the word unit merging and processing, and sequentially returning to the user end in a synchronous streaming output manner.

[0005] According to a second aspect of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more computer programs; and one or more processors executing one or more computer programs to implement the above-mentioned model streaming output method. BRIEF DESCRIPTION OF DRAWINGS

[0006] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0007] Figure 1 An application scenario diagram of a model streaming output method and an electronic device provided by an embodiment of the present application;

[0008] Figure 2 A flowchart of a model streaming output method provided by an embodiment of the present application;

[0009] Figure 3A is one of the scene schematic diagrams of the multi-model question and answer provided by an embodiment of the present application;

[0010] Figure 3B is another one of the scene schematic diagrams of the multi-model question and answer provided by an embodiment of the present application;

[0011] Figure 4 A block diagram of an electronic device suitable for the model streaming output method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0012] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely illustrative and is not intended to limit the scope of the present application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0013] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present application. The terms "include", "comprise" and the like used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0014] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0015] In the case of using expressions similar to "at least one of A, B, and C, etc.", in general, it should be interpreted to include at least one of the items, but not limited to the items (e.g., "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0016] With the rapid development and wide application of large language model technology, a single model has been difficult to meet the diversified needs of complex business scenarios. In an intelligent dialogue platform, usually, only one model instance is not run, but multiple models of different types and complementary functions are run to form a complete intelligent service ecosystem.

[0017] Exemplarily, these models usually include: a main model for core question answering and text generation, responsible for handling the main dialogue needs of users; an auxiliary model for sentiment analysis, style correction, tone adjustment, to ensure that the output content conforms to a specific language style and emotional tendency; a knowledge retrieval model for external knowledge calling and fact checking, to enhance the accuracy and timeliness of the answers; a service model for generating structured responses or performing specific actions.

[0018] In the above application scenario of multi-model collaboration, in order to provide smooth user experience and efficient system performance, the system must meet the following technical requirements: synchronous output is realized to ensure that the user can receive the model response in time; multi-model asynchronous generation is supported, taking into account the different characteristics of different model inference rates, to avoid affecting the overall system efficiency due to the slow processing of a certain model; perfect flow control mode is provided to prevent system overload caused by high-frequency output, and to prevent service interruption caused by network congestion; the output content is ordered, smooth and semantically coherent to maintain good user interaction experience.

[0019] However, the related art often has serious coroutine execution conflicts and resource management problems when processing multi-model streaming output. The existing server needs to use coroutine synchronization function or coroutine asynchronous function separately when implementing streaming output function. When the system tries to use coroutine synchronization and coroutine asynchronous mechanism at the same time, coroutine exception error will be thrown due to execution context conflict, which will cause the multi-model collaboration to fail to proceed normally.

[0020] At the same time, all coroutine tasks share the same event loop for scheduling and execution by default. When the coroutine internal processing logic of a certain model is complex, it will occupy a large amount of processing time of the event loop, causing other model tasks to be blocked, and seriously affecting the concurrent processing capability of the overall system.

[0021] More importantly, the related art generally lacks perfect flow control logic, which will cause system resource depletion and message backlog problems at the producer level, data processing problems and frequent network connection establishment and disconnection problems at the consumer level, network channel congestion problems at the network transmission level, and further exacerbation of system performance deterioration due to frequent data parsing consuming a large amount of computing resources.

[0022] Based on the above technical problems, the related art is difficult to meet the stability, real-time and high efficiency requirements of multi-model streaming output at the same time. To solve the above problems, the embodiment of the present application provides a model streaming output method and electronic equipment.

[0023] Figure 1 The application scenario diagram of the model streaming output method and electronic equipment provided by the embodiment of the present application is shown.

[0024] AsFigure 1 As shown, the application scenario 100 according to the embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105, which can be understood in the embodiment of the present application as a network infrastructure supporting multi-model streaming output data transmission. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc., for ensuring that streaming data can be transmitted in real time and stably between the user end and the server.

[0025] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. In the embodiment of the present application, the first terminal device 101, the second terminal device 102, and the third terminal device 103 serve as the user end, for sending target questions to the server 105 and receiving synchronous streaming output results from multi-model collaborative processing. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples), which can trigger multi-model concurrent processing requirements and present streaming output effects.

[0026] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting progressive presentation of streaming content, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc. For example, these terminal devices have the ability to receive and render streaming data in real time, and can gradually display the integrated output content from multiple models in the manner of a typewriter effect.

[0027] The server 105 can be a server providing multi-model streaming output services, such as an artificial intelligence service platform configured with multiple large language model instances and supporting the function of converting asynchronous streaming output to synchronous streaming output (only as an example). The server 105 serves as a carrier for multi-model collaboration, and can create corresponding event loops for each model in an asynchronous thread in response to the received target question, and implement streaming data aggregation and flow control of each asynchronous thread output through a message queue. The background management server can analyze and process the received user requests and other data, identify the types and quantities of models to be called, dynamically allocate computing resources, and feed back the streaming output results after merging to the terminal devices in a synchronous manner.

[0028] It should be noted that the streaming output method of the model provided in this application embodiment can generally be executed by server 105, where server 105 internally runs multiple model instances, each model instance corresponding to an independent asynchronous thread and event loop, and the technical conversion from asynchronous streaming output to synchronous streaming output is achieved through message queues. The streaming output method of the model provided in this application embodiment can also be executed by a distributed server cluster different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or server 105, where each server node can respectively host different types of model instances, and cross-node streaming data aggregation and collaborative processing are achieved through network communication.

[0029] Optionally, in a distributed deployment scenario, the message queue can be implemented using a distributed message middleware, which supports streaming data transmission and merging processing across server nodes, further improving the system's scalability and fault tolerance.

[0030] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0031] The following will be based on Figure 1 The following describes the streaming output method of the model in the embodiments of this application in detail, based on the described scenario.

[0032] Figure 2 This is a flowchart illustrating a streaming output method for a model provided in an embodiment of this application. Figure 2 As shown, the method may include the following operations.

[0033] Operation S210 responds to multiple models receiving the target question sent by the user terminal by creating a corresponding event loop in an asynchronous thread for each model.

[0034] Operation S220 executes the asynchronous streaming output task of the corresponding model based on the event loop in the asynchronous thread, generating streaming data for the target problem respectively.

[0035] Operation S230 writes the streaming data output by each asynchronous thread into the message queue. Each streaming data has a different first priority based on the importance of executing the asynchronous streaming output task according to the corresponding model.

[0036] Operation S240, in response to the streaming data in the message queue reaching a preset capacity threshold, determines the target streaming data with a first priority lower than the preset priority threshold from the message queue, and merges at least two tokens in the target streaming data.

[0037] Operation S250, sequentially obtains the word units from the message queue after the word unit merging processing, and returns to the user end in a synchronous streaming output manner.

[0038] In operation S210, the plurality of models refer to different function type language model instances running in parallel in the server, which can be understood in the embodiments of the application as a model set with different professional capabilities and processing characteristics, for collaborative processing of the target problem sent by the user end.

[0039] The user end refers to a client device or application program that initiates a query request to the server, which can be understood in the embodiments of the application as a terminal that interacts with the server and receives streaming output results. Illustratively, the user end includes but is not limited to a mobile application client, a web browser, a desktop application, a smart hardware device, etc.

[0040] The target problem refers to the specific query content or task instruction submitted by the user end to the server, which can be understood in the embodiments of the application as input information that triggers the collaborative work of multiple models.

[0041] Illustratively, the target problem includes but is not limited to natural language question answering, text creation tasks, knowledge query requests, sentiment analysis requirements, and multi-round dialogue instructions.

[0042] After receiving the target problem, the server creates a corresponding event loop for each model in an asynchronous thread. The asynchronous thread refers to an execution unit that runs independently of the main thread and can concurrently process tasks without blocking the execution of other threads; the event loop refers to an asynchronous programming task execution body responsible for scheduling and managing events, which can be understood as an independent running environment and task scheduler for each model instance, ensuring that multiple models can execute streaming output tasks in parallel without interfering with each other.

[0043] Illustratively, the asynchronous thread provides an independent running space for each model, avoiding mutual blocking of multiple model tasks in a single thread; the event loop is responsible for creating, scheduling and destroying coroutine tasks in each asynchronous thread, ensuring that the model's streaming output tasks can be executed efficiently and in order. The asynchronous thread and the event loop are used together to form a multi-model concurrent execution, where each asynchronous thread carries an event loop and each event loop manages all asynchronous tasks of the corresponding model.

[0044] It should be noted that in related technologies, multiple models usually share the same event loop, which can easily cause the entire system to be blocked when a model processes a complex task. The embodiments of the present application allocate independent asynchronous threads and event loops for each model, achieving running isolation between models, effectively avoiding the single-point blocking problem, and improving the concurrent processing capability and stability of the system.

[0045] In a feasible implementation, a dedicated asynchronous thread can be created for each model in sequence according to preset model configuration information, and an independent event loop instance can be initialized inside each asynchronous thread, where the model configuration information includes model identification, resource configuration parameters, priority settings and other key attributes.

[0046] In another feasible implementation, a plurality of asynchronous threads can be created in advance, and when a target problem is received, an available thread can be allocated to each model from the thread pool, and an event loop can be dynamically created in the allocated thread to improve the efficiency of thread creation.

[0047] In operation S220, the asynchronous streaming output task refers to a computing task that generates and outputs data content in a non-blocking manner in an asynchronous execution environment, which can be understood in the embodiment of the present application as an asynchronous execution process of reasoning and computing of each model based on the target problem and real-time generation of output results, for realizing continuous generation and immediate transmission of model output content.

[0048] The asynchronous streaming output task continuously generates output content during the execution process, and the output content is in the form of a data stream, that is, streaming data. Streaming data is a sequence of data units with time sequence characteristics generated by the model during the reasoning process, which can be understood as real-time output content generated by the model for the target problem, and is generated in the form of word units, phrases or sentences, for providing continuous and progressive answer content to the user end.

[0049] In a feasible implementation, the reasoning task of the corresponding model can be registered in the event loop of each asynchronous thread, and the reasoning and computing process of the model can be started through coroutine scheduling, and the model generates data units in real time during the reasoning process and encapsulates them into streaming data objects, and the streaming data is transmitted to the subsequent processing module through the callback function.

[0050] In operation S230, the message queue refers to an intermediate cache structure for storing and transmitting data, which can be understood in the embodiment of the present application as a data transmission channel connecting the asynchronous streaming output module and the synchronous streaming output module, for temporarily storing the streaming data generated by each model and managing them in a specific order.

[0051] The first priority refers to the importance of executing the asynchronous streaming output task by the model, which is a level assigned to the streaming data, for determining the processing order and merging strategy of the streaming data in the message queue, and realizing flow control of the output data of multiple models.

[0052] Optionally, the numerical range of the first priority can be set to an integer of 1-10, where the larger the value, the higher the priority, and the smaller the value, the lower the priority.

[0053] Similarly, the importance degree refers to an evaluation index of the task value and urgency degree of each model in processing the target problem, which can be understood as a basis for determining the first priority allocation of the streaming data, and is used to distinguish the processing priority of the output content of different models.

[0054] After each asynchronous thread generates the streaming data, the streaming data needs to be written into the message queue according to the preset data format. During the writing process, the system assigns each piece of streaming data with a corresponding first priority identifier according to the importance degree of each model in executing the asynchronous streaming output task.

[0055] In a feasible implementation, the basic importance degree weight of each model can be set in advance. When the model generates the streaming data, the first priority of the streaming data is calculated according to the basic weight combined with the urgency degree of the current task, and the streaming data object containing the priority information is added to the tail of the message queue.

[0056] In another feasible implementation, the execution state and output quality of each model can be dynamically monitored, and the importance degree evaluation result of the model is adjusted in real time, so as to assign more accurate first priority to the subsequently generated streaming data to adapt to the change of processing requirements in different application scenarios.

[0057] In operation S240, the preset capacity threshold refers to the upper limit value of the number of streaming data allowed to be stored in the message queue, which can be understood as a critical condition for triggering the flow control mechanism in the embodiments of the present application, and is used to prevent the system resources from being exhausted due to too much data accumulation in the message queue.

[0058] Exemplarily, the preset capacity threshold can be set to 100 or 500 pieces of streaming data according to the memory configuration and processing capacity of the server.

[0059] When the streaming data in the message queue reaches the above-mentioned capacity threshold, the system needs to determine the processing strategy through the preset priority threshold. The preset priority threshold refers to the first priority critical value used to filter the streaming data to be processed, which can be understood as a demarcation standard for distinguishing high-priority data and low-priority data.

[0060] Based on the preset priority threshold, the system can identify the target streaming data from the message queue. The target streaming data refers to a set of streaming data whose first priority is lower than the preset priority threshold and is filtered from the message queue, which can be understood as a data object that needs to be processed by merging.

[0061] Among them, the token refers to the smallest semantic unit in natural language processing, which can be understood as the basic data unit output by the model in the embodiments of the present application, including words, characters, punctuation marks and other text component elements.

[0062] The system performs a merging processing operation on the word units in the target stream data. The merging processing refers to the operation process of combining multiple independent word units into a single data unit, which is used to reduce the storage pressure of the message queue while maintaining the integrity of the output content.

[0063] In a feasible implementation, when the amount of stream data in the message queue reaches the preset capacity threshold, the flow control is started, and the target stream data with a relatively low priority is selected by comparing the first priority of each stream data with the preset priority threshold. Subsequently, the system performs merging processing on the word units in the target stream data, and integrates at least two adjacent or related word units.

[0064] In a feasible implementation, the current data amount of the message queue can be continuously monitored, and a warning is triggered when it is detected that the data amount reaches 80% of the preset capacity threshold, and the merging processing operation is immediately performed when it reaches 100%, by grouping and merging the continuous word units in the target stream data according to semantic correlation.

[0065] In operation S250, the process of obtaining word units can adopt the first-in-first-out principle of the queue, or can be sorted according to the first priority of the stream data and then extracted in turn.

[0066] In order to realize a unified user experience, the system can adopt a synchronous stream output mode. The synchronous stream output refers to a mode of transmitting data to the user end according to a unified time rhythm and output format, which can be understood as integrating stream data from different asynchronous threads into a single and continuous output stream, for eliminating the output time sequence difference caused by multi-model parallel processing.

[0067] In a feasible implementation, a fixed output time interval can be set, for example, one or more word units are obtained from the message queue every 50 milliseconds, and are immediately transmitted to the display interface of the user end through the network connection, realizing a progressive content presentation similar to the effect of a typewriter.

[0068] In another feasible implementation, the output rate can be dynamically adjusted according to the network status and processing capacity of the user end, and the number of word units transmitted at a time is appropriately increased when the network delay is high, and the output frequency is kept faster when the network status is good, so as to optimize the user experience in different network environments.

[0069] Please refer to Figure 3A , Figure 3A is one of the scene schematic diagrams of the multi-model question answering provided by the embodiments of the present application. As shown in Figure 3A , the user Tom sends a target question 301 "explain the wave-particle duality of quantum mechanics, and provide related exercises and learning suggestions" to the education platform through the learning application program of the tablet 300.

[0070] Figure 3B is a second scenario diagram of multi-model question answering provided by an embodiment of the present application. As shown, after the server 310 of the education platform receives the target question 301, it identifies that three professional models need to be called: the physical concept explanation model 311a, the exercise generation model 311b, and the learning path planning model 311c. The server 310 then creates asynchronous threads 312a, 312b, and 312c for each model, and creates a corresponding event loop in each asynchronous thread to implement an independent running environment for each model. Figure 3B

[0071] Based on the event loop in each asynchronous thread, the three models start asynchronous streaming output tasks 320a, 320b, and 320c in parallel. The physical concept explanation model 311a generates corresponding streaming data 321a in the event loop: “Wave-particle duality is a basic concept of quantum mechanics…”, the exercise generation model 311b generates streaming data 321b in the event loop: “Exercise 1: The double-slit experiment of light illustrates…”, and the learning path planning model 311c generates streaming data 321c in the event loop: “Suggested learning order: 1. Review of classical physics…”.

[0072] Each asynchronous thread writes the generated streaming data to the message queue 330, and during the writing process, the system assigns a first priority to each model according to the importance of the model in executing the asynchronous streaming output task. The first priority of the streaming data 321a is 9, the first priority of the streaming data 321b is 7, and the first priority of the streaming data 321c is 5. The message queue 330 stores the format of model ID, streaming content, and first priority.

[0073] When the streaming data in the message queue 330 reaches a preset capacity threshold, the system sets a preset priority threshold of 6 and identifies target streaming data with a first priority lower than 6, which mainly comes from the learning path planning model 311c. Therefore, at least two word units in the target streaming data are merged and processed, for example, the three word units “suggest”, “learn”, and “order” are merged into “suggest learning order”.

[0074] The system obtains word units from the message queue 330 after the merging and processing at fixed time intervals and returns the word units to the tablet 300 in a logical order using synchronous streaming output. The display interface of the tablet 300 presents the complete answer in a progressive manner, first displaying the physical concept explanation, then displaying the related exercises, and finally displaying the learning suggestions.

[0075] ​By creating corresponding event loops in asynchronous threads for each model in response to receiving the target question sent by the user terminal by the plurality of models, the plurality of models can execute asynchronous streaming output tasks based on independent event loops without blocking each other, effectively solving the problem of mutual influence caused by differences in processing speed.

[0076] The streaming data output by each asynchronous thread for the target question is written into a message queue, and different first priorities are set based on the importance of the asynchronous streaming output task of the corresponding model, a streaming data aggregation and scheduling mode is established, and the message queue effectively coordinates the difference in output frequency between different models as a buffer layer. When the streaming data in the message queue reaches a preset capacity threshold, by determining target streaming data with a first priority lower than a preset priority threshold from the message queue and merging at least two tokens in the target streaming data, both queue overflow and dynamic load balancing based on priority are achieved.

[0077] Finally, the tokens are obtained from the message queue after merging processing and returned to the user terminal in a synchronous streaming output manner, and the backend multi-model asynchronous concurrent processing is converted into front-end stable synchronous streaming output, thereby improving the overall stability of the multi-model concurrent execution of the streaming output task, effectively solving the technical problem of unstable overall streaming output caused by differences in processing speed and output frequency of each model in the related art, improving the receiving experience of the user terminal and improving the overall performance of the system.

[0078] Based on the above embodiments, as an optional implementation manner, the first priority of the model executing the asynchronous streaming output task can be determined in the following manner.

[0079] The first parameter representing the importance of the model in the system and the second parameter representing the dynamic performance of the model in the streaming output process are weighted and summed to determine the first priority of the streaming data corresponding to each model; wherein the first parameter includes at least one of the task type of the asynchronous streaming output task, the user level of the user terminal, the performance level of the model; the second parameter includes at least one of the output rate of the streaming data, the waiting time corresponding to the model, the data volume of the streaming data, and the load of the system.

[0080] The first parameter refers to a set of static attributes representing the importance of the model in the system, which can be understood in the embodiment of the present application as a quantitative indicator reflecting the relative position and role value of the model in the entire service system, and a basic weight coefficient can be assigned to the model.

[0081] Exemplarily, there is a positive correlation between the task type and the user level, and a high-level user usually corresponds to a more important task type, and both determine the basic weight of the business priority. There is a matching relationship between the model performance level and the task type, and a complex task type needs to match a high-performance level model to ensure the output quality and processing efficiency. The user level and the model performance level form a quality of service corresponding relationship, and a high-level user is preferentially allocated a high-performance model resource.

[0082] Similarly, the second parameter refers to a set of real-time attributes representing the dynamic performance of the model during the streaming output process, which can be understood in the embodiments of the present application as dynamic indicators reflecting the current running state and system load of the model, and the priority weight of the model can be adjusted in real time.

[0083] Exemplarily, the output rate and the waiting time are inversely proportional, and a higher output rate usually corresponds to a shorter waiting time, and both reflect the response performance of the model. There is an influence relationship between the data volume and the output rate, and excessive data volume may cause the output rate to decrease, and a balance needs to be established between the two. The system load has a constraint relationship with all other dynamic parameters, and the system in a high load state will simultaneously affect the performance of the output rate, the waiting time and the data volume.

[0084] In a feasible implementation, the relationship between the first parameter and the second parameter can be established by using the weighted summation calculation method. Specifically, the indicators in the first parameter form a static weight basis by weighted average, and the indicators in the second parameter form a dynamic weight adjustment by weighted average, and the two are linearly combined according to a preset proportion to obtain the final first priority, wherein the static weight provides a stable basic evaluation, and the dynamic weight provides real-time performance adjustment.

[0085] In another feasible implementation, when the system state reflected by the second parameter is good, the influence weight of the first parameter is increased to highlight the inherent importance of the model and the task; when the system state reflected by the second parameter is tense, the influence weight of the second parameter is increased, and the system load balancing and response efficiency are preferentially considered.

[0086] By using the embodiments of the present application, the organic combination of static importance and dynamic performance is realized by establishing a multi-level parameter correlation, a adjustable priority allocation method is provided, and the system adaptability in the multi-model concurrent scene is effectively improved.

[0087] On the basis of the above embodiments, as an optional embodiment, in order to further optimize the merging processing strategy of streaming data and ensure the semantic coherence and time sequence correctness of the output content, a second priority can also be determined for the word units in the streaming data, which can specifically include the following operations.

[0088] The second priority is determined according to the position number of each word in the corresponding model output sequence, and is used to determine the merging processing order of at least two words in the target streaming data; wherein the second priority of the word at the starting position of the output sequence is greater than the second priority of the word at the middle position of the output sequence.

[0089] The second priority refers to the weight level assigned to a single word based on the position feature of the word in the model output sequence, which can be understood as a numerical identifier used to determine the reserved priority order of the word in the merging process, and is used to ensure the semantic integrity of the output content after merging.

[0090] Similarly, the position number of the word in the output sequence refers to the arrangement order number of the word in the complete output sequence of the corresponding model, which can be understood as a numerical index identifying the generation time and semantic position of the word in the embodiments of the present application, and is used to reflect the importance of the word in the overall output content.

[0091] Exemplarily, the numerical range of the second priority can be set to 1-100 integers, wherein the larger the numerical value, the higher the importance of the position of the word, and the smaller the numerical value, the lower the importance of the position of the word.

[0092] The second priority of the word at the starting position of the output sequence is greater than the second priority of the word at the middle position of the output sequence. The word at the starting position usually carries key semantic information and context establishment function, and plays an important role in maintaining the logical coherence of the output content. In contrast, the words at the middle position are mostly connecting or modifying content, which can be reduced in transmission frequency through merging processing when necessary.

[0093] In a feasible implementation, the second priority can be calculated according to the position number of the word using a decreasing function, specifically, taking the total length of the output sequence as a reference value, the word with smaller position number is assigned higher second priority value, and the word with larger position number is assigned lower second priority value, and the end identifier word at the end of the sequence is set with higher second priority to ensure the output integrity.

[0094] In another feasible implementation, the second priority weight of the word at the beginning of the sentence, the key word, the semantic boundary word and other special position words can be additionally increased in combination with the semantic importance analysis in natural language processing, and the semantic weight is added to the standard position weight.

[0095] By establishing the second priority system based on the position sequence for the word units in the streaming data, the application embodiment can prioritize the word units in the semantic key positions during the merging process, effectively avoiding the semantic break and logical confusion problems caused by random merging, improving the streaming data merging process in the multi-model concurrent output scenario, and ensuring that the content transmitted to the user end always maintains good readability and semantic coherence.

[0096] On the basis of the above-mentioned embodiments, as an optional embodiment, the operation of merging at least two word units in the target streaming data in operation S240 can further include the following operations.

[0097] In operation S310, the system identifies the word units in the target streaming data whose second priority is lower than the preset word unit priority threshold, and selects at least two adjacent word units from the identified word units as the merging objects.

[0098] In operation S320, the system merges the word units in the merging objects in sequence based on the position sequence of each word unit in the original output sequence.

[0099] In operation S310, the merging object refers to the word unit set selected from the target streaming data that needs to be merged, which can be understood in the application embodiment as a combination of adjacent word units that meet the merging conditions.

[0100] After determining the target streaming data, the system can further analyze the word unit composition inside each target streaming data. For each target streaming data, the system compares the second priority of each word unit it contains with the preset word unit priority threshold, and identifies the word units whose second priority is lower than the threshold. Among the identified low-priority word units, the system selects at least two adjacent word units in the output sequence as the merging objects.

[0101] In a feasible implementation, a minimum merging unit quantity parameter can be set, for example, requiring at least 2 word units and at most 5 word units to be included in each merging, to avoid the problems of semantic ambiguity caused by excessive merging or limited effect caused by insufficient merging.

[0102] In operation S320, the original output sequence refers to the complete word unit sequence generated by the corresponding model when performing the asynchronous streaming output task, which can be understood in the application embodiment as the reference sequence that maintains the time sequence and semantic logic of the word units.

[0103] The system connects the word units in the merging objects in the order of time sequence from front to back according to their position sequence in the original output sequence, forming the new word units after merging. The merging process strictly follows the position sequence of the original output sequence, ensuring that the merged content maintains the same semantic logic and time sequence relationship as the original output.

[0104] By establishing the priority-based merging strategy, the system can perform the merging operation on the low-priority content under the premise of maintaining the integrity of the high-priority content, effectively alleviating the storage pressure of the message queue, ensuring the semantic coherence and logical integrity of the output content, and improving the stability of the streaming output and the user experience quality in the multi-model concurrent scenario.

[0105] On the basis of the above-mentioned embodiments, as an optional embodiment, the streaming output method of the above-mentioned model can further include the following operations.

[0106] In operation S410, in response to the capacity ratio of the current capacity of the message queue to the preset maximum capacity exceeding the preset threshold, the number of word units contained in the to-be-merged object is dynamically adjusted based on the capacity ratio, wherein the number of word units is positively correlated with the capacity ratio.

[0107] In operation S420, in response to the total output rate of each asynchronous thread being greater than the output rate of the synchronous streaming mode, the number of word units contained in the to-be-merged object is adjusted based on the rate difference between the total output rate and the output rate, wherein the number of word units is positively correlated with the rate difference.

[0108] In operation S430, in response to the proportion of the target streaming data of the first priority in the message queue being greater than the preset proportion threshold, the number of word units contained in the to-be-merged object is adjusted in proportion based on the difference between the proportion and the preset proportion threshold.

[0109] In operation S410, the capacity ratio refers to the proportional relationship between the number of streaming data currently stored in the message queue and the preset maximum capacity, which can be understood in the embodiment of the application as a quantitative indicator reflecting the current load degree of the message queue, for evaluating the storage pressure condition of the system.

[0110] When the capacity ratio exceeds the preset threshold, the system starts the dynamic adjustment strategy, and adjusts the number of word units contained in the to-be-merged object according to the specific value of the capacity ratio. The number of word units is positively correlated with the capacity ratio, that is, the higher the capacity ratio, the more word units contained in a single merging operation, thereby more effectively alleviating the storage pressure of the message queue.

[0111] In a feasible implementation, a linear adjustment function can be established to map the capacity ratio to a word unit quantity adjustment coefficient, and different capacity ratio intervals are set to correspond to different degrees of word unit quantity growth rate.

[0112] In operation S420, the total output rate refers to the total amount of streaming data generated by each asynchronous thread in a unit of time, which can be understood in the embodiment of the present application as an evaluation index reflecting the multi-model concurrent output capability, for measuring the overall performance of the data production end.

[0113] The rate difference refers to the numerical difference between the total output rate and the synchronous streaming mode output rate, which can be understood in the embodiment of the present application as a quantitative index reflecting the degree of imbalance between data production and consumption, for determining whether the input and output speed needs to be balanced through merging processing.

[0114] When the total output rate is greater than the output rate of the synchronous streaming mode, it indicates that the data production speed exceeds the consumption speed, which is likely to cause the message queue to be overloaded. At this time, the system adjusts the number of tokens in the object to be merged according to the rate difference, and the rate difference and the number of tokens are positively correlated, that is, the greater the rate difference, the more the number of tokens merged, so as to reduce the output frequency and balance the production and consumption speed.

[0115] In a feasible implementation, the output rate change of each asynchronous thread can be monitored in real time, the average total output rate in the sliding window is calculated, and compared with the fixed synchronous output rate. When the rate difference continuously exceeds the preset range, the number of tokens is adjusted.

[0116] In operation S430, the target streaming data refers to the streaming data in the message queue whose first priority is lower than the preset priority threshold, which can be understood as a set of low-priority data that needs to be focused on and optimized.

[0117] When the proportion is greater than the preset proportion threshold, it indicates that too much target streaming data has accumulated in the message queue, and the storage pressure needs to be relieved by increasing the merging intensity. The system calculates the difference between the current proportion and the preset proportion threshold, and adjusts the number of tokens contained in the object to be merged in proportion, and the adjustment amplitude is proportional to the size of the difference.

[0118] In a feasible implementation, a proportional adjustment strategy based on the proportion difference can be established to determine the corresponding token number growth proportion according to the specific value of the difference.

[0119] By establishing a multi-dimensional dynamic adjustment based on the message queue capacity, the output rate difference and the target streaming data distribution, the system can flexibly adjust the intensity and range of merging processing according to the real-time running state, which not only ensures the stable operation of the message queue, but also maintains the continuity and real-time of the streaming output, and improves the adaptive ability and overall performance of the system in the multi-model concurrent scene.

[0120] On the basis of the above-mentioned embodiments, as an optional embodiment, the operation S230 can further include the following operations:

[0121] In operation S510, the instance index is a unique identifier allocated for each model instance, which can be understood in the embodiments of the present application as a numerical value or a string mark for distinguishing the data sources of different asynchronous threads, and is used to ensure the traceability of data in the multi-model concurrent output.

[0122] In operation S520, the data structure format refers to a specification template defining the organization and field arrangement order of the message content, which can be understood in the embodiments of the present application as a format standard ensuring the consistency of data storage in the message queue, and is used to unify the representation methods of data output by different asynchronous threads.

[0123] In operation S530, the assembled message format data is sequentially written to the tail position of the message queue in the time sequence of the generation of the stream data.

[0124] In operation S510, the instance index is a unique identifier allocated for each model instance, which can be understood in the embodiments of the present application as a numerical value or a string mark for distinguishing the data sources of different asynchronous threads, and is used to ensure the traceability of data in the multi-model concurrent output.

[0125] For example, the system allocates a unique instance index for each asynchronous thread when initializing the asynchronous threads of each model. The instance index remains unchanged throughout the life cycle of the asynchronous thread, ensuring that all stream data generated by the thread can be correctly identified and classified.

[0126] In a feasible implementation, the instance index allocation rule can be predefined when the system starts, and the ordered index value can be set according to the type, function or importance of the model. When a new model instance is added, the next available instance index is automatically allocated according to the preset rule.

[0127] In another feasible implementation, the index allocation method based on the model configuration file can be used to extract the model name, version number and other information from the configuration parameters of the model to generate an instance index with uniqueness and readability.

[0128] In operation S520, the data structure format refers to a specification template defining the organization and field arrangement order of the message content, which can be understood in the embodiments of the present application as a format standard ensuring the consistency of data storage in the message queue, and is used to unify the representation methods of data output by different asynchronous threads.

[0129] The message format data refers to a standardized data object assembled according to the preset data structure format, which can be understood in the embodiments of the present application as a message queue storage unit containing complete information elements and uniform format, and is used for ordered storage and retrieval in the message queue.

[0130] Exemplarily, the system can assemble the content information of the streaming data, the corresponding instance index, and the calculated first priority in a specified order according to a preset data structure format. The assembly process needs to ensure the integrity of the field information and the uniformity of the format, and generate a standardized data object meeting the storage requirements of the message queue.

[0131] Exemplarily, the structure of the message format data can be designed as a dictionary format of {instance index, streaming data content, first priority, generation timestamp} or in a tuple format.

[0132] In a feasible implementation, a custom data class object can be defined as the message format, and each item of information is stored through the attribute field of the class.

[0133] In operation S530, the system continuously monitors the generation of the streaming data of each asynchronous thread, and when detecting the generation of new streaming data, immediately writes the assembled message format data into the message queue. The writing operation is strictly performed according to the actual generation time sequence of the streaming data, so as to ensure that the data generated first is written into the message queue first, and the data generated later is written into the message queue later.

[0134] In a feasible implementation, a timestamp label can be added to each message format data, and when multiple asynchronous threads generate streaming data at the same time in a very short time, the accurate writing sequence can be determined according to the difference of the timestamps.

[0135] By allocating a unique instance index to the streaming data of each asynchronous thread, the data traceability system is established, and the problem of unclear data attribution when multiple models are concurrently output is effectively solved. The standardized assembly method based on the preset data structure format ensures the consistency of the data format in the message queue. The ordered writing method according to the generation time sequence ensures the time sequence correctness of the data in the message queue, and avoids the problem of output logic confusion caused by the disorder of the writing sequence.

[0136] On the basis of the above embodiment, as an optional embodiment, the streaming output method of the above model can further include the following operations.

[0137] Operation S610, creating an iterable class object, wherein the iterable class object includes an initialization method and a next element obtaining method.

[0138] Operation S620, starting the asynchronous thread corresponding to each model to perform the streaming output task by calling the initialization method.

[0139] Operation S630, obtaining the tokens from the head position of the message queue in turn in a blocked waiting manner by cyclically calling the next element obtaining method.

[0140] Operation S640, repeat the next element acquisition method call process until the detection of message queue from each model of all streaming data output is completed.

[0141] In operation S610, the iterable class object refers to the class instance that implements the iterator protocol, which can be understood as a program object that supports accessing data elements one by one, used to encapsulate the control logic and data access interface of asynchronous streaming output to synchronous streaming output.

[0142] Among them, the initialization method refers to the constructor automatically executed when the iterable class object is created, which can be understood as a program entry responsible for setting the initial state of the object and starting the necessary resources, used to establish the connection relationship between the asynchronous thread and the message queue.

[0143] Among them, the next element acquisition method refers to the interface function provided by the iterable class object for obtaining the next data unit, which can be understood as a program method that implements synchronous blocking access to message queue data, used to return the streaming data content in order to the calling party.

[0144] The system creates an iterable class object instance according to the preset class definition template, which automatically initializes the necessary attribute variables and method interfaces during the creation process. The iterable class object serves as a bridging component between asynchronous streaming output and synchronous streaming output, encapsulating complex thread coordination and data synchronization operations.

[0145] Illustratively, the iterable class object can be designed as an object containing queue references, thread state identifiers, timeout configurations, and other core attributes, or implemented with other programming languages that support the iterator protocol to realize the corresponding class structure.

[0146] In a feasible implementation, the connection parameters of the message queue, the thread configuration information of each model, and the control parameters of the streaming output can be pre-configured in the constructor of the iterable class object, ensuring that the object has complete working ability after creation.

[0147] In operation S620, the system triggers the start process of the asynchronous thread corresponding to each model by calling the initialization method of the iterable class object. The initialization method creates independent asynchronous threads for each model in sequence during execution, and establishes a corresponding event loop in each asynchronous thread, ensuring that each model can execute the streaming output task concurrently.

[0148] The initialization method is also responsible for establishing the data transmission channel between the asynchronous thread and the message queue, configuring the necessary inter-thread communication parameters, and preparing for subsequent streaming data transmission.

[0149] In a feasible implementation, thread pool technology can be used in the initialization method to manage asynchronous threads corresponding to each model, and a unified thread scheduling strategy can be used to control the creation, execution and destruction of threads.

[0150] In another feasible implementation, thread monitoring and exception handling logic can be set in the initialization method, and when an asynchronous thread has an exception or fails to execute, thread restart or error recording operations are automatically performed.

[0151] In operation S630, blocking wait refers to a synchronization control method that suspends the current thread during program execution until a specific condition is met and then continues execution. It can be understood as a program control means to ensure synchronization and coordination between data acquisition operations and data production operations, and is used to avoid empty queue access and data loss problems.

[0152] The system realizes continuous access to streaming data in the message queue by calling the next element acquisition method in a loop. Each time the method is called, the next element acquisition method checks whether there is available data at the head of the message queue, and if there is, it returns the data immediately, and if there is not, it enters a blocking wait state until new data arrives.

[0153] In a feasible implementation, polling check logic can be set in the next element acquisition method, which checks the message queue state at fixed time intervals, and returns immediately when new data is detected at the head of the queue, and continues to wait for the next check period when the queue is empty.

[0154] In another feasible implementation, the blocking wait can be realized in an event-driven manner, and the execution of the next element acquisition method is triggered by the data arrival event of the message queue, avoiding unnecessary polling overhead.

[0155] In operation S640, the system repeatedly executes the call process of the next element acquisition method until it receives the streaming data output completion signal from each model. The detection of output completion can be realized through special end identifier data or asynchronous thread state change.

[0156] When it is detected that the streaming data output of all models has been completed, the next element acquisition method stops returning new data content, and the iteration process of the iterable class object ends, and the synchronous streaming output operation is completed.

[0157] In a feasible implementation, a unified output completion identifier can be set for each model, and when all models send the completion identifier, the next element acquisition method returns an iteration end signal to notify the caller to stop data acquisition operations.

[0158] By creating an iterable class object and implementing a standardized iterator interface, the application embodiment establishes a unified access method between asynchronous and synchronous streaming output, effectively solving the compatibility problem between different programming modes. The call mode of the initialization method ensures the orderly start of asynchronous threads and the correct allocation of resources, avoiding competition conditions and resource conflicts in the thread creation process. Through the blocked next element acquisition method, safe access and orderly acquisition of message queue data are realized, ensuring the integrity and continuity of streaming data in the synchronous output process, and improving the data transmission reliability and user experience consistency of multi-model concurrent streaming output.

[0159] On the basis of the above-mentioned embodiments, as an optional embodiment, operation S250 can further include the following operations.

[0160] In the case that the message queue is not empty, and the time interval between the current time and the last time when a token is read from the message queue is greater than the preset time interval, a token is read from the head position of the message queue after the merging processing.

[0161] The preset time interval refers to the minimum time difference between two consecutive token reading operations set by the system, which can be understood as a time reference for controlling the synchronous streaming output rate, and is used to ensure the stability of streaming data transmission to the user end.

[0162] For example, the system records the last time when a token is read from the message queue, and calculates the time interval between the current time and the last reading time every time a new token is read. Only when the time interval reaches or exceeds the preset time interval, and there is a token to be read in the message queue, the system performs the actual token reading operation.

[0163] The above reading method can be implemented based on the flow control principle of the bucket algorithm. The bucket algorithm refers to an algorithm strategy for controlling data flow rate through token generation and consumption. In the application embodiment, it can be understood as a flow management method for generating reading permissions at fixed time intervals to ensure that token reading operations are performed at a constant rate, which is used to smooth the irregularity of multi-model asynchronous output.

[0164] Specifically, the system maintains a virtual token bucket, and adds tokens to the bucket at a preset time interval. Each time a token reading operation is performed, a token is consumed. When there is no available token in the bucket, the reading operation is suspended until a new token is generated. The capacity of the token bucket can be set to 1 to ensure that the reading operation is strictly performed at a preset time interval, avoiding burst high-frequency reading.

[0165] In a feasible implementation, the system clock can be used to obtain the current timestamp, and by comparing the timestamp difference with the preset time interval, it is determined whether the current word reading operation is allowed to be executed, and the last reading time record is updated.

[0166] In another feasible implementation, a timer can be set to trigger the word reading task at a preset time interval, and when the timer is triggered, the non-empty state of the message queue is checked, and when the condition is met, a word is read from the head position and returned to the user end.

[0167] In the case of empty message queue, the system suspends the word reading operation, but continues to maintain the calculation of the time interval, ensuring that when the message queue regains data, it can immediately resume the reading process according to the correct time rhythm.

[0168] In a feasible implementation, when the message queue is empty, a polling check logic can be set, with half of the preset time interval as the check frequency, continuously monitoring the message queue state change, and when new data is detected and the time interval condition is met, the reading operation is immediately executed.

[0169] In another feasible implementation, an event notification method can be combined, which triggers a notification event when the message queue changes from an empty state to a non-empty state, and checks the time interval condition during event processing. If the condition is met, the word reading operation is executed.

[0170] When the time difference is less than the preset time interval, the system enters a waiting state until the time difference reaches the preset time interval requirement. During the waiting process, the system does not perform any word reading operation, and maintains the stability of the rhythm of the synchronous streaming output.

[0171] By using the embodiments of the present application, a stable and controllable synchronous streaming output rhythm is established based on the reading control condition of the preset time interval, effectively solving the problem of user experience fluctuation caused by the inconsistent output rate of multiple models. The application of the bucket algorithm ensures that the word reading operation is executed at a constant time interval, avoiding the processing burden on the user end caused by sudden high-frequency data transmission, and preventing the problem of response delay caused by slow reading rate. The double condition judgment of message queue non-empty check and time interval control ensures that the system can maintain a stable output rate in various running states, improving the reliability of multiple model concurrent streaming output and the consistency of user experience.

[0172] On the basis of the above embodiments, as an optional embodiment, in order to provide a more refined timeout control process and exception handling step, ensure the controllability and maintainability of the streaming output operation, operation S250 can further include the following operations.

[0173] Operation S710, set the waiting time threshold as a timeout control parameter.

[0174] Operation S720, in the case of empty message queue and current waiting time greater than the waiting time threshold, return timeout exception information and terminate the corresponding streaming output process.

[0175] Operation S730, in the case of empty message queue and current waiting time less than or equal to the waiting time threshold, keep the message queue in a blocked waiting state until the new streaming data is generated by each asynchronous thread and written to the message queue.

[0176] In operation S710, the system can use a static configuration method to pre-set a fixed waiting time threshold, or use a dynamic adjustment method to automatically adjust the threshold value according to the real-time system load and model performance.

[0177] In one possible implementation, based on historical output data statistical analysis of each model, the average data generation interval and standard deviation can be calculated, and the waiting time threshold can be set to the average interval plus several times the standard deviation value, to ensure that false alarms will not be triggered under normal circumstances. Timeout exception.

[0178] In another possible implementation, a hierarchical waiting time threshold system can be set, including a warning threshold and a termination threshold. When the waiting time reaches the warning threshold, a warning message is recorded but the waiting continues, and when the termination threshold is reached, the timeout termination operation is performed.

[0179] In operation S720, the system continuously monitors the duration of the empty state of the message queue, and compares the current waiting time with the set waiting time threshold. When it is detected that the waiting time exceeds the threshold limit, the timeout processing flow is started.

[0180] The timeout processing flow includes generating detailed timeout exception information, recording the current system state snapshot, cleaning up related resources, and safely terminating the streaming output process. The system ensures that other concurrent running streaming output tasks are not affected when performing timeout processing.

[0181] In one possible implementation, detailed state information of each asynchronous thread, historical write records of the message queue, current system resource usage, and the exact time of timeout occurrence can be recorded in the timeout exception information, providing information for subsequent fault analysis.

[0182] In another possible implementation, a gradual resource cleanup operation can be performed before terminating the streaming output process, including stopping related asynchronous threads, releasing occupied memory resources, closing network connections, and cleaning up temporary files, to ensure complete recovery of system resources.

[0183] In operation S730, when the message queue is empty but the waiting time has not exceeded the threshold, the system enters a controlled blocking waiting state. In this state, the system suspends the token reading operation, but continues to monitor the changes in the asynchronous thread state and the message queue.

[0184] In the blocking waiting state, the system uses an efficient event-driven approach, and the state transition is triggered by data writing events or timing check events of the asynchronous thread. When new streaming data is detected in the message queue, the system immediately exits the blocking waiting state and resumes the normal token reading and output operations.

[0185] In a feasible implementation, a multi-level waiting strategy can be set in the blocking waiting state. Initially, active waiting with a short interval is used, and the check interval is gradually extended as the waiting time increases, reducing system resource consumption while ensuring responsiveness.

[0186] In addition, the system also needs to handle external interrupt requests and cancellation operations in the blocking waiting state to ensure that it can respond in time and safely exit the waiting state when the user actively cancels the streaming output request.

[0187] By establishing a timeout handling workflow, the embodiments of the present application improve the traceability and maintainability of system exception handling. The accurate setting of the waiting time threshold ensures the accuracy and reasonableness of the timeout judgment, avoiding both premature termination and excessive waiting. The step-by-step timeout check and blocking waiting control realize the combination of the timeout protection function and the normal data processing flow, ensuring system stability while maximizing data output integrity.

[0188] On the basis of the above embodiments, as an optional embodiment, operation S210 can further include the following operations:

[0189] Operation S810 allocates independent asynchronous threads to each model.

[0190] Operation S820 creates an event loop in each asynchronous thread.

[0191] Operation S830 initializes the streaming output task in the corresponding asynchronous thread using the configuration parameters of each model based on the event loop in each asynchronous thread, wherein the configuration parameters include at least one of the model interface address, authentication information, and streaming output parameters.

[0192] In operation S810, the system can create a dedicated asynchronous thread for each model according to the preset multi-model list, ensuring that the streaming output operation of each model can run independently in an isolated execution environment. The independent asynchronous thread refers to an independent execution unit specifically allocated for a single model, which can be understood as a thread entity that realizes concurrent isolation and resource exclusivity between models, avoiding mutual interference and resource competition between different models.

[0193] When creating an asynchronous thread, the system sets a unique identifier for each thread and establishes a mapping relationship between the thread and the model, ensuring that subsequent task scheduling and resource management can accurately locate the corresponding thread entity.

[0194] In a feasible implementation, different priorities and resource quotas can be set for asynchronous threads according to the computational complexity and expected output frequency of each model, ensuring that high-priority models can obtain more system resources and execution time.

[0195] In operation S820, the system creates a dedicated event loop structure inside each asynchronous thread. The event loop refers to a loop scheduling structure for handling asynchronous events and callback functions, which can be understood in this embodiment as a scheduler that manages the execution order and timing of asynchronous tasks, used to implement non-blocking asynchronous operation execution and event response processing.

[0196] The creation process of the event loop includes initializing the event queue, setting the event dispatcher, and configuring the loop running parameters. Each event loop runs independently and only processes events and tasks in the corresponding asynchronous thread, avoiding cross-thread event interference.

[0197] In a feasible implementation, different event loop parameters can be configured for different types of models, including event queue capacity, loop check interval, and timeout handling strategy, to optimize event processing performance according to model characteristics.

[0198] In operation S830, the system uses the event loop created in each asynchronous thread to complete the initialization of the streaming output task in combination with the specific configuration parameters of the model. The configuration parameters refer to a set of parameters that describe the model access method and output behavior, which can be understood as configuration information that defines the model call interface, identity verification, and output format, used to ensure that the streaming output task can correctly connect and call the corresponding model service.

[0199] Among them, the model interface address refers to the network address information used to access the model service, which can be understood as a complete address containing the protocol type, server address, port number, and interface path, used to establish a network connection with the model service.

[0200] The authentication information refers to credential data for verifying the identity and authority of the calling party, which in the embodiments of the present application can be understood as identity verification credentials in the form of a key, an access token, or a digital certificate, etc., used to ensure the security and legality of model service calls.

[0201] The streaming output parameter refers to parameter settings for controlling the output behavior and format of the model, which in the embodiments of the present application can be understood as a combination of configuration items including output block size, encoding format, compression options, and transmission protocol, etc., used to customize the streaming data output mode of the model.

[0202] The system verifies the integrity and validity of the configuration parameters when initializing the streaming output task, establishes a connection with the model service, and configures the corresponding data reception and processing logic. The initialization process is executed in an asynchronous manner under the scheduling of the event loop, avoiding blocking the concurrent execution of other tasks.

[0203] In a feasible implementation, connection testing and parameter verification can be performed during the initialization process to ensure the accessibility of the model service and the correctness of the configuration parameters, and when problems are found, exceptions are reported in a timely manner and corresponding processing is performed.

[0204] After the streaming output task is initialized, each asynchronous thread enters a ready state, waiting to receive specific user query requests and start performing corresponding model calls and data processing operations.

[0205] By allocating independent asynchronous threads to each model and creating a dedicated event loop, a completely isolated concurrent execution environment is established, effectively avoiding resource competition and mutual interference problems between different models. The event loop-based streaming output task initialization method realizes efficient scheduling and management of asynchronous operations, ensuring that each model can execute streaming output operations in the optimal running environment.

[0206] Figure 4 A block diagram of an electronic device suitable for the model streaming output method according to the embodiments of the present application is schematically shown.

[0207] As Figure 4As shown, the electronic device according to the embodiments of the present application includes a processor 401 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 402 or a program loaded into a random access memory (RAM) 403 from a storage section 408. The processor 401 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 401 can also include an on-board memory for cache use. The processor 401 can include a single processing unit or multiple processing units to perform the various actions of the method processes according to the embodiments of the present application.

[0208] In the RAM 403, various programs and data required for the operation of the electronic device are stored. The processor 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. The processor 401 performs various operations of the method processes according to the embodiments of the present application by executing the programs in the ROM 402 and / or the RAM 403. Note that the programs can also be stored in one or more memories other than the ROM 402 and the RAM 403. The processor 401 can also perform various operations of the method processes according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0209] According to the embodiments of the present application, the electronic device can further include an input / output (I / O) interface 405 which is also connected to the bus 404. The electronic device can further include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as necessary. A removable recording medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read out therefrom is installed in the storage section 408 as necessary.

[0210] The present application also provides a computer readable storage medium, which can be included in the device / system described in the above embodiments, or can exist separately without being assembled into the device / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0211] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include the ROM 402 and / or the RAM 403 described above and / or one or more memory other than the ROM 402 and the RAM 403.

[0212] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the model streaming output method provided by the embodiments of the present application.

[0213] The above functions defined in the system / device of the embodiments of the present application are performed when the computer program is executed by the processor 401. According to an embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0214] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the detachable medium 411. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0215] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the detachable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0216] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, "C", or the like. Program code can execute entirely on a user's computing device, partly on the user's device, as a stand-alone software package, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0217] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0218] Those skilled in the art will understand that features recited in the various embodiments of the present application can be combined and / or integrated in various combinations and / or permutations, even if such combinations and / or permutations are not expressly noted in the present application. In particular, features recited in the various embodiments of the present application can be combined and / or integrated in various combinations and / or permutations without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

Claims

1. A method of streaming output of a model, the method comprising: The method comprises the following steps: in response to a plurality of models receiving a target question sent by a user terminal, creating a corresponding event loop in an asynchronous thread for each of the models; based on the event loop in the asynchronous thread, performing an asynchronous streaming output task of the corresponding model to generate streaming data for the target question; write the streaming data output by each of the asynchronous threads to a message queue, wherein each of the streaming data has a different first priority based on the importance of the corresponding model in performing the asynchronous streaming output task; in response to the streaming data in the message queue reaching a preset capacity threshold, determine target streaming data with a first priority lower than a preset priority threshold from the message queue, and merge at least two tokens in the target streaming data; sequentially obtain tokens from the message queue after the token merging processing, and return them to the user terminal in a synchronous streaming output manner; The method further comprises the following steps: determine a second priority for each of the tokens according to the position sequence number of each of the tokens in the output sequence of the corresponding model; the second priority is used to indicate the merging processing order of at least two tokens in the target streaming data; wherein the second priority of the token located at the starting position of the output sequence is greater than the second priority of the token located at the middle position of the output sequence; The merging processing of at least two tokens in the target streaming data comprises the following steps: identify tokens with a second priority lower than a preset token priority threshold in the target streaming data, and select at least two adjacent tokens from the identified tokens as merging objects; merge the tokens in the merging objects in order based on the position sequence of each token in the original output sequence; The writing of the streaming data output by each of the asynchronous threads to the message queue comprises the following steps: allocate an instance index corresponding to the corresponding model to the streaming data output by each of the asynchronous threads respectively, wherein the instance index is used to identify and distinguish the data sources of different models in the multi-model concurrent output scenario; based on a preset data structure format, assemble the content information of the streaming data, the instance index and the first priority into standardized message format data; write the assembled message format data to the tail position of the message queue in the generation time sequence of the streaming data.

2. The method of claim 1, wherein, The method further comprises the following steps: weight and sum a first parameter representing the importance of the model in the system and a second parameter representing the dynamic performance of the model in the streaming output process to determine the first priority of the streaming data corresponding to each of the models; wherein the first parameter comprises at least one of the task type of the asynchronous streaming output task, the user level of the user terminal, and the performance level of the model; and the second parameter comprises at least one of the output rate of the streaming data, the waiting time of the model, the data volume of the streaming data, and the load of the system.

3. The method of claim 1, wherein, The method further comprises the following steps: create an iterable class object, wherein the iterable class object comprises an initialization method and a next element acquisition method; The initialization method is called to start an asynchronous thread execution flow to output the task corresponding to each model; The next element acquisition method is called in a loop to acquire tokens from the head of the message queue in a blocking manner; The calling process of the next element acquisition method is repeated until all streaming data output from each model in the message queue is detected to be complete.

4. The method of claim 1, wherein, The method further comprises: In the case that the message queue is not empty, and the time interval between the current time and the last time the token is read from the message queue is greater than the preset time interval, a token is read from the head of the message queue after the merging processing.

5. The method of claim 4, wherein, The method further comprises: Setting a waiting time threshold as a timeout control parameter; In the case that the message queue is empty, and the current waiting time is greater than the waiting time threshold, returning a timeout exception message and terminating the corresponding streaming output process; In the case that the message queue is empty, and the current waiting time is less than or equal to the waiting time threshold, keeping the message queue in a blocking waiting state until new streaming data is generated by each asynchronous thread and written to the message queue.

6. The method of claim 1, wherein, The method further comprises: Allocating an independent asynchronous thread for each model; Creating an event loop in each asynchronous thread; Based on the event loop in each asynchronous thread, initializing the streaming output task in the corresponding asynchronous thread using the configuration parameters of each model, wherein the configuration parameters include at least one of the model interface address, authentication information and streaming output parameters.

7. An electronic device, comprising: The method further comprises: One or more processors; Memory for storing one or more computer programs, The one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-6.

Images