Method and system for streaming transmission

Abstract

This disclosure proposes a streaming transmission method and system. The method includes: receiving a request from a client and establishing a task based on the request; while executing the task, invoking a language model to continuously generate multiple data fragments and storing these data fragments in a streaming pool; receiving a query from the client corresponding to the task and determining whether the task is complete; in response to task completion, establishing a one-time connection with the client to transmit all stored data fragments to the client; and in response to task incompleteness, establishing a streaming connection with the client to transmit the currently stored data fragments to the client, continuously receiving subsequent data fragments from the language model, and transmitting the subsequent data fragments to the client via the streaming connection.

Claims

1. A streaming transmission method, applicable to a computer system, the streaming transmission method comprising: receiving a request from a client and establishing a task based on the request; while executing the task, calling a language model to continuously generate multiple data fragments and storing the data fragments in a streaming pool; receiving a query from the client corresponding to the task and determining whether the task is completed; in response to the task being completed, establishing a one-time connection with the client to transmit all the stored data fragments to the client; and in response to the task not being completed, establishing a streaming connection with the client to transmit the existing stored data fragments to the client, continuously receiving a subsequent data fragment from the language model, and transmitting the subsequent data fragment to the client through the streaming connection.

2. The streaming transmission method as described in claim 1 further includes: Create a task queue and add the task to the task queue; And execute a job application to retrieve the task from the task queue and execute the task by the job application.

3. The streaming transmission method as described in claim 2 further includes: Obtain a first hint and a function definition corresponding to the request, and pass the first hint and the function definition to the language model; It accepts a call function from the language model and at least one parameter corresponding to the call function; and the working application executes the call function based on the at least one parameter to obtain a function output.

4. The streaming transmission method as described in claim 3 further includes: The function output and a second prompt are sent to the language model, and a response is obtained from the language model. And add the response to the streaming pool.

5. The streaming transmission method as described in claim 1, wherein the client is an application backend, the streaming transmission method further includes: The application backend obtains user input from an application frontend; And the request is generated by the application backend based on the user's input.

6. The streaming transmission method as described in claim 1 further includes: In response to the client losing the data segments, the data segments are retrieved from the streaming pool and then transmitted back to the client.

7. The streaming transmission method as described in claim 1 further includes: In response to the completion of the task, the task result formed by these data fragments is stored in a database.

8. The streaming transmission method as described in claim 7, wherein the step of determining whether the task is completed includes: Check if the database contains the task results; if the database does not contain the task results, determine that the task has not been completed. And in response to the database containing the results of the task, determine that the task has been completed.

9. The streaming method as described in claim 8, wherein the step of establishing the one-time connection with the client to transmit all stored data fragments to the client in response to the completion of the task includes: Retrieve the task result from the database and transmit the result to the client.

10. The streaming transmission method as described in claim 1, wherein the request, the one-time connection, and the streaming connection conform to a Hypertext Transfer Protocol (HTTP), and the streaming connection is a Server-Sent Event (SSE) connection.

11. A streaming transmission system, comprising: One client; A server, communicatively connected to the client, is used to receive a request from the client and create a task based on the request. When executing the task, the server calls a language model to continuously generate multiple data fragments and stores these data fragments in a streaming pool. The server receives a query from the client corresponding to the task and determines whether the task is complete. In response to the task being complete, the server establishes a one-time connection with the client to transmit all the stored data fragments to the client. In response to the task not being complete, the server establishes a streaming connection with the client to transmit the existing stored data fragments to the client, continuously receives subsequent data fragments from the language model, and transmits the subsequent data fragments to the client through the streaming connection.

12. The streaming system of claim 11, wherein the server is further configured to establish a task queue and add the task to the task queue, wherein the server is configured to execute a job application to retrieve the task from the task queue and execute the task by the job application.

13. The streaming system of claim 12, wherein the server is further configured to obtain a first prompt and a function definition corresponding to the request, transmit the first prompt and the function definition to the language model, and accept a call function from the language model and at least one parameter corresponding to the call function, wherein the server is further configured to execute the call function according to the at least one parameter to obtain a function output.

14. The streaming system as claimed in claim 13, wherein the server is further configured to transmit the function output and a second prompt to the language model, obtain a response from the language model, and add the response to the streaming pool.

15. The streaming system as described in claim 11, wherein the client is an application backend, wherein the application backend obtains a user input from an application frontend and generates the request based on the user input.

16. The streaming system as claimed in claim 11, wherein in response to the client losing the data segments, the server retrieves the data segments from the streaming pool and retransmits the data segments to the client.

17. The streaming transmission system as claimed in claim 11, wherein, in response to the completion of the task, the server stores a task result formed from the data fragments in a database.

18. The streaming transmission system as described in claim 17, wherein the server queries the database to see if the task result is contained therein; in response to the database not containing the task result, the server determines that the task has not been completed; and in response to the database containing the task result, the server determines that the task has been completed.

19. The streaming transmission system as described in claim 18, wherein in response to the completion of the task, the server retrieves the task result from the database and transmits the task result to the client.

20. The streaming system as described in claim 11, wherein the request, the one-time connection, and the streaming connection conform to a Hypertext Transfer Protocol (HTTP), and the streaming connection is a Server-Sent Event (SSE) connection.

Images