Methods, apparatus, equipment, storage media and software products for data transmission

Abstract

This application discloses a data transmission method, apparatus, device, storage medium, and program product applied to a server. The method includes: when the number of tokens generated by a large model equals a first threshold, inputting the first threshold number of tokens into a word segmenter and sending a notification message to a listener; when the word segmenter determines that the number of received tokens equals a second threshold, sending the received second threshold number of tokens to a soft channel, the soft channel corresponding to the user request; after receiving the notification message, the listener reads tokens one by one from the soft channel; and the listener sends the read tokens to an external client. This improves the stability of streaming transmission.

Description

Technical Field

[0001] This application belongs to the field of communication technology, and in particular relates to a method, apparatus, device, storage medium and program product for data transmission. Background Technology

[0002] Currently, large models are widely used in open dialogue systems. Specifically, large models can be used for various natural language processing tasks, such as text generation, speech recognition, and translation.

[0003] The output of the large model can be categorized into two methods. The first is non-streaming inference, where the large model outputs all prediction results at once. The second is streaming inference, where the large model outputs prediction results gradually while the user is waiting. However, during the streaming transmission of the prediction results output by the large model, the stability of streaming transmission is insufficient, and prolonged communication can easily lead to disconnection. Summary of the Invention

[0004] This application provides a data transmission method, apparatus, device, storage medium, and program product that can improve the stability of streaming transmission.

[0005] In a first aspect, embodiments of this application provide a data transmission method applied to a server, the method comprising:

[0006] If the number of tokens generated by the large model is equal to the first threshold, input the first threshold number of tokens into the word segmenter and send a notification message to the listener;

[0007] If the word segmenter determines that the number of received tokens is equal to the second threshold, it sends the second threshold number of received tokens to the soft channel, which corresponds to the user request.

[0008] After receiving the notification message, the listener reads the tokens one by one from the soft channel;

[0009] The listener sends the read token to the external client.

[0010] In one possible implementation, before inputting the first threshold number of tokens into the tokenizer and sending a notification message to the listener, the method further includes:

[0011] Receive the soft channel ID and user request sent by the external client;

[0012] Input the statements in the user request into the large model;

[0013] Determine the correspondence between the soft channel ID and the soft channel.

[0014] In one possible implementation, before inputting the first threshold number of tokens into the token segmenter and sending a notification message to the listener, if the number of tokens generated in the large model equals a first threshold number, the method further includes:

[0015] Obtain the first quantity threshold and the second quantity threshold;

[0016] Obtain the token generated by the large model;

[0017] Determine whether the number of tokens generated by the large model is equal to the first quantity threshold.

[0018] In one possible implementation, obtaining the first quantity threshold and the second quantity threshold includes:

[0019] Obtain the model category information, number of soft channels, network bandwidth information, and corresponding basic throughput information of the large model;

[0020] The model category information, the number of soft channels, the network bandwidth information, and the basic throughput information are input into the output prediction model to obtain the prediction result of the output prediction model.

[0021] The first quantity threshold and the second quantity threshold are determined based on the prediction results.

[0022] In one possible implementation, determining the first quantity threshold and the second quantity threshold based on the prediction result includes:

[0023] If the predicted token generation speed in the prediction result is greater than the actual token generation speed, the historical quantity threshold is adjusted according to a preset step size to obtain the first quantity threshold.

[0024] Given the first quantity threshold, a preset multiple of the first quantity threshold is calculated to obtain the second quantity threshold.

[0025] In one possible implementation, after determining the first quantity threshold and the second quantity threshold based on the prediction result, the method further includes:

[0026] If the predicted token generation speed in the prediction results is greater than the actual token generation speed, the number of soft channels in the server will be reduced by a preset amount.

[0027] If the predicted token generation speed in the prediction results is less than or equal to the actual token generation speed, the number of soft channels in the server will be increased by a preset amount.

[0028] In one possible implementation, the soft channel includes a list object corresponding to the user request, the list object being used to store the tokens generated by the large model; sending the received second threshold number of tokens to the soft channel includes:

[0029] Find the first node using the first pointer in the list object;

[0030] Determine whether the first node has a pointer to the next node;

[0031] If the next node pointer exists at the first node, the second node is found according to the next node pointer;

[0032] Using the second node as the first node, the steps of determining whether the first node has a next node pointer and searching for the second node according to the next node pointer are executed until the first node found does not have a next node pointer.

[0033] Store the second threshold number of tokens into the first node, which does not have a next node pointer.

[0034] Secondly, embodiments of this application provide a data transmission apparatus applied to a server, the apparatus comprising:

[0035] The judgment module is used to input the first number of tokens into the word segmenter and send a notification message to the listener when the number of tokens generated by the large model is equal to the first number threshold.

[0036] The sending module is used to send the received second-threshold number of tokens to a soft channel when the word segmenter determines that the number of received tokens is equal to a second-threshold number, wherein the soft channel corresponds to the user request;

[0037] The reading module is used by the listener to read tokens one by one from the soft channel after receiving the notification message;

[0038] The sending module is also used for the listener to send the read token to an external client.

[0039] Thirdly, embodiments of this application provide an electronic device, the device including: a processor and a memory storing computer program instructions;

[0040] The method by which the processor implements data transfer as described in the first aspect when executing computer program instructions.

[0041] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data transmission method as described in the first aspect is implemented.

[0042] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the data transmission method as described in the first aspect.

[0043] This application discloses a data transmission method, apparatus, device, storage medium, and program product. After obtaining tokens generated by a large model, the server, upon determining that the number of tokens generated by the large model equals a first threshold, inputs the first threshold number of tokens into a token segmenter and then sends a notification message to a listener. The token segmenter, upon determining that the number of received tokens equals a second threshold, sends the received second threshold number of tokens to a soft channel. After receiving the notification message, the listener, according to the correspondence between the soft channel and the user request, reads tokens one by one from the soft channel corresponding to the user request and sends them to the external client. Thus, the server segments the tokens generated by the large model for transmission based on the first and second thresholds, achieving streaming transmission of the large model's output results and reducing data transmission latency. Furthermore, each user request corresponds to a soft channel, thereby achieving streaming transmission of the large model's output results. During streaming transmission, the use of soft channels avoids connection drops during transmission, thus improving data transmission stability. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a data transmission method provided in an embodiment of this application;

[0046] Figure 2 This is an exemplary schematic diagram of a data transmission structure provided in an embodiment of this application;

[0047] Figure 3 This is an exemplary schematic diagram of another data transmission structure provided in the embodiments of this application;

[0048] Figure 4 This is a flowchart illustrating a method for determining a quantity threshold provided in an embodiment of this application;

[0049] Figure 5 This is a flowchart illustrating a quantity threshold adjustment method provided in an embodiment of this application;

[0050] Figure 6 This is an exemplary schematic diagram of a quantity threshold adjustment method provided in an embodiment of this application;

[0051] Figure 7 This is a flowchart illustrating a data storage method provided in an embodiment of this application;

[0052] Figure 8 This is an exemplary schematic diagram of a data storage method provided in an embodiment of this application;

[0053] Figure 9 This is an exemplary schematic diagram of a data transmission queue provided in an embodiment of this application;

[0054] Figure 10 This is an exemplary schematic diagram of data writing provided in an embodiment of this application;

[0055] Figure 11 This is an exemplary schematic diagram of an operation permission acquisition method provided in an embodiment of this application;

[0056] Figure 12 This is an exemplary schematic diagram of another method for obtaining operation permissions provided in an embodiment of this application;

[0057] Figure 13 This is an exemplary schematic diagram of a data writing method provided in an embodiment of this application;

[0058] Figure 14 This is an exemplary schematic diagram of a data reading method provided in an embodiment of this application;

[0059] Figure 15 This is a schematic diagram of the structure of a resource scheduling device provided in an embodiment of this application;

[0060] Figure 16 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0062] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0063] To address the problems of the prior art, embodiments of this application provide a method, apparatus, device, storage medium, and program product for data transmission. The data transmission method provided in this application embodiment will be described first. Figure 1 As shown, this method is applied to a server, and the method includes:

[0064] S101. If the number of tokens generated by the large model is equal to the first quantity threshold, input the first quantity threshold tokens into the word segmenter and send a notification message to the listener.

[0065] The server includes multiple large models; in this embodiment, the large model corresponds to the user request. That is, after receiving a user request, the server inputs the user statement carried in the request into the corresponding large model, obtaining the output of the large model for each user statement. The output of the large model is in token form.

[0066] The first quantity threshold is dynamically adjusted based on the actual network conditions, and the specific adjustment method will be described in detail in subsequent embodiments.

[0067] Understandably, once the server detects that the number of tokens generated for a user request equals a first threshold, it can use a chunked transmission method to stream the first threshold number of tokens. In this way, the server does not need to anticipate the size of the large model's prediction result, thus adapting to the generation capabilities of large models.

[0068] The server retrieves the streaming results generated by the large model (Streamer), and uses Python's Asynchronous Input Output (asyncio) library and the yield keyword to obtain the token generated by the large model in real time from the streaming results.

[0069] S102. If the token segmenter determines that the number of received tokens is equal to the second quantity threshold, the received second quantity threshold tokens are sent to the soft channel.

[0070] Within this, soft channels correspond to user requests. Soft channels include first-in-first-out queues.

[0071] After an external client receives a user request, it allocates a soft channel for that request and synchronizes the soft channel identifier (id) with the server. Specifically, the server selects a soft channel from multiple pre-built soft channels locally and binds the soft channel ID. In this way, the server can use multiple soft channels to process multiple user requests in parallel.

[0072] Specifically, the second quantity threshold can be adjusted according to actual network resources. In particular, the second quantity threshold can be used by the Toknizer tokenizer to control the tokens sent to the soft channel.

[0073] S103. After receiving the notification message, the listener reads the tokens one by one from the soft channel.

[0074] The listener on the server can retrieve tokens one by one from the first-in-first-out queue.

[0075] S104. The listener sends the read token to the external client.

[0076] In this process, after the listener reads the end-of-stream character, it sends the end-of-stream character to the external client, and the external client stops receiving the streaming output; otherwise, it continues to receive the output.

[0077] Using the above method, after obtaining the tokens generated by the large model, the server, upon determining that the number of tokens generated by the large model equals a first threshold, inputs the first threshold number of tokens into the token segmenter and then sends a notification message to the listener. The token segmenter, upon determining that the number of received tokens equals a second threshold, sends the second threshold number of received tokens to the soft channel. After receiving the notification message, the listener, according to the correspondence between the soft channel and the user request, reads the tokens one by one from the soft channel corresponding to the user request and sends them to the external client. In this way, the server segments and transmits the tokens generated by the large model in blocks based on the first and second thresholds, achieving streaming transmission of the large model's output structure and reducing data transmission latency. Furthermore, each user request corresponds to a soft channel, thus achieving streaming transmission of the large model's output results. During streaming transmission, the use of soft channels avoids connection drops during transmission, thereby improving data transmission stability.

[0078] In some embodiments of this application, before inputting the first number of tokens into the word segmenter and sending a notification message to the listener in S101 (when the number of tokens generated by the large model is equal to a first number threshold), the server can see the correspondence between the soft channel and the soft channel ID. Specifically, this can be implemented as follows:

[0079] Step 1: Receive the soft channel ID and user request sent by the external client.

[0080] The server is equipped with a preset number of soft channels. After each soft channel completes the streaming transmission of the large model prediction results for a user request, it can still be used for other user requests.

[0081] Step 2: Input the statements from the user's request into the large model.

[0082] Step 3: Determine the correspondence between soft channel IDs and soft channels.

[0083] By using the method provided in this application embodiment, and by binding the correspondence between soft channel ID and soft channel, after the user request corresponding to the soft channel ID is input into the large model, the server can continuously read the output result of the large model through the corresponding soft channel, thereby ensuring the stability of the server's transmission of the large model's output result.

[0084] The following combination Figure 2 and Figure 3 This application introduces a data transmission method provided in its embodiments, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of a server-side overall module structure provided in an embodiment of this application.

[0085] In this embodiment, the server side includes a server and a client. After receiving the user's input statement, the external client generates a user request, constructs a statement based on the user request, and generates a soft channel ID. Then, the external client sends the statement and the soft channel ID to the server.

[0086] After receiving the soft channel ID and statement, the client synchronizes the soft channel ID and statement with the server. The server inputs the user statement into the large model corresponding to the user request. The large model outputs a token. After detecting the generation of the first token, the server generates a notification message and sends it to the client's listener, so that the listener continuously monitors the soft channel corresponding to the soft channel ID.

[0087] The server continuously retrieves tokens from the output of the large model and determines whether the number of tokens meets the chunk size (chunke_size), i.e., the first quantity threshold. If the number of tokens equals chunk_size, a chunk_size token is sent to the token outputter. The token outputter is the token segmenter mentioned above.

[0088] The token outputter determines whether the number of chunks sent by the server meets the second threshold. If it does, it sends the second threshold number of chunks to the soft channel corresponding to the soft channel ID, based on the binding relationship between the soft channel ID and the soft channel.

[0089] The listener reads tokens from the soft channel in a loop to generate tokens, and then sends the read tokens to the external client. The external client receives the tokens in a queue, thus achieving streaming output.

[0090] It should be noted that, Figure 2 The illustrated embodiment divides the server side into two functional modules: server-side and client-side. Figure 3 The illustrated embodiment does not divide the server side, such as Figure 3 As shown, after receiving the user's input, the external client generates a user request, constructs the user statement based on the request, and generates a soft channel ID. Then, the external client sends the user statement and the soft channel ID to the server proxy.

[0091] After receiving the soft channel ID and user statement, the server inputs the user statement into the large model corresponding to the user request. The large model outputs a token. After detecting the generation of the first token, the server generates a notification message and sends the notification message to the listener so that the listener can continuously monitor the soft channel corresponding to the soft channel ID.

[0092] The server continuously acquires tokens from the output of the large model and determines whether the number of tokens meets the `chunke_size` threshold. If the number of tokens equals `chunke_size`, it sends a `chunke_size` token to the token outputter. The token outputter is the token segmenter mentioned above.

[0093] The token outputter determines whether the number of chunks sent by the server meets the second threshold. If it does, it sends the second threshold number of chunks to the soft channel corresponding to the soft channel ID, based on the binding relationship between the soft channel ID and the soft channel.

[0094] The listener reads tokens from the soft channel in a loop to generate tokens, and then sends the read tokens to the external client. The external client receives the tokens, thus achieving streaming output.

[0095] It should be noted that, before inputting the first threshold number of tokens into the word segmenter and sending a notification message to the listener in the above-mentioned S101, when the number of tokens generated by the large model is equal to the first threshold number, the method further includes:

[0096] Obtain the first and second quantity thresholds; obtain the tokens generated by the large model; determine whether the number of tokens generated by the large model is equal to the first quantity threshold.

[0097] The first and second quantity thresholds are determined by the server based on the model type, the number of soft channels, network bandwidth information, and basic throughput information. Specifically, they can be determined as follows: Figure 4 The method shown determines the first quantity threshold and the second quantity threshold, such as Figure 4 As shown:

[0098] S401. Obtain the model category information, number of soft channels, network bandwidth information, and basic throughput information corresponding to the large model.

[0099] S402. Input the model category information, number of soft channels, network bandwidth information, and basic throughput information into the output prediction model to obtain the prediction results of the output prediction model.

[0100] The output prediction model can be a regression model from the SciPy Toolkit for Machine Learning (Sklearn). By using Ye Bayes regression, the model category information, number of soft channels, network bandwidth information, and basic throughput information are used to determine the fitting point of the large model generation speed, thereby predicting the generation speed.

[0101] S403. Determine the first quantity threshold and the second quantity threshold based on the prediction results.

[0102] Therefore, before obtaining the tokens generated by the large model, a first quantity threshold and a second quantity threshold can be obtained. These two thresholds can be adjusted based on the prediction results and the actual generation speed. This ensures that the first and second quantity thresholds better align with actual business needs. Consequently, the number of tokens generated by the large model is divided into blocks for transmission based on the first quantity threshold, resulting in more accurate data block sizes that meet actual business requirements and reduce transmission latency.

[0103] like Figure 5 As shown, regarding the above S403, determining the first and second quantity thresholds based on the prediction results can be specifically implemented as follows:

[0104] S4031. If the predicted token generation speed in the prediction result is greater than the actual token generation speed, adjust the historical quantity threshold according to the preset step size to obtain the first quantity threshold.

[0105] Understandably, if the predicted token generation speed in the prediction results is greater than the actual token generation speed, it means that the current server's transmission performance can support the transmission of larger data blocks. In order to improve the transmission rate, the historical quantity threshold can be adjusted according to a preset step size based on the historical quantity threshold to obtain the first quantity threshold.

[0106] S4032. Given a first quantity threshold, calculate a preset multiple of the first quantity threshold to obtain a second quantity threshold.

[0107] The preset multiplier is set based on experience.

[0108] Using the method provided in this application, when the predicted token generation speed in the prediction result is greater than the actual token generation speed, the historical quantity threshold can be adjusted. The upper limit of the threshold is increased based on the historical quantity threshold to obtain a first quantity threshold. Thus, based on the model prediction result, it is determined whether the current data block size is suitable for the current server's transmission performance, thereby achieving dynamic adjustment of the data block size and improving the accuracy of the first and second quantity thresholds.

[0109] In addition to adjusting the data block size based on the prediction results, the server can also adjust the number of locally configured soft channels using the prediction results. Specifically:

[0110] If the predicted token generation rate is higher than the actual token generation rate, the number of soft channels on the server will be reduced by a preset amount. If the predicted token generation rate is lower than or equal to the actual token generation rate, the number of soft channels on the server will be increased by a preset amount.

[0111] Understandably, if the predicted token generation rate is higher than the actual token generation rate, reducing the number of soft channels can reduce the number of user requests the server can process in parallel, thus freeing up more resources for data transmission. Conversely, if the predicted token generation rate is less than or equal to the actual token generation rate, increasing the number of soft channels can increase the number of user requests the server can process in parallel.

[0112] By using the method provided in this application embodiment, the number of soft channels in the server can be dynamically adjusted by comparing the predicted token generation speed in the prediction results with the actual token generation speed. This ensures that the generation speed is maintained while effectively utilizing the server's resources, thereby improving resource utilization efficiency.

[0113] The following combination Figure 6 Introducing the feedback generation capabilities of large model networks, such as Figure 6 As shown, Figure 6 An example is shown of a method for adjusting the number of soft channels based on the output of a regression model.

[0114] Specifically, the model category information, number of soft channels, network bandwidth information, and basic throughput information corresponding to the large model are input into the Sklearn regression model to obtain the predicted tokens / s output by the regression model. Then, based on the predicted tokens / s and the actual generation speed, it is determined whether the predicted tokens / s is higher than the actual generation speed. If it is higher, the number of soft channels is reduced.

[0115] Specifically, in Figure 6 In the example shown, the throughput is 300 tokens / s, the number of soft channels is 10, and the network bandwidth is 4k / s. The predicted tokens / s are obtained by using the AutoSklearnRegressor regression model in Sklearn. AutoSklearnRegressor is a regression model in automated machine learning.

[0116] In some embodiments of this application, when large models are streamed, the time required to complete the output is relatively long. Streaming output is characterized by a small number of characters output per cycle and a large number of output cycles. If the Hypertext Transfer Protocol (HTTP) is used for data transmission, the server needs to push data to external clients multiple times, and each data push requires establishing a connection and protocol parsing. Thus, the total time consumed by multiple data pushes is high. To reduce data transmission latency and alleviate data transmission pressure in high-concurrency scenarios, the soft channel ID and the in-memory database are bound in a first-in, first-out (FIFO) manner, and each user request uses a separate queue as the soft channel transmission token. Based on this, the soft channel includes a list object corresponding to the user request, which is used to store the tokens generated by the large model. Then, in S102 above, the received second threshold number of tokens are sent to the soft channel, such as... Figure 7 As shown, it can be implemented as follows:

[0117] S701. Find the first node by following the first pointer in the list object.

[0118] The list object includes a head pointer and a tail pointer. The first pointer is the head pointer, and the storage space between the storage locations pointed to by the head pointer and the tail pointer is the storage space of the token corresponding to the user request.

[0119] S702. Determine if the first node has a pointer to the next node.

[0120] S703. If a pointer to the next node exists for the first node, find the second node according to the pointer to the next node.

[0121] S704. Take the second node as the first node, and perform the step of determining whether the first node has a next node pointer. If the first node has a next node pointer, follow the next node pointer to find the second node until the first node found does not have a next node pointer.

[0122] Specifically, if the server determines that the current node has a pointer to the next node, it continues to read the next node until the node being read does not have a pointer to the next node.

[0123] S705. Store the second number of tokens into the first node where there is no next node pointer.

[0124] The method provided in this application embodiment, by setting a node pointer at the end of the node, allows searching for the next node according to the node pointer when the current node is full. This enables the storage of the latest generated tokens based on their generation time, achieving a first-in-first-out (FIFO) data flow and improving memory utilization.

[0125] like Figure 8 As shown, Figure 8 The example illustrates the storage structure of an in-memory database in a server, which can provide high-performance, low-latency message transmission capabilities in high-concurrency scenarios.

[0126] Specifically, elements are stored sequentially using a list structure, with each list storing a token corresponding to a user request. Each list object stores a head pointer and a tail pointer, pointing to the beginning and end of the list, respectively.

[0127] The first node is located based on the head pointer. Each node stores a list (clist), and each node has a pointer to that clist. When a new element is received by the soft channel, the server checks if the clist at the insertion position can accommodate the new element. If the clist can store the new element, it is stored directly in the clist. If the clist cannot store the new element, a new node is created, and a pointer to the next node (next node) is generated. The new element is then stored in the new node.

[0128] Each clist includes a size field, a count field, an element field, and an end field. The size field indicates the number of bytes occupied by the clist, the count field indicates the number of elements stored in the clist, and the element field is used to store the elements.

[0129] It should be noted that multiple queues can exist between the token outputter and the listener for data transmission, with one queue corresponding to each soft channel. In this embodiment, Figure 9 Five soft channels are shown as an example.

[0130] Each queue uses a circular array structure, with a size that is a power of 2. For example... Figure 10 As shown, Figure 10This is a logical diagram of a circular array. In practical applications, data can be written and deleted using a cursor and a next pointer.

[0131] Specifically, the cursor pointer is used to point to the last block that has been filled with elements, and the next pointer is used to point to the first block that has not been filled with elements. Figure 10 The example shows the cursor pointer pointing to position 8 and the next pointer pointing to position 9.

[0132] like Figure 11 As shown, if a thread D writes the letter D into the sequence, the server can increment the next pointer based on a compare-and-swap (CAS) read-and-increment operation. Assuming the next pointer currently points to 9, after thread D performs the get-and-increment operation, the next pointer points to 10. Therefore, thread D gains access to position 9 and can write or delete data at position 9.

[0133] like Figure 12 As shown, if thread E also has a data writing requirement, then the above get-and-increment operation will be executed, which will change the next pointer from position 10 to position 11, thus giving thread E the operation permission for position 10.

[0134] The following combination Figure 13 Introducing the complete process of data entering the queue:

[0135] Data entry into the queue process: Assuming the array size is 8 and 8 elements have already been submitted, when the server receives three new elements, these three new elements are written to the queue. First, the cursor pointer determines the position of the last element written to be position 7. The three positions that can be written to are obtained as positions 8, 9, and 10. The remainders of 8, 9, and 10 are then calculated: position 8 has a remainder of 0, position 9 has a remainder of 1, and position 10 has a remainder of 2. The server then retrieves positions 0, 1, and 2 and rewrites the data in these three memory areas. The available buffer positions 0, 1, and 2 are updated to 8, 9, and 10 respectively.

[0136] The data dequeue process is based on the first-in, first-out principle. Data is read from the head position, the data in buffer[head] is retrieved, the head pointer is moved forward one position, and the circular array is processed.

[0137] like Figure 14 As shown, Figure 14 The circular array is expanded to represent the current consumer's consumption queue. The listener requests access to position 8, first using CAS to preempt progress 8 in the shared queue, and then writing progress 7 to the independent queue. In a concurrent scenario, the independent queue refers to the current consumer's consumption queue progress, while the shared queue represents the overall consumption queue progress. Writing progress 7 to the independent queue marks the current consumption queue position.

[0138] Then, obtain the maximum readable position of progress 8. Since the above data enqueue process in the circular array is in the order of position 8, position 9, and position 10, the data stored in position 8 is the earliest data written, so the data stored in position 10 is returned.

[0139] Then, progress 8 is read, and the read data is sent to the consumer program for processing.

[0140] Finally, CAS preemption is performed on progress 9, and the above process is repeated.

[0141] Based on the same concept, embodiments of this application provide a data transmission apparatus, such as... Figure 15 As shown, the device, applied to a server, includes:

[0142] The judgment module 1501 is used to input the first number of tokens into the word segmenter and send a notification message to the listener when the number of tokens generated by the large model is equal to the first number threshold.

[0143] Sending module 1502 is used to send the received second number threshold tokens to a soft channel when the word segmenter determines that the number of received tokens is equal to a second number threshold, wherein the soft channel corresponds to the user request;

[0144] The reading module 1503 is used for the listener to read tokens one by one from the soft channel after receiving the notification message;

[0145] The sending module 1502 is also used for the listener to send the read token to an external client.

[0146] In one possible implementation, the device further includes:

[0147] The receiving module is used to receive the soft channel ID and user request sent by the external client;

[0148] The input module is used to input the statements in the user request into the large model;

[0149] The determination module is used to determine the correspondence between the soft channel ID and the soft channel.

[0150] In one possible implementation, the device further includes:

[0151] The acquisition module is used to acquire the first quantity threshold and the second quantity threshold; and to acquire the token generated by the large model;

[0152] The judgment module 1501 is also used to determine whether the number of tokens generated by the large model is equal to the first quantity threshold.

[0153] In one possible implementation, the acquisition module is also used for:

[0154] Obtain the model category information, number of soft channels, network bandwidth information, and corresponding basic throughput information of the large model;

[0155] The model category information, the number of soft channels, the network bandwidth information, and the basic throughput information are input into the output prediction model to obtain the prediction result of the output prediction model.

[0156] The first quantity threshold and the second quantity threshold are determined based on the prediction results.

[0157] In one possible implementation, the acquisition module is also used for:

[0158] If the predicted token generation speed in the prediction result is greater than the actual token generation speed, the historical quantity threshold is adjusted according to a preset step size to obtain the first quantity threshold.

[0159] Given the first quantity threshold, a preset multiple of the first quantity threshold is calculated to obtain the second quantity threshold.

[0160] In one possible implementation, the device further includes:

[0161] The reduction module is used to reduce the number of soft channels in the server by a preset amount when the predicted token generation speed in the prediction result is greater than the actual token generation speed.

[0162] An additional module is added to increase the number of soft channels in the server by a preset amount when the predicted token generation speed in the prediction result is less than or equal to the actual token generation speed.

[0163] In one possible implementation, the soft channel includes a list object corresponding to the user request, the list object being used to store the token generated by the large model; the sending module 1502 is used for:

[0164] Find the first node using the first pointer in the list object;

[0165] Determine whether the first node has a pointer to the next node;

[0166] If the next node pointer exists at the first node, the second node is found according to the next node pointer;

[0167] Using the second node as the first node, the steps of determining whether the first node has a next node pointer and searching for the second node according to the next node pointer are executed until the first node found does not have a next node pointer.

[0168] Store the second threshold number of tokens into the first node, which does not have a next node pointer.

[0169] It should be noted that the resource scheduling device is the same as the resource scheduling method described above. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0170] Figure 16 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0171] The electronic device may include a processor 1601 and a memory 1602 storing computer program instructions.

[0172] Specifically, the processor 1601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0173] Memory 1602 may include mass storage for data or instructions. For example, and not limitingly, memory 1602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1602 may include removable or non-removable (or fixed) media. Where appropriate, memory 1602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1602 is non-volatile solid-state memory.

[0174] In a particular embodiment, memory 1602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0175] The processor 1601 implements any of the data transmission methods described in the above embodiments by reading and executing computer program instructions stored in the memory 1602.

[0176] In one example, the electronic device may also include a communication interface 1603 and a bus 1604. For example, Figure 16 As shown, the processor 1601, memory 1602, and communication interface 1603 are connected through bus 1604 and complete communication with each other.

[0177] The communication interface 1603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0178] Bus 1604 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1604 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0179] Furthermore, in conjunction with the data transmission methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data transmission methods in the above embodiments.

[0180] This application also provides a computer program product, including a computer program, which, when executed, implements any of the data transmission methods described in the above embodiments.

[0181] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0182] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on machine-readable media or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0183] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0184] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0185] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for data transmission, characterized in that, Applied to a server, the method includes: If the number of tokens generated by the large model is equal to the first number threshold, the first number threshold of tokens is input into the word segmenter and a notification message is sent to the listener. The first number threshold is determined based on the speed at which the large model generates tokens. If the word segmenter determines that the number of received tokens is equal to the second threshold, the second threshold number of received tokens will be sent to the soft channel, which corresponds to the user request. After receiving the notification message, the listener reads the tokens one by one from the soft channel; The listener sends the read token to the external client.

2. The method according to claim 1, characterized in that, Before inputting the first threshold number of tokens into the tokenizer and sending a notification message to the listener, the method further includes: Receive the soft channel ID and user request sent by the external client; Input the statements in the user request into the large model; Determine the correspondence between the soft channel ID and the soft channel.

3. The method according to claim 1, characterized in that, Before inputting the first threshold number of tokens into the token segmenter and sending a notification message to the listener, when the number of tokens generated in the large model is equal to a first threshold number, the method further includes: Obtain the first quantity threshold and the second quantity threshold; Obtain the token generated by the large model; Determine whether the number of tokens generated by the large model is equal to the first quantity threshold.

4. The method according to claim 3, characterized in that, Obtaining the first quantity threshold and the second quantity threshold includes: Obtain the model category information, number of soft channels, network bandwidth information, and corresponding basic throughput information of the large model; The model category information, the number of soft channels, the network bandwidth information, and the basic throughput information are input into the output prediction model to obtain the prediction result of the output prediction model. The first quantity threshold and the second quantity threshold are determined based on the prediction results.

5. The method according to claim 4, characterized in that, Determining the first quantity threshold and the second quantity threshold based on the prediction result includes: If the predicted token generation speed in the prediction result is greater than the actual token generation speed, the historical quantity threshold is adjusted according to a preset step size to obtain the first quantity threshold. Given the first quantity threshold, a preset multiple of the first quantity threshold is calculated to obtain the second quantity threshold.

6. The method according to claim 4, characterized in that, After determining the first quantity threshold and the second quantity threshold based on the prediction result, the method further includes: If the predicted token generation speed in the prediction results is greater than the actual token generation speed, the number of soft channels in the server will be reduced by a preset amount. If the predicted token generation speed in the prediction results is less than or equal to the actual token generation speed, the number of soft channels in the server will be increased by a preset amount.

7. The method according to claim 1, characterized in that, The soft channel includes a list object corresponding to the user request, and the list object is used to store the token generated by the large model. Sending the received second threshold number of tokens to the soft channel includes: Find the first node using the first pointer in the list object; Determine whether the first node has a pointer to the next node; If the next node pointer exists at the first node, the second node is found according to the next node pointer; Using the second node as the first node, the steps of determining whether the first node has a next node pointer and searching for the second node according to the next node pointer are executed until the first node found does not have a next node pointer. Store the second threshold number of tokens into the first node, which does not have a next node pointer.

8. A data transmission apparatus, characterized in that, Applied to a server, the device includes: The judgment module is used to input the first number of tokens into the word segmenter and send a notification message to the listener when the number of tokens generated by the large model is equal to the first number threshold. The first number threshold is determined based on the speed at which the large model generates tokens. The sending module is used to send the received second-threshold number of tokens to a soft channel when the word segmenter determines that the number of received tokens is equal to a second-threshold number, wherein the soft channel corresponds to the user request; The reading module is used by the listener to read tokens one by one from the soft channel after receiving the notification message; The sending module is also used for the listener to send the read token to an external client.

9. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data transmission method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the data transmission method as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device causes the electronic device to perform the data transmission method as described in any one of claims 1-7.

Images