Request processing methods, apparatus, devices, storage media, and computer program products

By reading multimodal data from the local disk, encoding it, and persisting it, the memory consumption problem of the multimodal inference client is solved, and the performance and concurrency of the multimodal inference client are improved.

CN120704875BActive Publication Date: 2026-11-17TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510793972.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-06-13
Publication Date
2026-11-17
Estimated Expiration
2045-06-13

AI Technical Summary

Technical Problem

Multimodal inference clients consume large amounts of memory in local memory for extended periods, impacting terminal device performance and limiting the number of concurrent requests a single machine can handle.

Method used

Multimodal data is read from the local disk into local memory for encoding. After generating a multimodal inference request, the encoded data is persisted to the disk, the memory is released, and a new request is initiated if the request fails.

Benefits of technology

This reduces the memory footprint of the multimodal inference client, avoids performance impact, and increases the number of concurrent requests that a single machine can handle.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120704875B_ABST
    Figure CN120704875B_ABST
Patent Text Reader

Abstract

This application relates to a request processing method, apparatus, device, storage medium, and computer program product. The method includes: reading multimodal data from a local disk into local memory and encoding the data to obtain encoded data; generating and sending a multimodal inference request based on the encoded data; persisting the encoded data to the local disk and then releasing the multimodal data and corresponding encoded data from the local memory, wherein the encoded data on the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of request failure; and receiving the inference result fed back in response to the multimodal inference request. This method can reduce the memory footprint of the multimodal inference client, thereby avoiding performance impact on the terminal device where the multimodal inference client is located, and can also increase the number of concurrent requests that a single machine can handle.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a request processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Using a multimodal large language model to perform inference tasks on input multimodal data can yield more accurate inference results. Currently, for inference tasks generated from multimodal data, the response time of the modal inference server ranges from tens to hundreds of seconds. During this time, the multimodal data read by the multimodal inference client is stored in local memory, occupying local memory for a long time. Multimodal data such as images and videos consume even more memory. This not only affects the performance of the terminal device where the multimodal inference client is located, but also, if concurrent multimodal inference requests are made on the multimodal inference client, it will generate a huge memory consumption on the multimodal inference client, which will limit the number of concurrent requests that a single machine can handle. Summary of the Invention

[0003] Therefore, it is necessary to provide a request processing method, apparatus, device, storage medium, and computer program product to address the above-mentioned technical problems. This method can reduce the local memory usage of the multimodal inference client, thereby avoiding the impact on the performance of the terminal device where the multimodal inference client is located, and can also increase the number of concurrent requests that a single machine can handle.

[0004] Firstly, this application provides a request processing method. The method includes:

[0005] Multimodal data is read from the local disk and encoded in local memory to obtain encoded data;

[0006] Generate and send multimodal inference requests based on the encoded data;

[0007] After persisting the encoded data to the local disk, the multimodal data and corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0008] Receive the inference results in response to the multimodal inference request.

[0009] Secondly, this application also provides a request processing apparatus. The apparatus includes:

[0010] The data acquisition module is used to read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0011] The request processing module is used to generate and send multimodal inference requests based on the encoded data;

[0012] The data processing module persists the encoded data to the local disk and then releases the multimodal data and corresponding encoded data in the local memory. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0013] The inference result receiving module is used to receive the inference results fed back in response to multimodal inference requests.

[0014] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0015] Multimodal data is read from the local disk and encoded in local memory to obtain encoded data;

[0016] Generate and send multimodal inference requests based on the encoded data;

[0017] After persisting the encoded data to the local disk, the multimodal data and corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0018] Receive the inference results in response to the multimodal inference request.

[0019] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0020] Multimodal data is read from the local disk and encoded in local memory to obtain encoded data;

[0021] Generate and send multimodal inference requests based on the encoded data;

[0022] After persisting the encoded data to the local disk, the multimodal data and corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0023] Receive the inference results in response to the multimodal inference request.

[0024] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0025] Multimodal data is read from the local disk and encoded in local memory to obtain encoded data;

[0026] Generate and send multimodal inference requests based on the encoded data;

[0027] After persisting the encoded data to the local disk, the multimodal data and corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0028] Receive the inference results in response to the multimodal inference request.

[0029] The aforementioned request processing method, apparatus, computer device, storage medium, and computer program product read multimodal data from the local disk into local memory, encode the data to obtain encoded data, generate and send multimodal inference requests based on the encoded data, persist the encoded data to the local disk, and then release the multimodal data and corresponding encoded data from the local memory. By releasing the encoded data from the local memory, the system avoids data occupying local memory for a long time, reducing the impact on the request processing performance of the terminal device where the multimodal inference client is located. Furthermore, in the event of a request failure, the system can re-initiate the multimodal inference request based on the encoded data persisted to the local disk, thereby receiving the inference results fed back from the multimodal inference request. Thus, while avoiding the occupation of local memory, the system can also ensure data support for the generation and processing of multimodal inference requests. This reduces the local memory usage of the multimodal inference client, thereby preventing the performance of the terminal device where the multimodal inference client is located from being affected. Moreover, since it does not affect the generation and sending of multimodal inference requests, it increases the number of concurrent requests that a single machine can handle. Attached Figure Description

[0030] Figure 1 This is a diagram illustrating the local memory usage state when processing multimodal inference requests in related technologies.

[0031] Figure 2 This is a diagram illustrating the local memory usage state when concurrently processing multimodal inference requests in related technologies.

[0032] Figure 3 This is an application environment diagram of a request processing method in one embodiment;

[0033] Figure 4 This is a simplified flowchart of a request processing method in one embodiment;

[0034] Figure 5 This is a flowchart illustrating a request processing method in one embodiment;

[0035] Figure 6 This is a schematic diagram illustrating the local memory usage state when processing multimodal inference requests in one embodiment;

[0036] Figure 7This is a schematic diagram illustrating the local memory usage state when concurrently processing multimodal inference requests in one embodiment.

[0037] Figure 8 This is a flowchart illustrating the request processing method in another embodiment;

[0038] Figure 9 This is an example diagram illustrating the coded data and guidance text stored in local memory for multimodal data in one embodiment.

[0039] Figure 10 This is a flowchart illustrating the request processing method in yet another embodiment;

[0040] Figure 11 This is a structural example diagram of storing multiple request arrays in an array manner in one embodiment;

[0041] Figure 12 This is a complete flowchart of a request processing method in one embodiment;

[0042] Figure 13 This is a structural block diagram of a request processing device in one embodiment;

[0043] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] Using a multimodal large language model to perform inference tasks on input multimodal data can yield more accurate inference results. In related technologies, for inference tasks generated from multimodal data, the response time of the modal inference server varies from tens to hundreds of seconds. During this period, the multimodal data read by the multimodal inference client is stored in local memory, occupying local memory for an extended period. Specifically, for example… Figure 1 As shown. Figure 1 This diagram illustrates the local memory usage during the processing of a multimodal inference request. (Refer to...) Figure 1 The horizontal axis represents the different stages from when the multimodal inference client generates and sends an inference request to when the modal inference server returns the inference result. The vertical axis represents the local memory usage status of the device where the multimodal inference client is located. A blank background indicates low memory usage at the corresponding stage, while a dark slash indicates high memory usage at the corresponding stage. (See reference...) Figure 1As can be seen, in the relevant technologies, when the client generates a multimodal inference request, it needs to occupy local memory for multimodal data reading and encoding. After the multimodal inference request is sent to the inference server, the multimodal data still occupies local memory while waiting for the inference result to be returned, resulting in a long time for local memory to be occupied. Moreover, since multimodal data such as images and videos occupy even more memory, this will inevitably affect the performance of the terminal device where the multimodal inference client is located.

[0046] Furthermore, if concurrent multimodal inference requests are made on the multimodal inference client, specifically as follows: Figure 2 As shown, refer to Figure 2 This diagram illustrates the local memory usage during concurrent multimodal inference requests in related technologies. As can be seen, when multiple multimodal inference requests are concurrent, the multimodal data required by each multimodal inference request will occupy a large amount of local memory for a long time. With multiple multimodal inference requests occupying local memory at any given moment, a huge amount of memory is generated on the multimodal inference client at any given moment, which greatly limits the number of concurrent requests that a single machine can handle.

[0047] To address the aforementioned issues, embodiments of this application provide a request processing method that reduces the local memory footprint of the multimodal inference client, thereby preventing performance degradation of the terminal device hosting the multimodal inference client, and also increases the number of concurrent requests that a single machine can handle. The request processing method provided in this application can be applied to applications such as... Figure 3 The application environment is shown. Terminal 302 communicates with server 304 via a network. A multimodal inference client is deployed on terminal 302, and server 304 specifically responds to multimodal inference requests initiated by terminal 302 and provides corresponding inference results. A data storage system can store the data that server 304 needs to process. The data storage system can be integrated on server 304 or located in the cloud or on another inference server.

[0048] Specifically, taking the application deployed on terminal 302 as an example, terminal 302 reads multimodal data from the local disk into local memory, encodes the data to obtain encoded data, and generates and sends a multimodal inference request to server 304 based on the encoded data. After persisting the encoded data to the local disk, terminal 302 releases the multimodal data and corresponding encoded data from local memory. Then, terminal 302 receives the inference result fed back by server 304 in response to the multimodal inference request. By releasing the encoded data from local memory to avoid data occupying local memory for a long time, the impact on the request processing performance of the terminal device where the multimodal inference client is located is reduced. In the event of a request failure, the multimodal inference request can be re-initiated based on the encoded data persisted to the local disk. Thus, while avoiding the occupation of local memory, it can also ensure data support for the generation and processing of multimodal inference requests. This reduces the local memory usage of the multimodal inference client, thereby avoiding the impact on the performance of the terminal device where the multimodal inference client is located. Furthermore, since it does not affect the generation and sending of multimodal inference requests, it increases the number of concurrent requests that can be handled by a single machine.

[0049] The terminal 302 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 304 can be implemented using a standalone inference server or a cluster of inference servers. The request processing method provided in this embodiment can be applied to various scenarios, including but not limited to scenarios using cloud technology and artificial intelligence for processing multimodal inference requests.

[0050] Since the request processing method provided in this application embodiment is specifically applied to a scenario of processing multimodal reasoning requests, and involves related technologies such as multimodal reasoning and multimodal models, the technical terms of the related technologies are explained below:

[0051] 1. Multimodal large language model.

[0052] Multimodal large language models are extensions of large language models (LLM) that are capable of receiving and reasoning about multimodal (image, audio, video) information.

[0053] 2. Multimodal reasoning.

[0054] Multimodal reasoning is a reasoning task performed on multimodal inputs using a multimodal large language model.

[0055] 3. Reasoning request.

[0056] An inference request is a Hypertext Transfer Protocol (HTTP) request that encapsulates multimodal inference data. An inference request includes at least a prompt instructing the inference process on the multimodal data, encoded data of the multimodal input (e.g., binary encoded data of image data, binary encoded data of audio data, binary encoded data of video data), and is specifically stored in the HTTP request body in JSON format.

[0057] 4. Client-side application.

[0058] In this embodiment, the client is a multimodal inference client used to send multimodal inference requests. Furthermore, the client, through multithreading (processes), coroutines, or other methods, can simultaneously send multiple multimodal inference requests in a short period of time, and receive the inference results of each multimodal inference request synchronously or asynchronously.

[0059] To facilitate understanding, the following describes the process of handling multimodal inference requests in this embodiment. Figure 4 As shown, multimodal data is first read into local memory, then encoded to obtain encoded data, which is also saved to local memory. Based on this, a multimodal inference request is generated and sent. After generating the multimodal inference request, the encoded data is saved to the local disk, and the multimodal data and corresponding encoded data in local memory are released. Then, it is determined whether the inference result for the multimodal inference request has been returned. If the request fails, the encoded data needs to be read from the local disk, and a multimodal inference request is generated based on the encoded data in a similar manner as described above. If the request succeeds, the corresponding encoded data of the multimodal data stored on the local disk is deleted, and the returned inference result is received, completing the processing of this multimodal inference request.

[0060] The following examples illustrate this in detail: In one example, as... Figure 5 As shown, a request processing method is provided, which is applied to Figure 3 Taking terminal 302 as an example, it can be understood that this method can also be applied to a system including terminal 302 and server 304, and implemented through the interaction between terminal 302 and server 304. In this embodiment, the method includes the following steps:

[0061] Step 502: Read multimodal data from the local disk into local memory and encode the data to obtain encoded data.

[0062] Multimodal data refers to data that simultaneously contains two or more different modalities. These modalities typically include visual, linguistic, and sensor modalities. Visual modal data includes image data, video data, etc., while linguistic modal data includes text data, speech data, audio data, etc. Therefore, multimodal data must include at least two different modalities. For example, multimodal data includes image data and video data, or multimodal data includes image data and text data. The content of multimodal data is not limited here.

[0063] Secondly, the local disk refers to the storage space allocated within the terminal system's internal hard drive for data storage. In other words, "local" is obtained by partitioning the terminal system's internal hard drive. Typically, the local disk is used as the default storage space in the terminal system. Therefore, any hard drive partition within the terminal system can be defined as the local disk in this embodiment. Local memory is random access memory, meaning it's a high-speed register in the terminal system used to store information about currently running programs for execution by the Central Processing Unit (CPU). In this embodiment, the local memory stores the necessary multimodal data and encoded data when processing multimodal execution requests. The data stored in the local memory is dynamic; the terminal system's task calls can dynamically retrieve and release data from the local memory. The encoded data in this embodiment is used to generate multimodal inference requests for multimodal inference tasks.

[0064] Specifically, when there is a need for multimodal inference, a trigger operation for multimodal inference can be initiated on the terminal via an inference request object. This trigger operation enables the terminal to initiate and execute multimodal inference. Considering that multimodal inference is a task performed on multimodal input using a multimodal language model, when invoking the multimodal language model for inference, it is necessary to first obtain the multimodal data for multimodal inference. For the terminal, the local disk is the storage space used for data storage; that is, the terminal typically stores data on the local disk. Therefore, the terminal reads the multimodal data required for executing the multimodal inference task from the local disk. In other words, the terminal can first obtain the data identifier of the required multimodal data through the multimodal inference trigger operation, locate the required multimodal data from the data stored on the local disk using the data identifier, and then read the multimodal data from the local disk into local memory.

[0065] Based on this, the terminal saves the read multimodal data to local memory, encodes the acquired multimodal data to obtain encoded data, and stores the encoded data in local memory. At this point, local memory stores multimodal data and its corresponding encoded data. It is understandable that in practical applications, multimodal data can also be data input in real-time by the inference request object; that is, the inference request object inputs multimodal data to the local disk in real-time and performs trigger operations for multimodal inference, thereby enabling the terminal to directly determine the multimodal data input in real-time by the inference request object from the local disk and read the multimodal data from the local disk into local memory.

[0066] Furthermore, since the encoded data is used to generate multimodal inference requests for multimodal inference tasks, and the generated multimodal inference requests are HTTP requests that encapsulate multimodal inference data, and the data format for data encapsulation in HTTP requests is specifically binary, the data encoding process in this embodiment needs to consider the format requirements of request encapsulation. The data encoding process is described in detail below:

[0067] In one specific embodiment, reading multimodal data from a local disk into local memory and encoding the data to obtain encoded data includes: reading multimodal data from a local disk in binary format and storing the binary-formatted multimodal data in local memory; performing data conversion on the binary-formatted multimodal data to obtain a multimodal string, wherein the multimodal string is encoded data.

[0068] Specifically, the terminal reads multimodal data from the local disk in binary format; that is, the terminal reads multimodal data in binary format from the local disk and stores the read multimodal data in local memory. In practical applications, the terminal uses the Pickle module in Python or other serialization tools to convert the multimodal data into binary format and write it to local memory. In other words, the Pickle module creates a Python-specific binary format, which can perform read and write operations. Therefore, the Pickle module can read multimodal data in binary format from the local disk.

[0069] Then, the terminal performs data conversion on the binary format multimodal data to obtain a multimodal string. The aforementioned multimodal string is determined as encoded data, and the data conversion is specifically performed through Base64 encoding. Base64 encoding is used to convert binary data into American Standard Code for Information Interchange (ASCII) strings. Therefore, in this embodiment, the data conversion is specifically performed by Base64 encoding the binary format multimodal data to obtain a multimodal string (i.e., an ASCII string).

[0070] Furthermore, considering the need to generate multimodal inference requests from encoded data, the terminal not only needs to store the encoded data in local memory, but also needs to put the encoded data into the serialized structure of the request input.

[0071] Step 504: Generate and send a multimodal inference request based on the encoded data.

[0072] Specifically, as described in the foregoing embodiments, multimodal data and its corresponding encoded data are stored in local memory. After the terminal completes the reading and encoding of the multimodal data, it first starts a multimodal inference process to handle the multimodal inference task. This process is used to generate and send multimodal inference requests. There can be one or multiple multimodal inference processes, depending on the actual task requirements. The terminal then generates a multimodal inference request based on the encoded data stored in local memory. This involves encapsulating the encoded data into the request input to obtain the multimodal inference request. Specifically, the encoded data is stored in the HTTP request body and encapsulated to obtain the multimodal inference request. After generating the multimodal inference request, the terminal sends it to the inference server. The inference server responds to the request, extracts the encoded data from the request, and then calls the multimodal large language model to perform the inference result.

[0073] Based on this, if the inference server can successfully invoke the multimodal large language model to perform inference tasks on the encoded data of the multimodal data and obtain the inference result of the multimodal inference request, the inference server will send the inference result of the multimodal inference request back to the terminal. At this time, the terminal can execute step 508 to receive the inference result sent back by the inference server for the multimodal inference request. Conversely, if the inference server cannot successfully invoke the multimodal large language model to perform inference tasks on the encoded data of the multimodal data, that is, the inference server cannot send the inference result of the multimodal inference request back to the terminal, considering that the multimodal inference request may have failed to be sent, or that there are data errors or data loss in the encoded data carried in the multimodal inference request, the terminal can regenerate based on the encoded data and send the multimodal inference request to the inference server, so that the inference server can re-invoke the multimodal large language model to perform inference tasks on the encoded data of the multimodal data.

[0074] Step 506: After persisting the encoded data to the local disk, release the multimodal data and corresponding encoded data in the local memory. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0075] Specifically, after the terminal generates a multimodal inference request, the corresponding encoded data of the multimodal data can be persisted to the local disk. This means storing the encoded data of the multimodal data in the local disk and releasing the multimodal data and its corresponding encoded data from local memory. For example, multimodal data A1 is retrieved from the local disk and stored in local memory, and encoded to obtain encoded data B1 of multimodal data A1, which is then saved to local memory. At this point, local memory contains both multimodal data A1 and encoded data B1. Then, after the terminal generates a multimodal inference request C1 for multimodal data A1 based on encoded data B1, the encoded data B1 can be persisted to the local disk, and the multimodal data A1 and encoded data B1 from local memory can be released. This avoids multimodal data that consumes a significant amount of memory occupying local memory for extended periods.

[0076] As can be seen from the foregoing embodiments, if the inference server fails to successfully invoke the multimodal large language model to perform inference tasks on the encoded data of the multimodal data—that is, if the inference server cannot provide feedback to the terminal regarding the multimodal inference request—the terminal can regenerate and send a multimodal inference request to the inference server based on the encoded data. This allows the inference server to re-invoke the multimodal large language model to perform inference tasks on the encoded data of the multimodal data. Considering that the terminal releases the multimodal data and corresponding encoded data from its local memory after generating the multimodal inference request, and that the encoded data stored on the local disk can be used to re-initiate the multimodal inference request based on the encoded data in case of request failure, the regeneration of the multimodal inference request specifically requires the use of the encoded data on the local disk. This situation will be described in detail below:

[0077] In an optional embodiment, after generating and sending a multimodal inference request based on the encoded data, the request processing method further includes: if no inference result is received from the inference server in response to the multimodal inference request within a preset time interval, rereading the encoded data stored on the local disk, generating and sending a multimodal inference request to the inference server based on the reread encoded data.

[0078] The preset time interval represents the maximum time interval between receiving the inference result from the inference server in response to the multimodal inference request. In other words, the failure of the multimodal inference request can be determined after the preset time interval has elapsed. The preset time interval can be 3 seconds, 5 seconds, etc. The specific duration of the preset time interval needs to be flexibly determined based on the communication between the terminal and the inference server, as well as the actual needs of the multimodal inference request.

[0079] Specifically, after the terminal generates and sends a multimodal inference request based on the encoded data, a timer is started for the multimodal inference request. The start timestamp of the timer is the timestamp when the terminal sends the multimodal inference request to the inference server. Then, a request-response timer is started for the multimodal inference request. If no inference result is received from the inference server within a preset time interval, it indicates a possible problem: the multimodal inference request failed to send; there may be data errors or data loss in the encoded data carried in the multimodal inference request; or the inference server may be unable to call the multimodal large language model to perform inference tasks based on the encoded data. In this case, the terminal can regenerate and send the multimodal inference request to the inference server based on the encoded data. Since the terminal has released the multimodal data and corresponding encoded data from its local memory and persisted the encoded data to its local disk after generating the multimodal inference request, the terminal needs to reread the encoded data stored on the local disk and, according to the request generation method described in the previous embodiment, regenerate the multimodal inference request based on the reread encoded data, and then send the regenerated multimodal inference request to the inference server.

[0080] To facilitate understanding of the optimization of local memory usage through the aforementioned process, such as Figure 6 The diagram shown illustrates the local memory usage of multimodal data and corresponding encoded data during request processing. Figure 6 The middle horizontal axis represents the memory usage time axis corresponding to the use of local memory, and Figure 6 A blank background indicates that the local memory usage is relatively small. Figure 6 When the text is highlighted with a dark slash, it indicates a large amount of local memory is used. Therefore, during the generation and transmission of multimodal inference requests, as described in the preceding embodiments, the encoded information originally stored in local memory is persisted to the local disk after the multimodal inference request is sent, and the encoded information and multimodal data in local memory are released. When a multimodal inference request fails and needs to be resent, the encoded information is reread from the local disk to generate the multimodal inference request. Therefore, the time that multimodal data and encoded data occupy in local memory is limited to the period from assembling the multimodal inference request to sending it; there is no need to wait for the inference result to return. Thus, compared to [previous embodiments], this embodiment offers advantages in [the following context]. Figure 1The memory usage diagram shown reduces the time spent occupying local memory in the memory usage timeline. In other words, this embodiment does not occupy local memory for extended periods while waiting for the inference result of the multimodal inference request. Instead, after sending the multimodal inference request, it persists the encoded data to the local disk and releases the multimodal data and corresponding encoded data from local memory. Therefore, the corresponding data in local memory is released before the inference result is received. Thus, through... Figure 6 It can be seen that since the storage time interval between multimodal data and corresponding encoded data in local memory is only the time interval between obtaining multimodal data and sending multimodal inference request, there is no need to wait for the feedback of inference result for multimodal inference request, thereby reducing the memory usage of the local multimodal inference client.

[0081] Based on this, in the case of multiple concurrent multimodal inference requests, the data in local memory can be released for each multimodal inference request.

[0082] For ease of understanding, it is described by formula (1). Figure 2 The local memory usage during concurrent multimodal inference requests in the related technologies shown, that is, the local memory usage at any timestamp, is illustrated in the example of formula (1):

[0083] ;Formula (1)

[0084] in, This indicates the local memory usage at any given timestamp. This indicates the concurrency level of concurrent multimodal inference requests, meaning there are N multimodal inference requests being processed concurrently. This represents the cumulative time from the generation of a multimodal inference request to the receipt of the inference result, where n represents the percentage of time from sending the multimodal inference request to the return of the inference result. represents the memory size occupied by the multimodal inference request, and g represents the other memory usage of the multimodal inference request. In practical general scenarios, n >= 99%, meaning that the inference results of multimodal inference requests waiting to be returned at the same timestamp are far more than the multimodal inference requests that are being assembled but not yet sent. Furthermore, the memory size occupied by the multimodal inference request is much larger than the other memory usage of the multimodal inference request. Therefore, after simplifying formula (1), it is equivalent to formula (2):

[0085] ;Formula (2)

[0086] in, This indicates the local memory usage at any given timestamp. This indicates the request concurrency of concurrent multimodal inference requests. This represents the cumulative time from the generation of a multimodal inference request to the receipt of the inference result, where n represents the percentage of time from sending the multimodal inference request to the return of the inference result. This indicates the memory size occupied by the multimodal inference request.

[0087] It is evident that memory usage M is positively correlated with cumulative time T, number of concurrent users N, and time percentage n. When the number of concurrent users N is very high, or when network latency leads to a high cumulative time T, or when the inference problem is very complex leading to a high time percentage n, local memory will be consumed in large quantities.

[0088] like Figure 7 The image shown illustrates the local memory usage of concurrent multimodal inference requests after optimized request processing according to the embodiments of this application. Figure 7 The middle horizontal axis represents the memory usage time axis corresponding to the use of local memory, and Figure 7 A blank background indicates that the local memory usage is relatively small. Figure 7 Dark slashes indicate higher local memory usage. Since the multimodal data and corresponding encoded data in local memory are released after each multimodal inference request is sent, the local memory usage of each multimodal inference request decreases. This ensures that the generation and sending of multimodal inference requests are not affected during parallel processing, increasing the number of concurrent multimodal inference requests that can be processed, and thus increasing the number of concurrent requests that a single machine can handle. Figure 7 As can be seen, since the time period that occupies a large amount of local memory is reduced, n in formula (2) will also decrease. Therefore, the local memory usage after optimization is determined as shown in formula (3):

[0089] ;Formula (3)

[0090] in, This indicates the local memory usage at any given timestamp. This indicates the request concurrency of concurrent multimodal inference requests. This represents the cumulative time from the generation of a multimodal inference request to the receipt of the inference result. This represents the percentage of time from sending the multimodal inference request to receiving the inference result in this embodiment. This represents the memory size occupied by the multimodal inference request. From formulas (3) and (2), it can be seen that the memory optimization ratio of the method provided in this embodiment is specifically as follows: - 100%.

[0091] In practical applications, since the encoded data of multimodal inputs is stored in the HTTP request body in JSON format, and the HTTP request body in local memory exists as a Python object, this embodiment manages object memory in Python using reference counting and cyclic garbage collection (GC). Reference counting maintains a reference count for each object, and when the reference count reaches zero, the object is reclaimed, meaning the memory occupied by that object is released. Therefore, when the reference count reaches zero, the corresponding object is released. However, object reclamation in Python requires ensuring that the reclaimed object is no longer needed. Premature object cleanup may cause program crashes, while failing to release objects will cause memory leaks. Therefore, the following two points need to be implemented: 1. Avoid circular references in the HTTP request body. When circular references occur, generational cyclic release of the corresponding objects is required. 2. Manually delete and reduce the reference count at the end of the HTTP request body's lifecycle. This is because Python does not have standard lexical scoping, and objects within the release function need to be manually triggered.

[0092] Based on this, since the HTTP request body in local memory exists in the form of a Python object, the above two points should be considered to release the multimodal data and corresponding encoded data in local memory. The following will describe in detail the specific implementation of how to release the multimodal data and corresponding encoded data in local memory: In an optional embodiment, the request processing method further includes: generating a first local variable for the multimodal data and generating a second local variable for the encoded data, wherein the first reference count of the first local variable and the second reference count of the second local variable are both preset values.

[0093] In practical applications, a memory optimization example for multimodal inference on cloud virtual machines (CVM) is presented, combining formulas (2) and (3). Table 1 shows a comparison of request processing effects between related technologies and the technical solutions provided in this embodiment during CVM modal inference:

[0094] Concurrency Local memory usage Average memory usage Optimization ratio Related technologies 7000 200G 0.02G This embodiment 12000 40G 0.003G 757%

[0095] Table 1

[0096] As shown in Table 1, the technical solution provided in this embodiment can improve the concurrency of multimodal inference requests on the same local memory, that is, increase the number of concurrent multimodal inference requests, and reduce the local memory occupied by concurrent multimodal inference requests, with the memory optimization ratio increased to 757%.

[0097] Specifically, in Python, object memory reclamation works as follows: An object is created, and then another object referenced by a local variable `a` is created. At this point, the reference count of the local variable `a` is 1. Similarly, a new object is created, and then another new object referenced by a local variable `b` is created. The reference count of the local variable `b` referencing the new object is 1. Since local variable `b` is returned, and the new object referenced by `b` is the return value, the reference count of the local variable `b` referenced by the new object increases from 1 to 2. When the function executing on the object and the new object completes, the lifetimes of both the local variable `a` and the local variable `b` referencing the new object end upon function completion. This means that local variables `a` and `b` are in an invalid state. Therefore, the reference count of the local variable `a` is decremented by 1, and the reference count of the local variable `b` referenced by the new object is also decremented by 1. At this point, the reference count of the object pointed to by local variable `a` decreases from 1 to 0, indicating that the object pointed to by local variable `a` is no longer needed. Therefore, the memory of this object can be reclaimed, i.e., the object is released. Since the new object pointed to by the local variable b is the return value of the function, its reference count is 2. Therefore, the reference count of the new object pointed to by the local variable b is reduced by 1, from 2 to 1. So, the new object still cannot be reclaimed at this time.

[0098] Normally, when a function exits, the reference count of all local variables is subtracted. In this embodiment, the subtraction can also be performed manually at the location where memory needs to be released using Python syntax. That is, when the local variable 'a' is no longer needed, the reference count of the object pointed to by the local variable 'a' can be manually subtracted using the delete (del) syntax, and the memory occupied by the object pointed to by the local variable 'a' can be reclaimed.

[0099] In this embodiment, after generating and sending the multimodal inference request, the corresponding function will only end after receiving the inference result fed back to the multimodal inference request. Since the data objects in memory need to be released before receiving the inference result, it is necessary to perform a subtraction operation on the local variables that point to the multimodal data and the encoded data respectively in advance, so as to release the multimodal data and the encoded data in the local memory in advance.

[0100] Specifically, when the terminal reads multimodal data from the local disk into local memory, it is actually reading the multimodal data into the HTTP request body in local memory. At this point, a first local variable for the multimodal data is generated. Similarly, when the terminal encodes the multimodal data in local memory to obtain encoded data, it is also storing the encoded data into the HTTP request body in local memory. At this point, a second local variable for the encoded data is generated. Both the first reference count of the first local variable and the second reference count of the second local variable are preset values, typically 1. That is, the first reference count of the first local variable is specifically 1, and the second reference count of the second local variable is specifically 1.

[0101] Furthermore, since data in local memory is only released when the reference count is 0, the release of multimodal data and corresponding encoded data in local memory specifically includes: gradually adjusting the first reference count from a preset value to zero and releasing the multimodal data in local memory; gradually adjusting the second reference count from a preset value to zero and releasing the encoded data in local memory.

[0102] Specifically, the terminal performs a subtraction operation on the first reference count, deducting 1 from the first reference count with each subtraction operation, gradually adjusting the first reference count from a preset value to zero. For example, if the first reference count is 1, subtracting 1 changes it from 1 to 0, thereby releasing the multimodal data in the local memory pointed to by the gradually adjusted zero first reference count. Similarly, the terminal performs a subtraction operation on the second reference count, deducting 1 from the second reference count with each subtraction operation, gradually adjusting the second reference count from a preset value to zero. For example, if the second reference count is 2, subtracting 1 changes it from 2 to 1. The terminal can then continue subtracting 1 from the updated second reference count, changing it from 1 to 0, thereby releasing the encoded data in the local memory pointed to by the gradually adjusted zero second reference count.

[0103] Furthermore, in practical applications, deletion syntax can be used to release the memory occupied by multimodal data and encoded data in local memory in advance. This means directly deleting the first reference count and the second reference count, which means directly adjusting the first reference count and the second reference count to 0, so that the multimodal data and encoded data in local memory are released after the multimodal inference request is sent.

[0104] As mentioned above, circular references exist in reference counting, meaning two different objects point to each other. Since the reference count can only be decremented by 1 for one object at a time, these two mutually referencing objects will never be garbage collected. Therefore, circular garbage collection checks for circular references, and if one exists, decrements the reference count of the object with the circular reference by 1. This is described below: In a specific embodiment, the request processing method further includes: if it is determined that the first reference count and the second reference count are circularly referenced, simultaneously adjusting the first reference count from a preset value to zero and the second reference count from a preset value to zero.

[0105] Specifically, the circular garbage collection mechanism checks whether there is a circular reference between the multimodal data referenced by the first reference count and the encoded data referenced by the second reference count. If the multimodal data can point to the encoded data, and the encoded data can also point to the multimodal data, then it is determined that there is a circular reference between the first reference count and the second reference count. Then, by gradually adjusting the first reference count to zero from a preset value, the second reference count is also gradually adjusted to zero from a preset value. In other words, while subtracting from the first reference count, subtracting from the second reference count is also performed simultaneously. This ensures that both the multimodal data and the encoded data can be released in this embodiment.

[0106] Step 508: Receive the inference results in response to the multimodal inference request.

[0107] Specifically, as described in the foregoing embodiments, the terminal sends a multimodal inference request to the inference server. After successfully receiving the multimodal inference request for multimodal data, the inference server responds to the request by extracting the encoded data from the request and decoding it to obtain the corresponding multimodal data. It then calls the multimodal large language model to perform inference tasks on the multimodal data to obtain the inference result for the multimodal inference request. Finally, the inference server sends the inference result for the multimodal inference request back to the terminal, so that the terminal can receive the inference result.

[0108] In an optional embodiment, after receiving the inference result fed back for the multimodal inference request, the request processing method further includes: deleting the encoded data corresponding to the multimodal data stored on the local disk.

[0109] Specifically, if the terminal successfully receives the inference result in response to the multimodal inference request, and the inference result indicates that the inference was successful, meaning that there will be no failure of the multimodal inference request, then the corresponding encoded data of the multimodal data stored on the local disk will not be reread. To avoid occupying the storage space of the local disk, the terminal will delete the corresponding encoded data of the multimodal data stored on the local disk.

[0110] Understandably, in practical applications, there may be situations where multimodal inference requests fail to be sent. This means the server cannot receive the multimodal inference request and therefore cannot respond to it with the corresponding inference result. To ensure the reliability of multimodal inference request processing, a new multimodal inference request can be generated and sent to the inference server. This will be explained in detail below:

[0111] In an optional embodiment, before receiving the inference result feedback for the multimodal inference request, the request processing method further includes: if the multimodal inference request fails to be sent, rereading the encoded data stored on the local disk, generating and sending the multimodal inference request to the inference server based on the reread encoded data.

[0112] Specifically, after the terminal sends a multimodal inference request to the inference server, if it receives a request reception failure response from the inference server, it determines that the multimodal inference request has failed to send. Alternatively, the terminal can actively detect the failure and also determine that the multimodal inference request has failed. Or, as described in the previous embodiments, if no inference result is received from the inference server within a preset time interval, it can also be determined that the multimodal inference request has failed. The scenarios for multimodal inference request failure will not be elaborated here. Based on this, the terminal can regenerate and send the multimodal inference request to the inference server based on the encoded data. Since the terminal has released the multimodal data and corresponding encoded data in its local memory after generating the multimodal inference request, and persisted the corresponding encoded data of the multimodal data to the local disk, the terminal specifically needs to reread the encoded data stored on the local disk and, according to the request generation method described in the previous embodiments, regenerate the multimodal inference request based on the reread encoded data, and then send the regenerated multimodal inference request to the inference server.

[0113] It is understandable that in practical applications, there may be situations where the inference server fails to process the multimodal inference request, resulting in inference failure. In such cases, a new request for multimodal inference can be made to the inference server. This will be described in detail below: In an optional embodiment, after generating and sending a multimodal inference request based on the encoded data, the request processing method further includes: if the received inference result indicates that the inference has failed, rereading the encoded data stored on the local disk, generating and sending a multimodal inference request to the inference server based on the reread encoded data.

[0114] Specifically, after receiving the inference result in response to the multimodal inference request, the terminal determines the information carried by the inference result. If the inference result indicates inference failure, it means that the server has not successfully completed the inference for the multimodal data. In this case, it is still necessary to reread the encoded data stored on the local disk, regenerate the multimodal inference request based on the reread encoded data, and send the regenerated multimodal inference request to the inference server. The specific method is similar to that described in the previous embodiments and will not be repeated here.

[0115] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0116] In the above request processing method, the encoded data is released from local memory to avoid data occupying local memory for a long time, thereby reducing the impact on the request processing performance of the terminal device where the multimodal inference client is located. In the event of a request failure, the multimodal inference request can be re-initiated based on the encoded data persisted to the local disk, thereby receiving the inference results fed back for the multimodal inference request. Thus, while avoiding the occupation of local memory, it can also ensure that data support is provided for the generation and processing of multimodal inference requests. This reduces the local memory occupation of the multimodal inference client, thereby avoiding the impact on the performance of the terminal device where the multimodal inference client is located. Furthermore, since it does not affect the generation and issuance of multimodal inference requests, it increases the number of concurrent requests that a single machine can handle.

[0117] In one embodiment, such as Figure 8 As shown, the request processing method also includes:

[0118] Step 802: Obtain guidance text for instructing on reasoning about multimodal data.

[0119] The prompt text is used to instruct the inference process on the multimodal data. In this embodiment, the prompt text is the text input in real time by the inference requester. Specifically, the terminal obtains the prompt text that instructs the inference process on the multimodal data. That is, the inference requester inputs or selects the prompt text for the selected multimodal data on the terminal's display interface and triggers an operation on the prompt text, so that the terminal obtains the prompt text and stores it in local memory. For example, the multimodal data includes multiple images and multiple videos. The inference requester inputs the question "Does the image or video contain target item A? The model determines and outputs the result." At this time, the question input by the inference requester is the prompt text, that is, the prompt text is "Does the image or video contain target item A? The model determines and outputs the result."

[0120] As described in the foregoing embodiments, the encoded data is used to generate a multimodal inference request for the multimodal inference task. The generated multimodal inference request is an HTTP request that encapsulates the multimodal inference data. The data format for this HTTP request encapsulation is specifically binary. In practical applications, the terminal uses the Pickle module in Python or other serialization tools to convert the guidance text into binary format guidance text and write it to local memory. The specific reading and writing method for the binary format guidance text is similar to the reading and writing method for binary format multimodal data in the foregoing embodiments, and will not be repeated here. For ease of understanding, as... Figure 9 The encoded data 902 and the instruction text 904, which are stored in local memory, are shown.

[0121] As described in the foregoing embodiments, the data encoding process in this embodiment needs to consider the format requirements of the request encapsulation. That is, during the request generation process, both the multimodal data and the guidance text need to be converted into multimodal strings. This will be described in detail below:

[0122] In step 504, generating and sending a multimodal inference request based on the encoded data includes: step 804, generating and sending a multimodal inference request based on the encoded data and the guidance text.

[0123] The encoded data includes the encoded data for each modality of the multimodal data. Specifically, the terminal performs data conversion on the binary format multimodal data to obtain a multimodal string. This multimodal string is then identified as the encoded data of the multimodal data. Considering that the multimodal data includes multiple modalities, the terminal performs data conversion on each modality of the binary format multimodal data separately to obtain a string for each modality. This string is the encoded data for each modality.

[0124] Based on this, after the terminal completes the multimodal data reading and data encoding, the multimodal inference process for handling multimodal inference tasks is started first. For the multimodal inference process, a multimodal inference request is generated using the encoded data and guidance text stored in local memory. That is, the encoded data and guidance text are put into the request input and then the request is encapsulated to obtain the multimodal inference request. Specifically, the encoded data and guidance text are stored in the HTTP request body and encapsulated to obtain the multimodal inference request.

[0125] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0126] In this embodiment, during the generation of a multimodal inference request, guidance text indicating the inference of multimodal data is considered. This ensures that the multimodal inference request can instruct the decoding of encoded data to obtain multimodal data for inference, thereby guaranteeing that the generated multimodal inference request can accurately instruct the inference of multimodal data and improving the reliability of request processing.

[0127] Considering that in practical applications, multimodal data includes multiple modal data within each modality, and the processing resources required for request processing are substantial, it is advisable to group the modal data within the multimodal data and generate corresponding requests and receive inference results for each group. This will be discussed in detail below:

[0128] In one embodiment, such as Figure 10 As shown, the request processing method also includes:

[0129] Step 1002: Group the encoded data of each modality data to obtain multiple groups of encoded data, each group of encoded data including encoded data of at least one modality data.

[0130] Each set of coded data includes coded data for at least one modality, meaning each set should contain coded data for multiple modalities. This ensures that when constructing batches using coded data, inference can be performed on multiple modalities simultaneously, and that greater data throughput can be achieved during request transmission, thereby improving inference efficiency and request processing efficiency. Specifically, the terminal groups the coded data for each modality to obtain multiple sets of coded data. The terminal determines the total number of groups based on the number of coded data for each modality, and then groups the coded data for each modality according to the total number of groups, resulting in multiple sets of coded data matching the total number of groups. In this case, the number of multiple sets of coded data is the total number of groups. Alternatively, the terminal can determine the number of data included in each group based on its processing capabilities (such as the number of requests sent in parallel). In this case, the terminal groups the coded data for each modality according to the number of data, resulting in multiple sets of coded data. In this case, the number of coded data for multiple modalities included in the multiple sets of coded data is the number of data.

[0131] For example, multimodal data includes images D1, D2, D3, D4, D5, and D6, video E1, and video E2. Through the aforementioned embodiments, encoded data F1 for image D1, F2 for image D2, F3 for image D3, F4 for image D4, F5 for image D5, F6 for image D6, F7 for video E1, and F8 for video E2 can be obtained. In this case, encoded data F1 to F8 can be divided into two sets of encoded data, such as one set of encoded data G1 and one set of encoded data G2. One set of encoded data G1 includes encoded data F1, F2, F3, and F7, while the other set of encoded data G2 includes encoded data F4, F5, F6, and F8.

[0132] It is understandable that the premise for grouping the encoded data of each modality is that the input guidance text is the same. For example, the guidance text is "Does the image or video contain target item A? The model judges and outputs the result." In this case, the multimodal data includes multiple images and multiple videos, meaning that the guidance text is the text that instructs reasoning on the same multimodal data. Furthermore, no array is constructed for the encoded data. In this case, each group of encoded data is a single encoded data corresponding to one modality, and the processing method is the data processing method described in the previous embodiment.

[0133] Step 1004: For each set of encoded data, construct a request array based on the encoded data of at least one modal data included and the guidance text.

[0134] Specifically, since the guidance text is text that instructs inference on the same multimodal data, the terminal constructs an array for each set of coded data by combining the guidance text with the coded data of the modal data included in that set, thus constructing a request array corresponding to that set of coded data. For ease of understanding, based on the previous example, for coded data G1, a request array H1 corresponding to coded data G1 can be constructed together with the guidance text. In this case, request array H1 includes coded data F1, coded data F2, coded data F3, coded data F7, and guidance text used to instruct inference on images D1, D2, D3, and video E1. Similarly, for coded data G2, a request array H2 corresponding to coded data G2 can be constructed together with the guidance text. In this case, request array H1 includes coded data F4, coded data F5, coded data F6, coded data F8, and guidance text used to instruct inference on images D4, D5, D6, and video E2.

[0135] Step 1006: Save the request array constructed for each set of encoded data to local memory.

[0136] Specifically, the terminal saves the request array constructed for each set of encoded data to local memory. That is, the request arrays constructed for each set of encoded data are sequentially serialized and saved to local memory. Since request arrays are constructed in this embodiment, each request array needs to be saved as an array. For ease of understanding, as follows... Figure 11 The diagram shown illustrates the structure of storing multiple request arrays in an array manner. Request array 1102 and request array 1104 are stored sequentially in local memory in an array manner. Request array 1102 includes a set of encoded data 1 and instruction text 1, and request array 1104 includes another set of encoded data 2 and instruction text 1.

[0137] The purpose of request arrays is to improve throughput during request sending and receiving. Using the same instruction text for the same request array can reduce the inference time of the inference server. For example, if the instruction text is "What emotion is contained in the image?", and there are 100 images for which this judgment needs to be made, without request array partitioning, the terminal needs to generate 100 corresponding multimodal inference requests for each of the 100 images and the instruction text, and send them to the inference server so that the inference server can process each of the 100 multimodal inference requests separately. However, if the encoded data of the 100 images is partitioned, such as dividing the encoded data of 4 images into one group, resulting in 25 groups of encoded data, then the terminal only needs to generate 25 corresponding multimodal inference requests for each of the 25 groups of encoded data and the instruction text, and send them to the inference server so that the inference server can process each of the 25 multimodal inference requests separately. It is understandable that the total number of data in each group of encoded data needs to be determined based on the actual business scenario. The total number of data in each group of encoded data can be 2, 4, 8, 16, etc., and the total number of data can be dynamically adjusted based on the communication quality between the terminal and the server. Therefore, no specific limit is made here.

[0138] Step 804: Generate and send a multimodal inference request based on the encoded data and the guidance text. Specifically, this includes: Step 1008: For each set of encoded data, generate and send a multimodal inference request based on the request array.

[0139] Specifically, since the request array constructed from each set of encoded data is stored in local memory, during the generation of the multimodal inference request, the terminal stores the encoded data and guidance text in the request array constructed from each set of encoded data in the HTTP request body, then encapsulates it to obtain the multimodal inference request, and sends the multimodal inference request to the inference server. For example, to further describe the above embodiment, for a set of encoded data G1, a multimodal inference request C1 is generated through the request array H1 corresponding to the set of encoded data G1, and the multimodal inference request C1 is sent to the inference server. At this time, the multimodal inference request C1 is used to instruct the inference server to perform inference using the modal data corresponding to the encoded data F1, encoded data F2, encoded data F3, and encoded data F7 in the multimodal inference request C1, respectively. Similarly, for another set of encoded data G2, a multimodal inference request C2 is generated through the request array H2 corresponding to the set of encoded data G2, and sent to the inference server. At this time, the multimodal inference request C2 is used to instruct the inference server to perform inference using the modal data corresponding to the encoded data F4, encoded data F5, encoded data F6, and encoded data F8 in the multimodal inference request C1.

[0140] Step 506: After persisting the encoded data to the local disk, release the multimodal data and corresponding encoded data in the local memory, including:

[0141] Step 1010: After persisting the encoded data and guidance text to the local disk, release the multimodal data, corresponding encoded data, and guidance text in the local memory.

[0142] Specifically, after the terminal generates and sends a multimodal inference request, since generating the multimodal inference request in this embodiment requires the encoded data and guidance text in the request array constructed based on each set of encoded data, the terminal needs to persist the encoded data and guidance text to the local disk at this time, and then release the multimodal data, the corresponding encoded data and guidance text in the local memory.

[0143] For example, based on the example of the foregoing embodiment, since the multimodal inference request C1 is generated through the request array H1 corresponding to the encoded data G1, after the terminal sends the multimodal inference request C1 to the inference server, it will first persist the guidance text and the encoded data F1, encoded data F2, encoded data F3, and encoded data F7 in the encoded data G1 to the local disk, and then release the image D1 corresponding to the encoded data F1, the image D2 corresponding to the encoded data F2, the image D3 corresponding to the encoded data F3, the video E1 corresponding to the encoded data F7, the guidance text, the encoded data F1, encoded data F2, encoded data F3, and encoded data F7 from the local memory.

[0144] Similarly, since the multimodal inference request C2 is generated through the request array H2 corresponding to the encoded data G2, after the terminal sends the multimodal inference request C2 to the inference server, it will first persist the guidance text and the encoded data F4, F5, F6, and F8 in the encoded data G2 to the local disk, and then release the image D4 corresponding to the encoded data F4, the image D5 corresponding to the encoded data F5, the image D6 corresponding to the encoded data F6, the video E2 corresponding to the encoded data F8, the guidance text, the encoded data F4, F5, F6, and F8 in the local memory.

[0145] Step 508: Receive the inference results returned in response to the multimodal inference request, specifically including:

[0146] Step 1012: For each set of coded data, receive the inference results fed back in response to the multimodal inference request.

[0147] The inference results include: inference results obtained by inferring the guidance text and each piece of encoded data separately within the same set of encoded data. Specifically, as described in the foregoing embodiments, the terminal sends a multimodal inference request matching each request array to the inference server. After successfully receiving the multimodal inference request for each request array, the inference server responds to the multimodal inference request by extracting multiple pieces of encoded data from the request array, decoding the multiple pieces of encoded data to obtain the corresponding multiple modal data, and then calling the multimodal large language model to perform inference tasks on the multiple modal data to obtain the inference results obtained by inferring the guidance text and the modal data of each piece of encoded data separately. Then, the inference server feeds back the inference results obtained by inferring the multiple modal data separately to the terminal, so that the terminal receives the inference results fed back from the multimodal inference request for each set of encoded data.

[0148] For example, based on the examples of the foregoing embodiments, for a set of encoded data G1, it is possible to receive the inference results of image D1 corresponding to encoded data F1, image D2 corresponding to encoded data F2, image D3 corresponding to encoded data F3, and video E1 corresponding to encoded data F7 within that set of encoded data G1. For another set of encoded data G2, it is possible to receive the inference results of image D4 corresponding to encoded data F4, image D5 corresponding to encoded data F5, image D6 corresponding to encoded data F6, and video E2 corresponding to encoded data F8 within that set of encoded data G2.

[0149] Understandably, for each set of encoded data, if the terminal successfully receives the inference result feedback from the multimodal inference request for that set of encoded data, and each inference result indicates that the inference was successful, then there will be no failure of the multimodal inference request. In other words, the encoded data and guidance text corresponding to the multimodal data stored on the local disk will not be reread. To avoid occupying the storage space of the local disk, the terminal will delete the guidance text and the encoded data corresponding to the multimodal data stored on the local disk.

[0150] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0151] In this embodiment, when constructing batches using encoded data, it is possible to ensure that multiple modal data can be inferred during inference, and that encoded data of multiple modal data in the same group can be transmitted at once during request transmission, so as to achieve greater data throughput for multiple multimodal requests, thereby improving inference efficiency and request processing efficiency.

[0152] Based on the detailed description of the foregoing embodiments, the complete flow of the request processing method in the embodiments of this application will be described below. In one embodiment, such as Figure 12 As shown, a request processing method is provided, which is applied to Figure 3 Taking terminal 302 as an example, it can be understood that this method can also be applied to a system including terminal 302 and server 304, and implemented through the interaction between terminal 302 and server 304. In this embodiment, the method includes the following steps:

[0153] Step 1201: Read multimodal data from the local disk in binary format and store the binary multimodal data in local memory.

[0154] Specifically, the terminal reads the multimodal data required to perform the multimodal inference task from the local disk in binary format. That is, the terminal can first obtain the data identifier of the required multimodal data for the trigger operation of multimodal inference, locate the required multimodal data from the data stored on the local disk through the data identifier, and then read the multimodal data from the local disk into the local memory in binary format.

[0155] Step 1202: Obtain guidance text for instructing on reasoning about multimodal data.

[0156] Specifically, the terminal acquires guidance text that instructs inference on multimodal data. This means the inference requester inputs or selects guidance text for the chosen multimodal data on the terminal's display interface and triggers an operation based on the guidance text, causing the terminal to acquire and store the guidance text in local memory. In practical applications, the terminal uses the Pickle module in Python or other serialization tools to convert the guidance text into binary format and write it to local memory. The specific reading and writing method for binary format guidance text is similar to the reading and writing method for binary format multimodal data in the aforementioned embodiments, and will not be repeated here.

[0157] Step 1203: Perform data conversion on the binary format multimodal data to obtain a multimodal string, which is encoded data.

[0158] Specifically, the terminal stores the read binary-formatted multimodal data in its local memory. Then, the terminal performs data conversion on the binary-formatted multimodal data to obtain a multimodal string. This multimodal string is then designated as encoded data, and the data conversion is performed using Base64 encoding. Specifically, the binary-formatted multimodal data is Base64 encoded to obtain a multimodal string (i.e., an ASCII string). At this point, the local memory stores the binary-formatted multimodal data and its corresponding encoded data (i.e., the ASCII string).

[0159] Step 1204: Group the encoded data for each modality to obtain multiple groups of encoded data.

[0160] Specifically, the terminal determines the total number of groups based on the amount of encoded data for each modality, and then groups the encoded data for each modality according to the total number of groups, resulting in multiple sets of encoded data matching the total number of groups. In this case, the number of these multiple sets of encoded data is the total number of groups. Alternatively, the terminal can determine the amount of data included in each group based on its processing capabilities. In this case, the terminal groups the encoded data for each modality according to the amount of data, resulting in multiple sets of encoded data. In this case, the number of encoded data from multiple modalities included in these multiple sets of encoded data is the total number of data.

[0161] Step 1205: For each set of encoded data, construct a request array based on the encoded data of at least one modal data included and the guidance text.

[0162] Specifically, since the guidance text is the text that instructs reasoning on the same multimodal data, the terminal constructs an array for each set of encoded data by combining the guidance text with the encoded data of the modal data included in that set of encoded data, and constructs the request array corresponding to that set of encoded data.

[0163] Step 1206: Save the request array constructed for each set of encoded data to local memory.

[0164] Specifically, the terminal saves the request array constructed for each set of encoded data to local memory. That is, the request array constructed by the terminal for each set of encoded data is serialized and saved to local memory sequentially. Since a request array is constructed in this embodiment, each request array needs to be saved in an array format.

[0165] Step 1207: For each set of encoded data, generate and send a multimodal inference request based on the request array.

[0166] Specifically, since the request array constructed from each set of encoded data is stored in local memory, during the process of generating a multimodal inference request, the terminal specifically stores the encoded data and guidance text in the request array constructed from each set of encoded data in the HTTP request body, then encapsulates and processes it to obtain the multimodal inference request, and sends the multimodal inference request to the inference server.

[0167] Step 1208: After persisting the encoded data and guidance text to the local disk, release the multimodal data, corresponding encoded data, and guidance text in the local memory.

[0168] Specifically, after the terminal generates and sends a multimodal inference request, since generating the multimodal inference request in this embodiment requires the encoded data and guidance text in the request array constructed based on each set of encoded data, the terminal needs to persist the encoded data and guidance text to the local disk at this time, and then release the multimodal data, the corresponding encoded data and guidance text in the local memory.

[0169] Step 1209: In the event that the multimodal inference request fails to be sent, the request array of each set of encoded data stored on the local disk is reread, and a multimodal inference request is generated and sent to the inference server based on the request array of each set of encoded data that has been reread.

[0170] Specifically, after the terminal sends a multimodal inference request to the inference server, if it receives a request reception failure response from the inference server, it determines that the multimodal inference request has failed to be sent. Alternatively, if the terminal actively detects that the multimodal inference request has failed to be sent, it can also determine that the multimodal inference request has failed. The scenarios for multimodal inference request failure will not be elaborated here. Based on this, the terminal can reread the request array for each set of encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the reread request array for each set of encoded data. Since the terminal has released the multimodal data, corresponding encoded data, and guidance text in its local memory after generating the multimodal inference request, and persisted the guidance text and the corresponding encoded data of the multimodal data to the local disk, the terminal specifically needs to reread the request array for each set of encoded data stored on the local disk, and regenerate the multimodal inference request for each set of encoded data according to the request generation method described in the aforementioned embodiments, and send the regenerated multimodal inference request for each set of encoded data to the inference server.

[0171] Step 1210: If no inference result is received from the inference server in response to the multimodal inference request within a preset time interval, the request array of each set of encoded data stored on the local disk is reread, and a multimodal inference request is generated and sent to the inference server based on the request array of each set of encoded data that has been reread.

[0172] Specifically, after the terminal sends a multimodal inference request to the inference server, it starts a timer for the multimodal inference request. The start timestamp of the timer is the timestamp when the terminal sends the multimodal inference request to the inference server. Then, it starts timing the request and response for the multimodal inference request. If no inference result is received from the inference server in response to the multimodal inference request within the preset time interval, it indicates that there may be a problem with the multimodal inference request being sent, or that there is a problem with the encoded data or guidance text carried in the multimodal inference request having data errors or data loss, or that the inference server is unable to call the multimodal large language model to perform inference tasks based on the encoded data, etc.

[0173] Based on this, the terminal can reread the request array of each set of encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the reread request array of each set of encoded data. Since the terminal has released the multimodal data, corresponding encoded data and guidance text in the local memory after generating the multimodal inference request, and persisted the guidance text and the corresponding encoded data of the multimodal data to the local disk, the terminal specifically needs to reread the request array of each set of encoded data stored on the local disk, and regenerate the multimodal inference request of each set of encoded data according to the request generation method described in the previous embodiment, and send the regenerated multimodal inference request of each set of encoded data to the inference server.

[0174] Step 1211: For each set of coded data, receive the inference results fed back in response to the multimodal inference request.

[0175] Specifically, as described in the foregoing embodiments, the terminal sends a multimodal inference request matching each request array to the inference server. After successfully receiving the multimodal inference request for each request array, the inference server responds to the multimodal inference request, extracts multiple encoded data from the request array in the multimodal inference request, decodes the multiple encoded data to obtain the corresponding multiple modal data, and then calls the multimodal large language model to perform inference tasks on the multiple modal data to obtain the inference results obtained by inferring the guidance text and the modal data of each encoded data separately. Then, the inference server feeds back the inference results obtained by inferring the multiple modal data separately to the terminal, so that the terminal can receive the inference results fed back by the multimodal inference request for each set of encoded data.

[0176] Step 1212: If the received inference result indicates that the inference has failed, reread the request array of each set of encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the request array of each set of encoded data that has been reread.

[0177] Specifically, after receiving the inference result in response to the multimodal inference request, the terminal determines the information carried by the inference result. If the inference result indicates inference failure, it means that the server has not successfully completed the inference for the multimodal data. Therefore, it is still necessary to reread the request array stored on the local disk, generate and send a multimodal inference request to the inference server based on the request array of each reread set of encoded data. Since the terminal has released the multimodal data, corresponding encoded data and guidance text in local memory after generating the multimodal inference request, and persisted the guidance text and the corresponding encoded data of the multimodal data to the local disk, the terminal specifically needs to reread the request array of each reread set of encoded data stored on the local disk, and regenerate the multimodal inference request for each set of encoded data according to the request generation method described in the previous embodiment, and send the regenerated multimodal inference request for each set of encoded data to the inference server.

[0178] Step 1213: Delete the request array for each set of encoded data stored on the local disk.

[0179] Specifically, for each set of encoded data, if the terminal successfully receives the inference result feedback of the multimodal inference request for that set of encoded data, and each inference result indicates that the inference was successful, that is, there will be no failure of the multimodal inference request at this time, that is, the corresponding encoded data and guidance text of the multimodal data stored on the local disk will not be reread. In order to avoid occupying the storage space of the local disk, the terminal will delete the guidance text and the corresponding encoded data of the multimodal data stored on the local disk.

[0180] It should be understood that the specific implementation methods of steps 1201 to 1213 are similar to those of the aforementioned embodiments, and will not be repeated here.

[0181] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0182] Based on the same inventive concept, this application also provides a request processing apparatus for implementing the request processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more request processing apparatus embodiments provided below can be found in the limitations of the request processing method described above, and will not be repeated here.

[0183] In one embodiment, such as Figure 13 As shown, a request processing apparatus is provided, including: a data acquisition module 1302, a request processing module 1304, a data processing module 1306, and an inference result receiving module 1308, wherein:

[0184] The data acquisition module 1302 is used to read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0185] Request processing module 1304 is used to generate and send multimodal inference requests based on encoded data;

[0186] The data processing module 1306 is used to persist the encoded data to the local disk and then release the multimodal data and corresponding encoded data in the local memory. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0187] The inference result receiving module 1308 is used to receive the inference results fed back in response to the multimodal inference request.

[0188] In one embodiment, the data processing module is further configured to delete the corresponding encoded data of the multimodal data stored on the local disk after receiving the inference result fed back in response to the multimodal inference request.

[0189] In one embodiment, the request processing module is further configured to, before receiving the inference result fed back for the multimodal inference request, reread the encoded data stored on the local disk in the event that the multimodal inference request fails to be sent, generate and send the multimodal inference request to the inference server based on the reread encoded data.

[0190] In one embodiment, the request processing module is further configured to, after generating and sending a multimodal inference request based on the encoded data, if the received inference result indicates that the inference has failed, reread the encoded data stored on the local disk, and generate and send a multimodal inference request to the inference server based on the reread encoded data.

[0191] In one embodiment, the request processing module is further configured to, after generating and sending a multimodal inference request based on the encoded data, if no inference result is received from the inference server in response to the multimodal inference request within a preset time interval, reread the encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the reread encoded data.

[0192] In one embodiment, the data acquisition module is specifically used to read multimodal data from the local disk in binary format and store the binary-formatted multimodal data in local memory; and to perform data conversion on the binary-formatted multimodal data to obtain a multimodal string, wherein the multimodal string is encoded data.

[0193] In one embodiment, the data acquisition module is further configured to acquire guidance text for instructing reasoning on multimodal data;

[0194] The request processing module is specifically used to generate and send multimodal inference requests based on coded data and instruction text.

[0195] In one embodiment, the encoded data includes encoded data for each modality of the multimodal data;

[0196] The data acquisition module is also used to group the encoded data of each modality data to obtain multiple sets of encoded data, each set of encoded data including encoded data of at least one modality data; for each set of encoded data, a request array is constructed based on the encoded data of the included at least one modality data and the encoded data of the guidance text; and the request array constructed for each set of encoded data is saved to local memory.

[0197] In one embodiment, the request processing module is specifically used to generate and send a multimodal inference request based on the request array for each set of coded data.

[0198] The data processing module is specifically used to persist the encoded data and guidance text to the local disk and then release the multimodal data, corresponding encoded data and guidance text in the local memory.

[0199] The inference result receiving module is specifically used to receive the inference results fed back to the multimodal inference request for each set of coded data; the inference results include: the inference results obtained by inferring the guidance text and each set of coded data separately.

[0200] In one embodiment, the data processing module is further configured to generate a first local variable for multimodal data and a second local variable for coded data, wherein the first reference count of the first local variable and the second reference count of the second local variable are both preset values.

[0201] The data processing module is specifically used to gradually adjust the first reference count from a preset value to zero and release the multimodal data in local memory; and to gradually adjust the second reference count from a preset value to zero and release the encoded data in local memory.

[0202] In one embodiment, the data processing module is further configured to, when determining that the first reference count and the second reference count are circularly referenced to each other, simultaneously adjust the first reference count from a preset value to zero and the second reference count from a preset value to zero.

[0203] In one embodiment, a computer device is provided, which can be an inference server or a terminal. This embodiment uses the computer device as a terminal as an example, and its internal structure diagram is as follows: Figure 14As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a request processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0204] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0205] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0206] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0207] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0208] It should be noted that the request processing (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0209] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0210] The technical features in the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A request processing method, characterized in that, The method includes: Multimodal data is read from the local disk into local memory and encoded to obtain encoded data; the encoded data includes the encoded data for each modality of the multimodal data. Obtain guidance text to instruct the inference of the multimodal data; The encoded data for each modality is grouped to obtain multiple sets of encoded data, each set of encoded data including encoded data for at least one modality; For each set of encoded data, a request array is constructed based on the encoded data of at least one modal data included and the guidance text; The request array constructed for each set of encoded data is saved to the local memory; For each set of encoded data, a multimodal inference request is generated and sent based on the request array; After persisting the encoded data and the guidance text to the local disk, the multimodal data, the corresponding encoded data, and the guidance text in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure. Receive the inference results fed back in response to the multimodal inference request.

2. The method according to claim 1, characterized in that, After receiving the inference result in response to the multimodal inference request, the method further includes: Delete the corresponding encoded data of the multimodal data stored on the local disk.

3. The method according to claim 1, characterized in that, Before receiving the inference result fed back in response to the multimodal inference request, the method further includes: If the multimodal inference request fails to be sent, the encoded data stored on the local disk is reread, and a multimodal inference request is generated and sent to the inference server based on the reread encoded data.

4. The method according to claim 1, characterized in that, After generating and sending a multimodal inference request based on the encoded data, the method further includes: If the received inference result indicates that the inference has failed, the encoded data stored on the local disk is reread, and a multimodal inference request is generated and sent to the inference server based on the reread encoded data.

5. The method according to claim 1, characterized in that, After generating and sending a multimodal inference request based on the encoded data, the method further includes: If no inference result is received from the inference server in response to the multimodal inference request within a preset time interval, the encoded data stored on the local disk is reread, and a multimodal inference request is generated and sent to the inference server based on the reread encoded data.

6. The method according to claim 1, characterized in that, The process of reading multimodal data from the local disk into local memory and encoding it to obtain encoded data includes: Read multimodal data from the local disk in binary format and store the binary-formatted multimodal data in local memory; The binary format multimodal data is converted to obtain a multimodal string, which is encoded data.

7. The method according to claim 1, characterized in that, Receiving the inference result in response to the multimodal inference request includes: For each set of coded data, receive the reasoning result fed back in response to the multimodal reasoning request; the reasoning result includes: the reasoning result obtained by reasoning the guidance text and each set of coded data separately.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: A first local variable is generated for the multimodal data, and a second local variable is generated for the encoded data. The first reference count of the first local variable and the second reference count of the second local variable are both preset values. The process of releasing the multimodal data and corresponding encoded data from local memory includes: The first reference count is gradually adjusted from the preset value to zero, and the multimodal data in local memory is released; The second reference count is gradually adjusted from the preset value to zero, and the encoded data in local memory is released.

9. The method according to claim 8, characterized in that, The method further includes: If it is determined that the first reference count and the second reference count are circularly referenced, the second reference count is gradually adjusted from the preset value to zero while the first reference count is gradually adjusted from the preset value to zero.

10. A request processing apparatus, characterized in that, The device includes: The data acquisition module is used to read multimodal data from the local disk into local memory and encode the data to obtain encoded data; the encoded data includes the encoded data of each modality in the multimodal data; and to acquire guidance text for instructing reasoning on the multimodal data; The data processing module is used to group the encoded data of each modality data to obtain multiple sets of encoded data, each set of encoded data including encoded data of at least one modality data; for each set of encoded data, a request array is constructed based on the encoded data of the included at least one modality data and the guidance text; and the request array constructed for each set of encoded data is saved to the local memory. The request processing module is used to generate and send a multimodal inference request based on the request array for each set of coded data. The data processing module is further configured to persist the encoded data and the guidance text to the local disk, and then release the multimodal data, the corresponding encoded data and the guidance text in the local memory. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of request failure. The inference result receiving module is used to receive the inference results fed back in response to the multimodal inference request.

11. The apparatus according to claim 10, characterized in that, The data processing module is also used to delete the corresponding encoded data of the multimodal data stored on the local disk.

12. The apparatus according to claim 10, characterized in that, The request processing module is further configured to, in the event that the multimodal inference request fails to be sent, reread the encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the reread encoded data.

13. The apparatus according to claim 10, characterized in that, The request processing module is further configured to, in the event that the received inference result indicates that the inference has failed, reread the encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the reread encoded data.

14. The apparatus according to claim 10, characterized in that, The request processing module is further configured to, if no inference result is received from the inference server in response to the multimodal inference request within a preset time interval, reread the encoded data stored on the local disk, generate and send a multimodal inference request to the inference server based on the reread encoded data.

15. The apparatus according to claim 10, characterized in that, The data acquisition module is specifically used to read multimodal data from the local disk in binary format and store the binary format multimodal data in local memory; and to perform data conversion on the binary format multimodal data to obtain a multimodal string, wherein the multimodal string is encoded data.

16. The apparatus according to claim 10, characterized in that, The reasoning result receiving module is specifically used to receive the reasoning result fed back to the multimodal reasoning request for each set of coded data; the reasoning result includes: the reasoning result obtained by reasoning the guidance text and each set of coded data separately in the same set of coded data.

17. The apparatus according to any one of claims 10 to 16, characterized in that, The data processing module is further configured to generate a first local variable for the multimodal data and a second local variable for the encoded data, wherein the first reference count of the first local variable and the second reference count of the second local variable are both preset values. The data processing module is specifically used to gradually adjust the first reference count from the preset value to zero and release the multimodal data in local memory; The second reference count is gradually adjusted from the preset value to zero, and the encoded data in local memory is released.

18. The apparatus according to claim 17, characterized in that, The data processing module is further configured to, when determining that the first reference count and the second reference count are circularly referenced to each other, simultaneously adjust the first reference count from the preset value to zero and the second reference count from the preset value to zero.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Network request processing method and device, electronic equipment and storage medium

    CN114915659A

  • Multi-modal task processing method and system, multi-modal dialogue task processing method and system, and device

    WO2025026013A1