Request processing method and device, equipment, storage medium and computer program product

Abstract

The invention relates to a request processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: reading multi-modal data from a local disk to a local memory for data coding to obtain coded data; generating and sending a multi-modal reasoning request according to the coded data; after the coded data is persisted to a local disk, multi-modal data and corresponding coded data in a local memory are released, and the coded data in the local disk is used for re-initiating a multi-modal reasoning request according to the coded data under the condition that the request fails; and receiving a reasoning result fed back for the multi-modal reasoning request. By adopting the method, the memory occupation of the local where the multi-modal reasoning client is located can be reduced, so that the performance of the terminal equipment where the multi-modal reasoning client is located is prevented from being influenced, and the number of concurrent requests of a single machine can be increased.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a request processing method, apparatus, device, storage medium, and computer program product. Background Art

[0002] Using a large multimodal language model to perform inference tasks on input multimodal data can produce more accurate inference results. Currently, the response time of the modal inference server for inference tasks generated based on multimodal data ranges from tens to hundreds of seconds. During this period, the multimodal data read by the multimodal inference client is stored in the local memory, occupying the local memory for a long time. Multimodal data such as images and videos takes up even more memory, which not only affects the performance of the terminal device where the multimodal inference client is located, but also, if multimodal inference requests are concurrently made on the multimodal inference client, it will generate a huge memory usage on the multimodal inference client, which in turn limits the number of concurrent requests that can be processed on a single machine. Summary of the Invention

[0003] Based on this, it is necessary to provide a request processing method, device, equipment, storage medium and computer program product to address the above technical problems. This method can reduce the local memory usage of the multimodal reasoning client, thereby avoiding affecting the performance of the terminal device where the multimodal reasoning client is located, and can also increase the number of concurrent requests that can be processed by a single machine.

[0004] In a first aspect, the present application provides a request processing method. The method comprises:

[0005] Read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0006] Generate and send multimodal reasoning requests based on the encoded data;

[0007] After persisting the encoded data to the local disk, the multimodal data and the corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data if the request fails;

[0008] Receive inference results for multimodal inference requests.

[0009] In a second aspect, the present application further provides a request processing device. The device includes:

[0010] The data acquisition module is used to read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0011] A request processing module, used to generate and send multimodal reasoning requests based on the encoded data;

[0012] The data processing module persists the encoded data to the local disk and releases the multimodal data and corresponding encoded data in the local memory. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure.

[0013] The inference result receiving module is used to receive the inference result of the multimodal inference request feedback.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0015] Read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0016] Generate and send multimodal reasoning requests based on the encoded data;

[0017] After persisting the encoded data to the local disk, the multimodal data and the corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data if the request fails;

[0018] Receive inference results for multimodal inference requests.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0020] Read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0021] Generate and send multimodal reasoning requests based on the encoded data;

[0022] After persisting the encoded data to the local disk, the multimodal data and the corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data if the request fails;

[0023] Receive inference results for multimodal inference requests.

[0024] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0025] Read multimodal data from the local disk into the local memory for data encoding to obtain encoded data;

[0026] Generate and send multimodal reasoning requests based on the encoded data;

[0027] After persisting the encoded data to the local disk, the multimodal data and the corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data if the request fails;

[0028] Receive inference results for multimodal inference requests.

[0029] The above-mentioned request processing method, device, computer equipment, storage medium and computer program product read multimodal data from the local disk to the local memory for data encoding to obtain encoded data, and generate and send a multimodal reasoning request based on the encoded data. After persisting the encoded data to the local disk, the multimodal data and corresponding encoded data in the local memory are released. By releasing the encoded data from the local memory to avoid the data occupying the local memory for a long time, the impact on the request processing performance of the terminal device where the multimodal reasoning client is located is reduced. In the event of a request failure, the multimodal reasoning request can be re-initiated based on the encoded data persisted to the local disk, thereby receiving the inference result feedback for the multimodal reasoning request. On the basis of avoiding occupying local memory, it can also ensure that data support is provided for the generation and processing of the multimodal reasoning request, thereby reducing the local memory usage of the multimodal reasoning client, thereby avoiding affecting the performance of the terminal device where the multimodal reasoning client is located, and since it does not affect the generation and issuance of multimodal reasoning requests, the number of concurrent requests that can be processed by a single machine is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the local memory usage status when processing multimodal reasoning requests in related technologies;

[0031] Figure 2 This is a schematic diagram of the local memory usage status when concurrently processing multimodal reasoning requests in the related technology;

[0032] Figure 3 A diagram of an application environment of a request processing method in one embodiment;

[0033] Figure 4 is a brief flowchart of a request processing method in one embodiment;

[0034] Figure 5 Schematic diagram of a request processing method in one embodiment;

[0035] Figure 6 A schematic diagram of the local memory usage status when processing a multimodal reasoning request in one embodiment;

[0036] Figure 7A schematic diagram of the local memory usage status when concurrently processing multimodal reasoning requests in one embodiment;

[0037] Figure 8 is a flowchart of a request processing method in another embodiment;

[0038] Figure 9 FIG2 is an example diagram of storing encoded data and guidance text of multimodal data in a local memory in one embodiment;

[0039] Figure 10 1 is a flowchart of a request processing method in another embodiment;

[0040] Figure 11 This is a diagram showing an example of a structure for storing multiple request arrays in an array manner in one embodiment;

[0041] Figure 12 A complete flowchart of a request processing method in one embodiment is shown;

[0042] Figure 13 is a structural block diagram of a request processing device in one embodiment;

[0043] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] Using a large multimodal language model to perform reasoning tasks on input multimodal data can obtain more accurate reasoning results. In related technologies, for reasoning tasks generated based on multimodal data, the response time of the modal reasoning server ranges from tens of seconds to hundreds of seconds. During this period, the multimodal data read by the multimodal reasoning client is stored in the local memory, occupying the local memory for a long time. Figure 1 shown. Figure 1 A diagram showing the local memory usage during the processing of a multimodal reasoning request. Figure 1 The horizontal diagram represents the different stages from when the multimodal reasoning client generates and sends an inference request to when the modal reasoning server returns the inference result. The vertical diagram represents the local memory usage of the device where the multimodal reasoning client is located. A blank background indicates that the memory usage of the corresponding stage is low, and a dark slash indicates that the memory usage of the corresponding stage is high. Figure 1It can be seen that in the related technology, when the client generates a multimodal reasoning request, it needs to occupy local memory for multimodal data reading and encoding. After the multimodal reasoning request is sent to the inference server, while waiting for the inference result to be returned, the multimodal data still occupies the local memory, resulting in a longer time of occupying the local memory. Moreover, since multimodal data such as pictures and videos occupy more memory, this will inevitably affect the performance of the terminal device where the multimodal reasoning client is located.

[0046] Moreover, if the multimodal reasoning client concurrently requests multimodal reasoning, as shown in the following example: Figure 2 As shown, refer to Figure 2 , which is a schematic diagram of the local memory usage during concurrent multimodal reasoning requests in the related art. It can be seen that in the case of multiple concurrent multimodal reasoning requests, the multimodal data required for each multimodal reasoning request will occupy a large amount of local memory for a long time. At each moment, multiple multimodal reasoning requests occupy the local memory, resulting in a huge memory usage on the multimodal reasoning client at each moment, which greatly limits the number of concurrent requests that can be processed on a single machine.

[0047] To solve the above problems, the embodiment of the present application provides a request processing method that can reduce the local memory usage of the multimodal reasoning client, thereby avoiding affecting the performance of the terminal device where the multimodal reasoning client is located, and can also increase the number of concurrent requests that can be made on a single machine. The request processing method provided in the embodiment of the present application can be applied to Figure 3 In the application environment shown, terminal 302 communicates with server 304 via a network. A multimodal reasoning client is deployed on terminal 302, and server 304 is specifically configured to respond to multimodal reasoning requests initiated by terminal 302 and provide corresponding reasoning results. A data storage system can store data that server 304 needs to process. The data storage system can be integrated with server 304, or placed in the cloud or on another inference server.

[0048] Specifically, using an example application deployed on terminal 302, terminal 302 reads multimodal data from a local disk, encodes the data into local memory, and generates and sends a multimodal inference request based on the encoded data to server 304. After persisting the encoded data to the local disk, terminal 302 releases the multimodal data and the corresponding encoded data in local memory. Terminal 302 then receives the inference result from server 304 in response to the multimodal inference request. By releasing the encoded data from local memory to prevent the data from occupying local memory for a long time, the impact on the request processing performance of the terminal device where the multimodal inference client resides is reduced. In the event of a request failure, the multimodal inference request can be re-initiated based on the encoded data persisted to the local disk. This avoids occupying local memory while ensuring data support for the generation and processing of the multimodal inference request. This reduces the memory usage of the local device where the multimodal inference client resides, thereby preventing performance impacts on the terminal device where the multimodal inference client resides. Furthermore, since the generation and issuance of multimodal inference requests are not affected, the number of concurrent requests that can be processed by a single machine is increased.

[0049] Among them, terminal 302 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, Internet of Things devices, and portable wearable devices. Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Server 304 can be implemented as an independent inference server or an inference server cluster consisting of multiple inference servers. The request processing method provided in the embodiments of the present application can be applied to various scenarios, including but not limited to scenarios where cloud technology and artificial intelligence are used to process multimodal inference requests.

[0050] Since the request processing method provided in the embodiments of the present application is specifically applied to the scenario of request processing for multimodal reasoning requests and involves related technologies such as multimodal reasoning and multimodal models, the technical terms of the related technologies are explained below:

[0051] 1. Multimodal large language model.

[0052] The multimodal large language model is an extension of the Large Language Models (LLM) that has the ability to receive and reason about multimodal (image, audio, video) information.

[0053] 2. Multimodal reasoning.

[0054] Multimodal reasoning is the task of reasoning on multimodal inputs using a large multimodal language model.

[0055] 3. Reasoning request.

[0056] An inference request is a Hypertext Transfer Protocol (HTTP) request that encapsulates multimodal inference data. It includes at least a prompt for inferencing multimodal data and the encoded data of the multimodal input (e.g., binary encoded image data, audio data, or video data). The request is stored in the HTTP request body in JSON format.

[0057] 4. Client.

[0058] In this embodiment of the present application, the client is a multimodal reasoning client for sending multimodal reasoning requests. Furthermore, the client, through multithreading (processes) and coroutines, is capable of sending multiple multimodal reasoning requests simultaneously and in parallel within a short period of time, and receiving the reasoning results for each multimodal reasoning request synchronously or asynchronously.

[0059] For ease of understanding, the following describes the process of the method for processing multimodal reasoning requests in this embodiment. Figure 4 As shown, the multimodal data is first read into the local memory, and then the multimodal data is encoded to obtain the encoded data which is also saved into the local memory. Based on this, a multimodal reasoning request is generated based on the encoded data, and the multimodal reasoning request is sent. After the multimodal reasoning request is generated, the encoded data can be saved to the local disk, and the multimodal data and the corresponding encoded data in the local memory are released. Then it is determined whether the reasoning result for the multimodal reasoning request is returned. If the request fails, it is necessary to read the encoded data from the local disk, and generate a multimodal reasoning request based on the encoded data in a similar manner as described above. If the request is successful, the corresponding encoded data of the multimodal data stored in the local disk is deleted, and the returned reasoning result is received to complete the processing of this multimodal reasoning request.

[0060] The following examples are used to illustrate: In one embodiment, Figure 5 As shown, a request processing method is provided, which is applied to Figure 3 The terminal 302 in FIG. 3 is used as an example for explanation. It is understandable that the method can also be applied to a system including the terminal 302 and the server 304 and implemented through the interaction between the terminal 302 and the server 304. In this embodiment, the method includes the following steps:

[0061] Step 502: read the multimodal data from the local disk into the local memory and perform data encoding to obtain encoded data.

[0062] Multimodal data refers to data that simultaneously contains two or more different modalities. These modalities typically include visual modalities, language modalities, and sensor modalities. Visual modal data includes image data, video data, etc., while language modal data includes text data, speech data, audio data, etc. Therefore, multimodal data includes data of at least two different modalities. For example, multimodal data includes image data and video data, or image data and text data. The data content of multimodal data is not limited here.

[0063] Secondly, the local disk refers to the storage space that is divided out from the hard disk inside the terminal system for data storage, that is, the local is obtained by partitioning the hard disk inside the terminal system. Usually, the local disk is used as the default storage space in the terminal system. Therefore, the hard disk partition inside the terminal system can be defined as a local disk in this embodiment. The local memory is a random access memory, that is, the local memory is a high-speed register used to store information of the currently running program in the terminal system for the central processing unit (CPU) to call and execute. That is, in the embodiment of the present application, the local memory stores the required multimodal data and encoded data when processing multimodal execution requests, that is, the data stored in the local memory is dynamic, and the task call of the local memory terminal system can dynamically call and release data. And the encoded data in the embodiment of the present application is used to generate multimodal reasoning requests for multimodal reasoning tasks.

[0064] Specifically, when there is a need to perform multimodal reasoning, a trigger operation for multimodal reasoning can be performed on the terminal through the reasoning request object, so that the terminal starts to determine the execution of multimodal reasoning through the trigger operation for multimodal reasoning. Considering that multimodal reasoning is an inference task that uses a multimodal large language model to perform multimodal input, when calling the multimodal large language model to perform the inference task, it is necessary to first obtain the multimodal data for multimodal reasoning, and for the terminal, the local disk is a storage space for data storage, that is, the terminal usually stores data through the local disk. Therefore, the terminal reads the multimodal data required to perform the multimodal reasoning task from the local disk, that is, the terminal can first obtain the data identifier of the required multimodal data for the trigger operation of multimodal reasoning, locate the required multimodal data from the data stored on the local disk through the data identifier, and then read the multimodal data from the local disk to the local memory.

[0065] Based on this, the terminal saves the read multimodal data to the local memory, and encodes the acquired multimodal data to obtain encoded data, and stores the obtained encoded data in the local memory. At this time, the local memory stores multimodal data and encoded data corresponding to the multimodal data. It can be understood that in actual applications, the multimodal data can also be data input in real time by the inference request object, that is, the inference request object inputs the multimodal data to the local disk in real time, and performs a trigger operation for multimodal reasoning, so that the terminal directly determines the multimodal data input in real time by the inference request object from the local disk, and reads the multimodal data from the local disk to the local memory.

[0066] Furthermore, since the encoded data is used to generate a multimodal reasoning request for a multimodal reasoning task, and the generated multimodal reasoning request is an HTTP request that encapsulates the multimodal reasoning data, the data format of the HTTP request for data encapsulation is specifically a binary format. That is, the data encoding process in this embodiment needs to consider the format requirements of the request encapsulation. The data encoding process is described in detail below:

[0067] In a specific embodiment, multimodal data is read from a local disk to a local memory for data encoding to obtain encoded data, including: reading multimodal data from a local disk in binary format, and storing the multimodal data in binary format to the local memory; performing data conversion on the multimodal data in binary format to obtain a multimodal string, where the multimodal string is the encoded data.

[0068] Specifically, the terminal reads multimodal data from the local disk in binary format. That is, the terminal reads the binary multimodal data from the local disk in binary format and stores the read binary multimodal data in local memory. In actual applications, the terminal uses the Python Pickle module or other serialization tools to convert the multimodal data into binary format and write it to local memory. The Pickle module creates a Python-specific binary format that can perform object reading and writing operations. Therefore, the Pickle module can read binary multimodal data from the local disk in binary format.

[0069] Then, the terminal performs data conversion on the multimodal data in binary format to obtain a multimodal character string, determines the aforementioned multimodal character string as encoded data, and performs data conversion specifically through Base64 encoding, and Base64 encoding is used to convert binary data into an American Standard Code for Information Interchange (ASCII) character string. Therefore, the specific method of performing data conversion in this embodiment is: Base64 encoding the multimodal data in binary format to obtain a multimodal character string (i.e., an ASCII character string).

[0070] And considering the need to generate multimodal reasoning requests through encoded data, the terminal not only needs to store the encoded data in local memory, but also needs to put the encoded data into the serialized structure of the request input.

[0071] Step 504: Generate and send a multimodal reasoning request based on the encoded data.

[0072] Specifically, as can be seen from the aforementioned embodiments, local memory stores multimodal data and corresponding encoded data. After the terminal completes reading and encoding the multimodal data, it first initiates a multimodal inference process for handling the multimodal inference task. This multimodal inference process is used to generate and send multimodal inference requests. The aforementioned multimodal inference process can be single or multiple, depending on the actual multimodal inference task requirements. The terminal then generates a multimodal inference request for the multimodal inference process using the encoded data stored in the local memory. Specifically, the encoded data is placed into the request input and then encapsulated to obtain the multimodal inference request. Specifically, the encoded data is stored in the HTTP request body and encapsulated to obtain the multimodal inference request. After generating the multimodal inference request, the terminal then sends the multimodal inference request to the inference server, causing the inference server to respond to the multimodal inference request and extract the encoded data in the multimodal inference request, thereby invoking the multimodal large language model to perform inference on the multimodal inference request for the inference task.

[0073] Based on this, if the inference server can successfully call the multimodal large language model to perform an inference task on the encoded data of the multimodal data and obtain the inference result of the multimodal inference request, the inference server will feedback the inference result of the multimodal inference request to the terminal. At this time, the terminal can execute step 508 to receive the inference result of the multimodal inference request fed back by the inference server. On the contrary, if the inference server cannot successfully call the multimodal large language model to perform an inference task on the encoded data of the multimodal data, that is, the inference server cannot feedback the inference result of the multimodal inference request to the terminal, considering that the multimodal inference request may fail to be sent, or the encoded data carried in the multimodal inference request has data errors or data loss problems, the terminal can regenerate a multimodal inference request based on the encoded data and send it to the inference server, so that the inference server can re-call the multimodal large language model to perform the inference task on the encoded data of the multimodal data.

[0074] Step 506: After persisting the encoded data to the local disk, the multimodal data and the corresponding encoded data in the local memory are released. The encoded data in the local disk is used to re-initiate the multimodal reasoning request based on the encoded data in the event of a request failure.

[0075] Specifically, after the terminal generates a multimodal reasoning request, the corresponding encoded data of the multimodal data can be persisted to the local disk, that is, the corresponding encoded data of the multimodal data can be stored in the local disk, and the multimodal data and the corresponding encoded data in the local memory can be released. For example, multimodal data A1 is obtained from the local disk to the local memory, and the data encoding processing is performed to obtain the encoded data B1 of the multimodal data A1 and save it to the local memory. At this time, the local memory stores the multimodal data A1 and the encoded data B1. Then, after the terminal generates a multimodal reasoning request C1 for the multimodal data A1 based on the encoded data B1, the encoded data B1 can be persisted to the local disk, and the multimodal data A1 and the encoded data B1 in the local memory can be released. This avoids multimodal data that takes up more memory space from occupying the local memory for a long time.

[0076] And through the above-mentioned embodiments, it can be known that if the inference server cannot successfully call the multimodal large language model to perform the inference task on the encoded data of the multimodal data, that is, the inference server cannot feedback the inference result of the multimodal inference request to the terminal, then at this time the terminal can regenerate based on the encoded data and send a multimodal inference request to the inference server, so that the inference server can re-call the multimodal large language model to perform the inference task on the encoded data of the multimodal data. Considering that after generating the multimodal inference request, the terminal has released the multimodal data and the corresponding encoded data in the local memory, and the encoded data stored on the local disk can be used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure, that is, in the process of regenerating the multimodal inference request, it is specifically necessary to use the encoded data on the local disk. This situation is described in detail below:

[0077] In an optional embodiment, after generating and sending a multimodal reasoning request based on the encoded data, the request processing method further includes: if no inference result is received from the inference server for the multimodal reasoning request within a preset time interval, re-reading the encoded data stored on the local disk, and generating and sending a multimodal reasoning request to the inference server based on the re-read encoded data.

[0078] Among them, the preset time interval represents the maximum time interval for receiving the inference results fed back by the inference server for the multimodal inference request, that is, after exceeding the preset time interval, it can be determined that the multimodal inference request has failed. Then the preset time interval can be 3 seconds, 5 seconds, etc. The specific duration of the preset time interval needs to be flexibly determined based on the communication situation between the terminal and the inference server, as well as the actual needs of the multimodal inference request.

[0079] Specifically, after the terminal generates and sends a multimodal inference request based on the encoded data, a timer for the multimodal inference request is started. That is, the start timestamp of the timer is the timestamp when the terminal sends the multimodal inference request to the inference server. Then, the request response timer for the multimodal inference request is started. If the inference result of the multimodal inference request is not received from the inference server within the preset time interval, it may indicate that the multimodal inference request failed to be sent, or that the encoded data carried in the multimodal inference request has data errors or data loss, or that the inference server is unable to call the multimodal large language model to perform inference tasks based on the encoded data. At this time, the terminal can regenerate and send a multimodal inference request to the inference server based on the encoded data. Since the terminal has released the multimodal data and corresponding encoded data in the local memory after generating the multimodal inference request, and persisted the corresponding encoded data of the multimodal data to the local disk, the terminal needs to reread the encoded data stored on the local disk and regenerate the multimodal inference request based on the reread encoded data according to the request generation method described in the above embodiment, and send the regenerated multimodal inference request to the inference server.

[0080] To facilitate understanding of the optimization of local memory usage through the above process, such as Figure 6 Schematic diagram of the local memory occupied by multimodal data and corresponding encoded data when processing a request, Figure 6 The horizontal axis in the middle represents the memory usage time axis corresponding to the time when local memory is occupied, and Figure 6 When the background color is blank, it means that the local memory is small. Figure 6 When the color in the middle is a dark slash, it means that a large amount of local memory is occupied. In the process of generating and sending the multimodal reasoning request, through the method introduced in the aforementioned embodiment, since in this embodiment the coding information originally stored in the local memory is persisted to the local disk after the multimodal reasoning request is sent, and the coding information and multimodal data in the local memory are released, when the multimodal reasoning request needs to be sent again due to the failure of the multimodal reasoning request, the coding information is re-read from the local disk to generate the multimodal reasoning request. Therefore, the length of time that the multimodal data and the coding data occupy the memory in the local memory is limited to the time from assembling the multimodal reasoning request to sending the multimodal reasoning request, and there is no need to wait for the reasoning result of the multimodal reasoning request to be returned. Therefore, compared with Figure 1The memory usage diagram shown can reduce the length of time that local memory is occupied in the memory usage timeline. That is, the embodiment of the present application does not wait for the inference result of the multimodal inference request and occupy the local memory for a long time, but after sending the multimodal inference request, the encoded data is persisted to the local disk, and the multimodal data and corresponding encoded data in the local memory are released. Then, the corresponding data in the local memory is released before the inference result is received. Therefore, by Figure 6 It can be seen that since the storage time interval between multimodal data and the corresponding encoded data in the local memory is only the time interval from obtaining the multimodal data to sending the multimodal reasoning request, there is no need to wait for the feedback of the inference results of the multimodal reasoning request, thereby reducing the local memory usage of the multimodal reasoning client.

[0081] Based on this, in the case of multiple concurrent multimodal reasoning requests, data in the local memory can be released correspondingly for each multimodal reasoning request.

[0082] For ease of understanding, we use formula (1) to describe Figure 2 The local memory usage during concurrent multimodal reasoning requests in the related art shown, that is, the local memory usage at any timestamp is as shown in formula (1):

[0083] ;Formula (1)

[0084] in, Indicates the local memory usage at any timestamp, Indicates the request concurrency of concurrent multimodal reasoning requests, that is, there are N multimodal reasoning requests processed concurrently. It represents the cumulative time from the generation of a multimodal inference request to the receipt of the inference result of the multimodal inference request. n represents the ratio of the time from sending the multimodal inference request to the return of the inference result of the multimodal inference request. represents the memory size occupied by the multimodal inference request, and g represents the other memory usage of the multimodal inference request. In practical common scenarios, n >= 99%, which means that the inference results of the multimodal inference request waiting to be returned at the same timestamp are far greater than the multimodal inference request being assembled but not yet sent. Furthermore, the memory size occupied by the multimodal inference request is much larger than the other memory usage of the multimodal inference request. Therefore, after simplifying formula (1), it is equivalent to formula (2):

[0085] ;Formula (2)

[0086] in, Indicates the local memory usage at any timestamp, Indicates the request concurrency of concurrent multimodal reasoning requests, It represents the cumulative time from the generation of a multimodal inference request to the receipt of the inference result of the multimodal inference request. n represents the ratio of the time from sending the multimodal inference request to the return of the inference result of the multimodal inference request. Indicates the memory size occupied by multimodal inference requests.

[0087] It can be seen that the memory usage M is positively correlated with the cumulative time T, the number of concurrent connections N, and the time proportion n. When the number of concurrent connections N is very high, or when the cumulative time T is high due to network delays, or when the inference problem is very complex and the time proportion n is high, the local memory will be greatly occupied.

[0088] like Figure 7 As shown, it is an example of the local memory occupied by concurrent multimodal reasoning requests after optimizing request processing through the embodiment of the present application. Figure 7 The horizontal axis in the middle represents the memory usage time axis corresponding to the time when local memory is occupied, and Figure 7 When the background color is blank, it means that the local memory is small. Figure 7 When the color in the middle is a dark slash, it means that the local memory is occupied more. Since the multimodal data and corresponding encoded data in the local memory are released after each multimodal reasoning request is sent, the local memory occupied by each multimodal reasoning request can be reduced, thereby ensuring that the generation and issuance of multimodal reasoning requests are not affected when multimodal reasoning requests are processed in parallel, increasing the number of multimodal reasoning requests that can be processed in parallel, and thus increasing the number of concurrent requests that can be processed on a single machine. Figure 7 It can be seen that since the time period of occupying a large amount of local memory is reduced, n in formula (2) will also be reduced. Therefore, after optimization, the local memory occupancy is determined as shown in formula (3):

[0089] ;Formula (3)

[0090] in, Indicates the local memory usage at any timestamp, Indicates the request concurrency of concurrent multimodal reasoning requests, Indicates the cumulative time from the generation of a multimodal reasoning request to the receipt of the inference result of the multimodal reasoning request. Indicates the time ratio from sending a multimodal reasoning request to returning the reasoning result of the multimodal reasoning request in this embodiment, Represents the memory size occupied by the multimodal reasoning request. According to formula (3) and formula (2), the memory optimization ratio of the method provided in the embodiment of the present application is specifically: - 100%.

[0091] In practical applications, since the encoded data of multimodal input is stored in the HTTP request body in JSON format, and the HTTP request body in local memory exists as a Python object, this embodiment manages object memory in Python using reference counting and cyclic garbage collection (GC). Reference counting maintains a reference count for each object. When the reference count reaches zero, the object is recycled, releasing the memory occupied by the object. Therefore, when the reference count reaches zero, the corresponding object is released. However, object recycling in Python requires ensuring that the recycled object is no longer needed. Premature object cleanup can cause program crashes, while not releasing objects can cause memory leaks. Therefore, the following two points are necessary: ​​1. Avoid circular references in the HTTP request body. When circular references occur, a generational cycle is performed to properly release the corresponding objects. 2. Manually delete and reduce the reference count at the end of the HTTP request body's lifecycle. This is because Python does not have standard lexical scope, and the release of objects within functions must be triggered manually.

[0092] Based on this, since the http request body in the local memory exists in the form of a Python object, the above two points should be considered to release the multimodal data and the corresponding encoded data in the local memory. The specific implementation method of how to release the multimodal data and the corresponding encoded data in the local memory will be introduced in detail below: In an optional embodiment, the request processing method also includes: generating a first local variable for the multimodal data, and generating a second local variable for the encoded data, and the first reference count of the first local variable and the second reference count of the second local variable are both preset values.

[0093] In practical applications, a memory optimization example is used for multimodal reasoning in a cloud virtual machine (CVM). Formula (2) and Formula (3) are combined for introduction. Table 1 shows the comparative effects of request processing using related technologies and the technical solution provided in this embodiment during CVM modal reasoning:

[0094] Concurrency Local memory usage Average memory usage Optimize ratio Related technologies 7000 200G 0.02G This embodiment 12000 40G 0.003G 757%

[0095] Table 1

[0096] As can be seen from Table 1, the technical solution provided in this embodiment can improve the concurrency of multimodal reasoning requests on the same local memory, that is, increase the number of concurrently processed multimodal reasoning requests, and can reduce the local memory occupied by concurrent multimodal reasoning requests, and the memory optimization ratio is increased to 757%.

[0097] Specifically, the method for reclaiming object memory in Python is as follows: an object is created, and the resulting object is referenced by local variable a. At this point, the reference count of the local variable a of the referenced object is 1. Similarly, a new object is created, and then a new object is referenced by local variable b. The reference count of the local variable b of the referenced new object is 1. However, since local variable b is returned at this time, and the new object referenced by local variable b is used as the return value, the reference count of the local variable b of the referenced new object changes from 1 to 2. Then, when the function executing the object and the new object completes, the lifecycles of the local variable a referencing the object and the local variable b referencing the new object are determined to end based on the completion of the function execution. In other words, local variables a and b become invalid. The reference count of the local variable a of the referenced object is decremented by 1, and the reference count of the local variable b of the referenced new object is also decremented by 1. At this point, the reference count of the object pointed to by local variable a changes from 1 to 0, indicating that the object pointed to by local variable a is no longer needed. Therefore, the memory of the object can be reclaimed, that is, the object is released. Since the new object pointed to by the local variable b is the return value of the function, its reference count is 2. Then the reference count of the new object pointed to by the local variable b is reduced by 1, from 2 to 1, so the new object cannot be recycled at this time.

[0098] Generally speaking, when a function exits, the reference counts of all local variables are subtracted. In this embodiment, Python syntax can also be used to manually subtract at the location where memory needs to be released. That is, when the local variable a is not needed, the reference count of the object pointed to by the local variable a can be manually subtracted through the delete (del) syntax. At this time, the memory occupied by the object pointed to by the local variable a can be reclaimed.

[0099] In this embodiment, after the multimodal inference request is generated and sent, the corresponding function will not end until the inference result feedback for the multimodal inference request is received. Since the data object in the memory needs to be released before the inference result is received, it is necessary to perform a subtraction operation in advance on the local variables pointing to the multimodal data and the encoded data respectively, so that the multimodal data and the encoded data in the local memory can be released in advance.

[0100] Specifically, when the terminal reads multimodal data from the local disk into local memory, it actually reads the multimodal data into the HTTP request body in local memory, generating a first local variable for the multimodal data. Similarly, when the terminal encodes the multimodal data in local memory to obtain encoded data, it actually stores the encoded data into the HTTP request body in local memory, generating a second local variable for the encoded data. At this point, the first reference count of the first local variable and the second reference count of the second local variable both have preset values, typically 1. Specifically, the first reference count of the first local variable is 1, and the second reference count of the second local variable is 1.

[0101] And since the data in the local memory will be released only when the reference count is 0, based on this, the multimodal data and the corresponding encoded data in the local memory are released, specifically including: gradually adjusting the first reference count from a preset value to zero, and releasing the multimodal data in the local memory; gradually adjusting the second reference count from a preset value to zero, and releasing the encoded data in the local memory.

[0102] Specifically, the terminal performs a subtraction operation on the first reference count, and each subtraction operation specifically subtracts 1 from the first reference count, so that the first reference count is gradually adjusted from a preset value to zero. For example, if the first reference count is 1, then by subtracting 1, it can be changed from 1 to 0 through the subtraction operation, thereby releasing the multimodal data in the local memory pointed to by the first reference count that is gradually adjusted to zero. Similarly, the terminal performs a subtraction operation on the second reference count, and each subtraction operation specifically subtracts 1 from the second reference count, so that the second reference count is gradually adjusted from a preset value to zero. For example, if the second reference count is 2, then by subtracting 1, it can be changed from 2 to 1 through the subtraction operation. At this time, the subtraction operation can continue to be performed on the updated second reference count, and at this time, the second reference count changes from 1 to 0, thereby releasing the encoded data in the local memory pointed to by the second reference count that is gradually adjusted to zero.

[0103] In actual applications, the deletion syntax can also be used to release the memory occupied by the multimodal data and encoded data in the local memory in advance, that is, directly perform the deletion operation on the first reference count and the second reference count, that is, directly adjust the first reference count and the second reference count to 0, so that after the multimodal reasoning request is sent, the multimodal data and encoded data in the local memory are released.

[0104] As can be seen from the above introduction, there is a circular reference in the reference count, that is, two different objects point to each other. Since the reference count can only be reduced by 1 for one object at a time, these two objects that reference each other cannot be recycled in any way. Then, the presence of a circular reference is detected through cyclic garbage collection. If so, the reference count of the object with the circular reference is reduced by 1. This is introduced below: In a specific embodiment, the request processing method further includes: when it is determined that the first reference count and the second reference count are circularly referenced to each other, while gradually adjusting the first reference count from a preset value to zero, the second reference count is gradually adjusted from a preset value to zero.

[0105] Specifically, a circular garbage collection mechanism is used to determine whether a circular reference exists between the multimodal data referenced by the first reference count and the encoded data referenced by the second reference count. If the multimodal data can point to the encoded data, and the encoded data can also point to the multimodal data, then it is determined that the first reference count and the second reference count are circularly referenced to each other. In this manner, while the first reference count is gradually adjusted from a preset value to zero, the second reference count is also gradually adjusted from a preset value to zero. That is, while the first reference count is subtracted, the second reference count is also subtracted. This ensures that the multimodal data and the encoded data can be released together in this embodiment.

[0106] Step 508: Receive the inference result of the multimodal inference request feedback.

[0107] Specifically, as can be seen from the introduction in the aforementioned embodiments, the terminal sends a multimodal reasoning request to the inference server. Then, after successfully receiving the multimodal reasoning request for multimodal data, the inference server responds to the multimodal reasoning request, extracts the encoded data in the multimodal reasoning request, and decodes the encoded data to obtain the corresponding multimodal data, thereby calling the multimodal large language model to perform the reasoning task on the multimodal data to obtain the reasoning result for the multimodal reasoning request. Then, the inference server feeds back the reasoning result for the multimodal reasoning request to the terminal, so that the terminal receives the reasoning result fed back for the multimodal reasoning request.

[0108] In an optional embodiment, after receiving the inference result fed back in response to the multimodal inference request, the request processing method further includes: deleting the encoded data corresponding to the multimodal data stored in the local disk.

[0109] Specifically, on the basis that the terminal successfully receives the inference result feedback for the multimodal inference request, and the inference result indicates that the inference is successful, that is, there will be no request failure for the multimodal inference request at this time, then the corresponding encoded data of the multimodal data stored in the local disk will not be re-read. In order to avoid occupying the storage space of the local disk, the terminal will delete the corresponding encoded data of the multimodal data stored in the local disk.

[0110] It is understandable that in actual applications, there may be a situation where the multimodal reasoning request fails to be sent. That is, at this time, the server cannot receive the multimodal reasoning request, and therefore cannot respond to the multimodal reasoning request to obtain the corresponding reasoning result. To ensure the reliability of the request processing of the multimodal reasoning request, the multimodal reasoning request can also be regenerated and sent to the inference server. The following is a detailed introduction:

[0111] In an optional embodiment, before receiving the inference result feedback for the multimodal inference request, the request processing method also includes: in the event that the multimodal inference request fails to be sent, re-reading the encoded data stored on the local disk, generating and sending the multimodal inference request to the inference server based on the re-read encoded data.

[0112] Specifically, after the terminal sends a multimodal reasoning request to the inference server, if it receives a request reception failure response from the inference server, it determines that the multimodal reasoning request has failed to be sent. Alternatively, the terminal proactively detects that the multimodal reasoning request has failed to be sent, which can also determine that the multimodal reasoning request has failed to be sent. Alternatively, as described in the aforementioned embodiment, if the inference result of the multimodal reasoning request is not received from the inference server within a preset time interval, it can also determine that the multimodal reasoning request has failed to be sent. The scenario of the multimodal reasoning request failing to be sent is not described here. Based on this, the terminal can regenerate and send a multimodal reasoning request to the inference server based on the encoded data. Since the terminal has released the multimodal data and corresponding encoded data in the local memory after generating the multimodal reasoning request, and persisted the corresponding encoded data of the multimodal data to the local disk, the terminal specifically needs to reread the encoded data stored on the local disk and regenerate the multimodal reasoning request based on the reread encoded data according to the request generation method described in the aforementioned embodiment, and send the regenerated multimodal reasoning request to the inference server.

[0113] It is understandable that in actual applications, there may be situations where the inference server fails to successfully process the multimodal inference request, resulting in inference failure. At this time, multimodal inference can be re-requested from the inference server. This is introduced in detail below: In an optional embodiment, after generating and sending a multimodal inference request based on the encoded data, the request processing method also includes: when the received inference result indicates that the inference failed, re-reading the encoded data stored on the local disk, and generating and sending a multimodal inference request to the inference server based on the re-read encoded data.

[0114] Specifically, after receiving the inference result for the multimodal inference request, the terminal determines the information carried by the inference result. If the inference result indicates that the inference failed, it means that the server did not successfully complete the inference for the multimodal data. In this case, it is still necessary to reread the encoded data stored on the local disk, regenerate the multimodal inference request based on the reread encoded data, and send the regenerated multimodal inference request to the inference server. The specific method is similar to that described in the previous embodiment and will not be repeated here.

[0115] It is understandable that the above examples are used to understand the present solution and should not be understood as specific limitations of the present solution.

[0116] In the above request processing method, the encoded data is released from the local memory to avoid the data occupying the local memory for a long time, thereby reducing the impact on the request processing performance of the terminal device where the multimodal reasoning client is located. In the event of a request failure, the multimodal reasoning request can be re-initiated based on the encoded data persisted to the local disk, thereby receiving the inference results fed back to the multimodal reasoning request. In this way, while avoiding occupying the local memory, it can also ensure that data support is provided for the generation and processing of the multimodal reasoning request, thereby reducing the local memory usage of the multimodal reasoning client, thereby avoiding affecting the performance of the terminal device where the multimodal reasoning client is located, and since it will not affect the generation and issuance of multimodal reasoning requests, the number of concurrent requests that can be processed by a single machine is increased.

[0117] In one embodiment, Figure 8 As shown, the request processing method also includes:

[0118] Step 802: Obtain guidance text for instructing reasoning on multimodal data.

[0119] Among them, the guidance text (i.e., prompt text) is used to instruct the reasoning of multimodal data. In this embodiment, the guidance text is the text input in real time by the reasoning request object. Specifically, the terminal obtains the guidance text used to instruct the reasoning of multimodal data, that is, the reasoning request object inputs or selects the guidance text for the selected multimodal data on the display interface of the terminal, and triggers the guidance text, so that the terminal obtains the guidance text and stores it in the local memory. For example, the multimodal data includes multiple images and multiple videos. The reasoning request object inputs the question "Does the image or video contain the target object A? The model judges and outputs the result." At this time, the question input by the reasoning request object is the guidance text, that is, the guidance text is "Does the image or video contain the target object A? The model judges and outputs the result."

[0120] As can be seen from the introduction in the aforementioned embodiments, the encoded data is used to generate a multimodal reasoning request for a multimodal reasoning task, and the generated multimodal reasoning request is an http request that encapsulates the multimodal reasoning data. The data format of the http request for data encapsulation is specifically a binary format. In actual applications, the terminal specifically converts the guidance text into a binary format guidance text through the Pickle module in Python or other serialization tools and writes it to the local memory. The specific reading and writing method of the binary format guidance text is similar to the reading and writing method for the binary format multimodal data in the aforementioned embodiments, and will not be repeated here. For ease of understanding, as Figure 9 The illustrated embodiment stores encoded data 902 of multimodal data and guidance text 904 in local memory.

[0121] As can be seen from the introduction in the aforementioned embodiment, the data encoding process in this embodiment needs to take into account the format requirements of the request encapsulation, that is, in the process of request generation, it is necessary to convert both the multimodal data and the guidance text into a multimodal string. This is described in detail below:

[0122] In step 504 , a multimodal reasoning request is generated and sent according to the encoded data, including: step 804 , a multimodal reasoning request is generated and sent according to the encoded data and the guidance text.

[0123] The encoded data includes the encoded data for each type of modal data in the multimodal data. Specifically, the terminal performs data conversion on the binary multimodal data to obtain a multimodal string, and determines the multimodal string as the encoded data for the multimodal data. Furthermore, considering that the multimodal data includes multiple modal data, the terminal performs data conversion on each type of modal data in the binary multimodal data to obtain a string for each type of modal data. The string for each type of modal data is the encoded data for each type of modal data.

[0124] Based on this, after the terminal completes multimodal data reading and data encoding, it first starts the multimodal reasoning process for processing the multimodal reasoning task. For the multimodal reasoning process, a multimodal reasoning request is generated through the encoded data and guidance text stored in the local memory. That is, the encoded data and guidance text are put into the request input and the request is encapsulated to obtain a multimodal reasoning request. Specifically, the encoded data and guidance text are saved in the http request body for encapsulation processing to obtain a multimodal reasoning request.

[0125] It is understandable that the above examples are used to understand the present solution and should not be understood as specific limitations of the present solution.

[0126] In this embodiment, in the process of generating a multimodal reasoning request, the guidance text indicating the reasoning of multimodal data is considered, so that the multimodal reasoning request can indicate that the encoded data is decoded to obtain multimodal data for reasoning, thereby ensuring that the generated multimodal reasoning request can accurately indicate the reasoning of multimodal data and improve the reliability of request processing.

[0127] Considering that in practical applications, multimodal data includes multiple modal data under each modality, the processing resources called for request processing are relatively large. In this case, it is possible to consider grouping the modal data in the multimodal data and generating corresponding requests and receiving inference results for each group. The following is a detailed introduction to this:

[0128] In one embodiment, Figure 10 As shown, the request processing method also includes:

[0129] Step 1002 : Grouping the coded data of each modal data to obtain multiple groups of coded data, each group of coded data including coded data of at least one modal data.

[0130] Each set of coded data includes coded data of at least one modality, that is, each set of coded data should include coded data of multiple modalities. This ensures that when batches are constructed from the coded data, inference can be performed on multiple modalities simultaneously. This also allows for greater data throughput during request transmission, thereby improving inference efficiency and request processing efficiency. Specifically, the terminal groups the coded data of each modality to obtain multiple sets of coded data. The terminal also determines the total number of groups based on the number of data in the coded data of each modality, and then groups the coded data of each modality according to the total number of groups, obtaining multiple sets of coded data that match the total number of groups. In this case, the number of sets of coded data is the total number of groups. Alternatively, the terminal may determine the number of data included in each set based on terminal processing capacity (such as the number of requests to be sent in parallel). In this case, the coded data of each modality is grouped according to the number of data, obtaining multiple sets of coded data. In this case, the number of coded data of the multiple modalities included in the multiple sets of coded data is the number of data.

[0131] For example, multimodal data includes image D1, image D2, image D3, image D4, image D5, image D6, video E1, and video E2. The aforementioned embodiment can yield coded data F1 for image D1, coded data F2 for image D2, coded data F3 for image D3, coded data F4 for image D4, coded data F5 for image D5, coded data F6 for image D6, coded data F7 for video E1, and coded data F8 for video E2. In this case, coded data F1 through F8 can be divided into two groups of coded data, such as a group of coded data G1 and a group of coded data G2. Group G1 includes coded data F1, coded data F2, coded data F3, and coded data F7, while group G2 includes coded data F4, coded data F5, coded data F6, and coded data F8.

[0132] It is understood that grouping the coded data for each modal data segment requires that the input guidance text be identical. For example, if the guidance text is "Whether the image or video contains target item A, the model determines and outputs the result," then the multimodal data includes multiple images and multiple videos. In other words, the guidance text is instructing reasoning for the same multimodal data. Furthermore, the coded data is not arrayed; in this case, each group of coded data is a single coded data segment corresponding to a single modal data segment, and the processing method is the same as that described in the previous embodiment.

[0133] Step 1004: For each set of coded data, a request array is constructed based on the coded data of at least one modality data and the guide text.

[0134] Specifically, since the guidance text is a text that instructs reasoning on the same multimodal data, the terminal constructs an array of the guidance text and the coded data of the modal data included in the group of coded data for each set of coded data, thereby constructing a request array corresponding to the group of coded data. For ease of understanding, based on the above example, it can be seen that for coded data G1, a request array H1 corresponding to coded data G1 can be constructed together with the guidance text. In this case, request array H1 includes coded data F1, coded data F2, coded data F3, coded data F7, and guidance text for instructing reasoning on images D1, D2, image D3, and video E1. Similarly, for coded data G2, a request array H2 corresponding to coded data G2 can be constructed together with the guidance text. In this case, request array H1 includes coded data F4, coded data F5, coded data F6, coded data F8, and guidance text for instructing reasoning on images D4, D5, image D6, and video E2.

[0135] Step 1006: Save the request array constructed for each set of encoded data to the local memory.

[0136] Specifically, the terminal saves the request array constructed for each set of coded data to the local memory, that is, the request array constructed for each set of coded data by the terminal is serialized and saved to the local memory in sequence. Since a request array is constructed in this embodiment, each request array needs to be saved in an array manner. For ease of understanding, Figure 11 The structural diagram shown is for storing multiple request arrays in an array manner, in which request array 1102 and request array 1104 are stored in sequence in the local memory in an array manner, and request array 1102 includes a set of coded data 1 and guidance text 1, and request array 1104 includes another set of coded data 2 and guidance text 1.

[0137] Since the technical purpose of creating a request array is to improve the throughput of request transmission and reception, reasoning with the same instruction text for the same request array can reduce the inference time of the inference server. For example, if the instruction text is "What emotions are included in the image?", and there are 100 images, each of which needs to be judged based on "What emotions are included in the image?" Without partitioning the request array, the terminal needs to generate 100 corresponding multimodal inference requests for each of the 100 images and the instruction text, and send them to the inference server, allowing the inference server to perform inference processing on each of the 100 multimodal inference requests. If the encoded data of the 100 images is partitioned, such as the encoded data of four images is divided into one group of encoded data, 25 groups of encoded data are obtained. In this case, the terminal only needs to generate 25 corresponding multimodal inference requests for each of the 25 groups of encoded data and the instruction text, and send them to the inference server, allowing the inference server to perform inference processing on each of the 25 multimodal inference requests. It is understandable that the total number of data in each set of encoded data needs to be determined based on the actual business scenario. The total number of data in each set of encoded data can be 2, 4, 8, 16, etc., and the total number of data can be dynamically adjusted based on the communication quality between the terminal and the server. Therefore, no specific limitation is made here.

[0138] Step 804 , generating and sending a multimodal reasoning request based on the coded data and the guidance text, specifically includes: Step 1008 , generating and sending a multimodal reasoning request based on the request array for each set of coded data.

[0139] Specifically, since the request array constructed from each set of encoded data is stored in local memory, when generating a multimodal inference request, the terminal specifically stores the encoded data and guidance text in the request array constructed from each set of encoded data in the HTTP request body, then performs encapsulation processing to obtain a multimodal inference request, and sends the multimodal inference request to the inference server. For example, further describing the aforementioned embodiment, for a set of encoded data G1, a multimodal inference request C1 is generated using the request array H1 corresponding to the set of encoded data G1, and the multimodal inference request C1 is sent to the inference server. In this case, the multimodal inference request C1 is used to instruct the inference server to perform inference using the modal data corresponding to the encoded data F1, F2, F3, and F7 in the multimodal inference request C1. Similarly, for another group of encoded data G2, a multimodal inference request C2 is generated through the request array H2 corresponding to the group of encoded data G2, and the multimodal inference request C2 is sent to the inference server. At this time, the multimodal inference request C2 is used to instruct the inference server to perform inference through the modal data corresponding to the encoded data F4, encoded data F5, encoded data F6, and encoded data F8 in the multimodal inference request C1.

[0140] Step 506, after persisting the encoded data to the local disk, releases the multimodal data and the corresponding encoded data in the local memory, including:

[0141] Step 1010: After persisting the coded data and the guidance text to the local disk, the multimodal data, the corresponding coded data and the guidance text in the local memory are released.

[0142] Specifically, after the terminal generates and sends a multimodal reasoning request, since the generation of the multimodal reasoning request in this embodiment requires the encoded data and guidance text in the request array constructed based on each set of encoded data, the terminal needs to persist the encoded data and guidance text to the local disk at this time, and then release the multimodal data, corresponding encoded data and guidance text in the local memory.

[0143] For example, based on the example of the aforementioned embodiment, since the multimodal reasoning request C1 is generated by the request array H1 corresponding to the coding data G1, after the terminal sends the multimodal reasoning request C1 to the inference server, the guidance text and the coding data F1, coding data F2, coding data F3, and coding data F7 in the group of coding data G1 will first be persisted to the local disk, and then the image D1 corresponding to the coding data F1, the image D2 corresponding to the coding data F2, the image D3 corresponding to the coding data F3, the video E1 corresponding to the coding data F7, the guidance text, the coding data F1, the coding data F2, the coding data F3, and the coding data F7 in the local memory will be released.

[0144] Similarly, since the multimodal inference request C2 is generated by the request array H2 corresponding to the coding data G2, after the terminal sends the multimodal inference request C2 to the inference server, the guidance text and the coding data F4, coding data F5, coding data F6, and coding data F8 in the group of coding data G2 will first be persisted to the local disk, and then the image D4 corresponding to the coding data F4, the image D5 corresponding to the coding data F5, the image D6 corresponding to the coding data F6, the video E2 corresponding to the coding data F8, the guidance text, the coding data F4, the coding data F5, the coding data F6, and the coding data F8 in the local memory will be released.

[0145] Step 508, receiving the inference result of the multimodal inference request, specifically includes:

[0146] Step 1012: For each set of encoded data, receive the inference result fed back in response to the multimodal inference request.

[0147] The inference results include: inference results obtained by separately inferring the guide text and each encoded data in the same set of encoded data. Specifically, as described in the aforementioned embodiment, the terminal sends a multimodal inference request matching each request array to the inference server. After successfully receiving the multimodal inference request for each request array, the inference server responds to the multimodal inference request and extracts multiple encoded data in the request array in the multimodal inference request. The server then decodes the multiple encoded data to obtain corresponding multiple modal data, thereby calling the multimodal large language model to perform inference tasks on the multiple modal data to obtain inference results obtained by separately inferring the guide text and the modal data of each encoded data. The inference server then feeds back the inference results obtained by separately inferring the multiple modal data to the terminal, so that the terminal receives the inference results fed back by the multimodal inference request for each set of encoded data.

[0148] For example, further describing the example of the aforementioned embodiment, for a set of coded data G1, the inference results for image D1 corresponding to coded data F1, image D2 corresponding to coded data F2, image D3 corresponding to coded data F3, and video E1 corresponding to coded data F7 can be received. For another set of coded data G2, the inference results for image D4 corresponding to coded data F4, image D5 corresponding to coded data F5, image D6 corresponding to coded data F6, and video E2 corresponding to coded data F8 can be received.

[0149] It can be understood that, for each set of encoded data, if the terminal successfully receives the inference result feedback of the multimodal inference request for the set of encoded data, and each inference result indicates that the inference is successful, that is, there will be no request failure for the multimodal inference request at this time, that is, the encoded data and guidance text corresponding to the multimodal data stored in the local disk will not be re-read. In order to avoid occupying the storage space of the local disk, the terminal will delete the guidance text and the encoded data corresponding to the multimodal data stored in the local disk.

[0150] It is understandable that the above examples are used to understand the present solution and should not be understood as specific limitations of the present solution.

[0151] In this embodiment, by constructing batches through encoded data, it can be ensured that multiple modal data can be inferred during inference, and the encoded data of multiple modal data in the same group can be transmitted at one time when requesting transmission, so as to achieve a higher data throughput for multiple multimodal requests, thereby improving the inference efficiency and the request processing efficiency.

[0152] Based on the detailed description of the above embodiments, the complete process of the request processing method in the embodiment of the present application will be described below. In one embodiment, Figure 12 As shown, a request processing method is provided, which is applied to Figure 3 The terminal 302 in the example is used for explanation. It is understandable that the method can also be implemented in a system including the terminal 302 and the server 304 and through the interaction between the terminal 302 and the server 304. In this embodiment, the method includes the following steps:

[0153] Step 1201 : Read multimodal data from a local disk in binary format, and store the multimodal data in binary format into a local memory.

[0154] Specifically, the terminal reads the multimodal data required to perform the multimodal reasoning task from the local disk in binary format. That is, the terminal can first obtain the data identifier of the required multimodal data for the trigger operation of multimodal reasoning, locate the required multimodal data from the data stored on the local disk through the data identifier, and then read the multimodal data from the local disk to the local memory in binary format.

[0155] Step 1202: Obtain guidance text for instructing reasoning on multimodal data.

[0156] Specifically, the terminal obtains guidance text for instructing reasoning about multimodal data. That is, the reasoning request subject inputs or selects guidance text for the selected multimodal data on the terminal's display interface and triggers the guidance text, so that the terminal obtains the guidance text and stores it in local memory. In actual applications, the terminal specifically converts the guidance text into binary format guidance text using the Python Pickle module or other serialization tools and writes it to local memory. The specific reading and writing methods of the binary format guidance text are similar to the methods for reading and writing binary format multimodal data in the aforementioned embodiment and will not be repeated here.

[0157] Step 1203 : performing data conversion on the multimodal data in binary format to obtain a multimodal character string, where the multimodal character string is encoded data.

[0158] Specifically, the terminal stores the read binary multimodal data in local memory. It then converts the binary multimodal data into a multimodal string, identifies the multimodal string as encoded data, and performs the data conversion using Base64 encoding. Specifically, the data conversion involves Base64 encoding the binary multimodal data to obtain a multimodal string (i.e., an ASCII string). At this point, the local memory stores the binary multimodal data and the corresponding encoded data (i.e., an ASCII string).

[0159] Step 1204 , grouping the coded data of each modality data to obtain multiple groups of coded data.

[0160] Specifically, the terminal determines the total number of groups based on the amount of data in the coded data of each modal data type, and then groups the coded data of each modal data type according to the total number of groups, resulting in multiple groups of coded data that match the total number of groups. In this case, the number of coded data groups is the total number of groups. Alternatively, the terminal may determine the amount of data included in each group based on terminal processing capability. In this case, the coded data of each modal data type is grouped according to the data amount, resulting in multiple groups of coded data. In this case, the number of coded data for the multiple modal data types included in the multiple groups of coded data is the data amount.

[0161] Step 1205: For each set of coded data, a request array is constructed based on the coded data of at least one modality data and the guide text.

[0162] Specifically, since the guidance text is a text that indicates reasoning for the same multimodal data, the terminal constructs an array of the guidance text and the encoded data of the modal data included in each group of encoded data to construct a request array corresponding to the group of encoded data.

[0163] Step 1206: Save the request array constructed for each set of encoded data to the local memory.

[0164] Specifically, the terminal saves the request array constructed for each set of coded data to the local memory, that is, the request array constructed for each set of coded data by the terminal is serialized and saved to the local memory in sequence. Since a request array is constructed in this embodiment, each request array needs to be saved in an array manner.

[0165] Step 1207 : For each set of encoded data, generate and send a multimodal reasoning request according to the request array.

[0166] Specifically, since the request array constructed by each set of encoded data is stored in the local memory, in the process of generating a multimodal reasoning request, the terminal specifically saves the encoded data and guidance text in the request array constructed by each set of encoded data in the http request body, then encapsulates it to obtain the multimodal reasoning request, and sends the multimodal reasoning request to the inference server.

[0167] Step 1208: After persisting the coded data and the guidance text to the local disk, the multimodal data, the corresponding coded data and the guidance text in the local memory are released.

[0168] Specifically, after the terminal generates and sends a multimodal reasoning request, since the generation of the multimodal reasoning request in this embodiment requires the encoded data and guidance text in the request array constructed based on each set of encoded data, the terminal needs to persist the encoded data and guidance text to the local disk at this time, and then release the multimodal data, corresponding encoded data and guidance text in the local memory.

[0169] Step 1209: If the multimodal inference request fails to be sent, the request array of each set of encoded data stored in the local disk is re-read, and a multimodal inference request is generated and sent to the inference server based on the re-read request array of each set of encoded data.

[0170] Specifically, after the terminal sends a multimodal reasoning request to the inference server, if it receives a request reception failure response from the inference server, it determines that the multimodal reasoning request has failed to be sent. Alternatively, the terminal can proactively detect that the multimodal reasoning request has failed to be sent, and can also determine that the multimodal reasoning request has failed to be sent. The scenario of the multimodal reasoning request failing to be sent will not be described in detail here. Based on this, the terminal can re-read the request array for each set of encoded data stored on the local disk, generate and send a multimodal reasoning request to the inference server based on the request array for each set of encoded data re-read. Since after generating the multimodal reasoning request, the terminal has released the multimodal data, corresponding encoded data, and guidance text in the local memory, and persisted the guidance text and the encoded data corresponding to the multimodal data to the local disk, the terminal specifically needs to re-read the request array for each set of encoded data re-read stored on the local disk, and regenerate the multimodal reasoning request for each set of encoded data based on the request array for each set of encoded data re-read according to the request generation method described in the aforementioned embodiment, and send the re-generated multimodal reasoning request for each set of encoded data to the inference server.

[0171] Step 1210: If no inference result is received from the inference server for the multimodal inference request within a preset time interval, the request array of each set of encoded data stored on the local disk is re-read, and a multimodal inference request is generated and sent to the inference server based on the re-read request array of each set of encoded data.

[0172] Specifically, after the terminal sends a multimodal reasoning request to the inference server, the timer for the multimodal reasoning request is started, that is, the start timestamp of the timer is the timestamp when the terminal sends the multimodal reasoning request to the inference server, and then the request response timing for the multimodal reasoning request is started. If the inference result of the multimodal reasoning request is not received from the inference server within the preset time interval, it means that the multimodal reasoning request may have failed to be sent, or there may be data errors or data loss in the encoded data or guidance text carried in the multimodal reasoning request, or the inference server cannot call the multimodal large language model to perform the reasoning task for the encoded data, etc.

[0173] Based on this, the terminal can re-read the request array of each set of encoded data stored on the local disk, generate and send a multimodal reasoning request to the inference server based on the request array of each set of encoded data re-read. Since after generating the multimodal reasoning request, the terminal has released the multimodal data, corresponding encoded data and guidance text in the local memory, and persisted the guidance text and the encoding data corresponding to the multimodal data to the local disk, the terminal specifically needs to re-read the request array of each set of encoded data re-read and stored on the local disk, and regenerate the multimodal reasoning request for each set of encoded data based on the request array of each set of encoded data re-read according to the request generation method introduced in the aforementioned embodiment, and send the regenerated multimodal reasoning request for each set of encoded data to the inference server.

[0174] Step 1211: For each set of encoded data, receive the inference result fed back in response to the multimodal inference request.

[0175] Specifically, as can be seen from the introduction in the aforementioned embodiments, the terminal sends a multimodal reasoning request that matches each request array to the inference server. Then, after successfully receiving the multimodal reasoning request for each request array, the inference server responds to the multimodal reasoning request and extracts multiple encoded data in the request array in the multimodal reasoning request, and then decodes the multiple encoded data to obtain corresponding multiple modal data, thereby calling the multimodal large language model to perform reasoning tasks on multiple modal data to obtain the inference results obtained by separately reasoning the guidance text and the modal data of each encoded data. Then, the inference server feeds back the inference results obtained by separately reasoning on multiple modal data to the terminal, so that the terminal receives the inference results fed back by the multimodal reasoning request for each group of encoded data.

[0176] Step 1212 , when the received inference result indicates that the inference fails, re-read the request array of each set of encoded data stored in the local disk, and generate and send a multimodal inference request to the inference server based on the re-read request array of each set of encoded data.

[0177] Specifically, after the terminal receives the inference result feedback for the multimodal inference request, it determines the information carried by the inference result. If the inference result indicates that the inference failed, it means that the server has not successfully completed the inference for the multimodal data. Then it is still necessary to re-read the request array stored on the local disk, and generate and send a multimodal inference request to the inference server based on the request array of each set of encoded data re-read. Since after generating the multimodal inference request, the terminal has released the multimodal data, corresponding encoded data and guidance text in the local memory, and persisted the guidance text and the encoding data corresponding to the multimodal data to the local disk, therefore, the terminal specifically needs to re-read the request array of each set of encoded data re-read and stored on the local disk, and according to the request generation method introduced in the aforementioned embodiment, regenerate the multimodal inference request for each set of encoded data based on the request array of each set of encoded data re-read, and send the re-generated multimodal inference request for each set of encoded data to the inference server.

[0178] Step 1213: Delete the request array of each set of encoded data stored in the local disk.

[0179] Specifically, for each set of encoded data, if the terminal successfully receives the inference result feedback of the multimodal inference request for the set of encoded data, and each inference result indicates that the inference is successful, that is, there will be no request failure for the multimodal inference request at this time, that is, the encoded data and guidance text corresponding to the multimodal data stored in the local disk will not be re-read. In order to avoid occupying the storage space of the local disk, the terminal will delete the guidance text and the encoded data corresponding to the multimodal data stored in the local disk.

[0180] It should be understood that the specific implementation of steps 1201 to 1213 are similar to those in the above embodiment and will not be repeated here.

[0181] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0182] Based on the same inventive concept, embodiments of the present application also provide a request processing device for implementing the aforementioned request processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more request processing device embodiments provided below can be found in the above-mentioned limitations on the request processing method and will not be further elaborated here.

[0183] In one embodiment, Figure 13 As shown, a request processing device is provided, including: a data acquisition module 1302, a request processing module 1304, a data processing module 1306 and an inference result receiving module 1308, wherein:

[0184] The data acquisition module 1302 is used to read the multimodal data from the local disk into the local memory and perform data encoding to obtain encoded data;

[0185] The request processing module 1304 is used to generate and send a multimodal reasoning request based on the encoded data;

[0186] The data processing module 1306 is used to persist the encoded data to the local disk and then release the multimodal data and the corresponding encoded data in the local memory. The encoded data in the local disk is used to re-initiate the multimodal inference request based on the encoded data if the request fails.

[0187] The reasoning result receiving module 1308 is configured to receive the reasoning result fed back to the multimodal reasoning request.

[0188] In one embodiment, the data processing module is further configured to delete the encoded data corresponding to the multimodal data stored in the local disk after receiving the inference result fed back in response to the multimodal inference request.

[0189] In one embodiment, the request processing module is also used to re-read the encoded data stored on the local disk before receiving the inference result feedback for the multimodal inference request, and generate and send the multimodal inference request to the inference server based on the re-read encoded data in the event that the multimodal inference request fails to be sent.

[0190] In one embodiment, the request processing module is also used to generate and send a multimodal reasoning request based on the encoded data, and when the received reasoning result indicates that the reasoning failed, re-read the encoded data stored on the local disk, and generate and send a multimodal reasoning request to the inference server based on the re-read encoded data.

[0191] In one embodiment, the request processing module is also used to generate and send a multimodal inference request based on the encoded data. If no inference result is received from the inference server for the multimodal inference request within a preset time interval, the encoded data stored on the local disk is re-read, and a multimodal inference request is generated and sent to the inference server based on the re-read encoded data.

[0192] In one embodiment, the data acquisition module is specifically used to read multimodal data from a local disk in binary format and store the multimodal data in binary format to a local memory; the multimodal data in binary format is converted into a multimodal string, which is encoded data.

[0193] In one embodiment, the data acquisition module is further configured to acquire a guidance text for instructing to perform reasoning on the multimodal data;

[0194] The request processing module is specifically used to generate and send multimodal reasoning requests based on the encoded data and guidance text.

[0195] In one embodiment, the encoded data includes encoded data for each modality of the multimodal data;

[0196] The data acquisition module is also used to group the coded data of each modal data to obtain multiple groups of coded data, each group of coded data includes coded data of at least one modal data; for each group of coded data, a request array is constructed based on the coded data of at least one modal data and the coded data of the guidance text; and the request array constructed for each group of coded data is saved to local memory.

[0197] In one embodiment, the request processing module is specifically configured to generate and send a multimodal reasoning request according to the request array for each set of encoded data;

[0198] The data processing module is specifically used to persist the coded data and the guidance text to the local disk and then release the multimodal data, the corresponding coded data and the guidance text in the local memory;

[0199] The inference result receiving module is specifically used to receive the inference result of the multimodal inference request feedback for each set of coded data; the inference result includes: the inference result obtained by separately inferring the guide text and each coded data in the same set of coded data.

[0200] In one embodiment, the data processing module is further configured to generate a first local variable for the multimodal data and a second local variable for the encoded data, wherein a first reference count of the first local variable and a second reference count of the second local variable are both preset values;

[0201] The data processing module is specifically used to gradually adjust the first reference count from a preset value to zero and release the multimodal data in the local memory; gradually adjust the second reference count from a preset value to zero and release the encoded data in the local memory.

[0202] In one embodiment, the data processing module is further configured to, when determining that the first reference count and the second reference count are circularly referenced to each other, gradually adjust the first reference count from a preset value to zero while also gradually adjusting the second reference count from a preset value to zero.

[0203] In one embodiment, a computer device is provided. The computer device can be an inference server or a terminal. In this embodiment, the computer device is used as an example to introduce the terminal. The internal structure diagram can be shown as follows: Figure 14As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication), or other technologies. When the computer program is executed by the processor, it implements a request processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0204] Those skilled in the art will understand that Figure 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0205] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0206] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0207] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0208] It should be noted that the request processing (including but not limited to the target device information, target personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the target or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0209] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLCs), and the like.

[0210] The technical feature information of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical feature information in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical feature information, they should be considered to be within the scope of this specification.

[0211] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A request processing method, characterized in that: The method comprises: Read multimodal data from the local disk into the local memory for data encoding to obtain encoded data; generating and sending a multimodal reasoning request based on the encoded data; After persisting the encoded data to the local disk, releasing the multimodal data and corresponding encoded data in the local memory, the encoded data in the local disk being used to re-initiate the multimodal inference request based on the encoded data in the event of a request failure; Receive a reasoning result fed back to the multimodal reasoning request.

2. The method according to claim 1, characterized in that After receiving the inference result fed back in response to the multimodal reasoning request, the method further includes: Delete the encoded data corresponding to the multimodal data stored in the local disk.

3. The method according to claim 1, characterized in that Before receiving the inference result fed back in response to the multimodal reasoning request, the method further includes: In the case where the multimodal reasoning request fails to be sent, the encoded data stored in the local disk is re-read, and a multimodal reasoning request is generated and sent to the reasoning server based on the re-read encoded data.

4. The method according to claim 1, wherein After generating and sending a multimodal reasoning request according to the encoded data, the method further includes: In a case where the received inference result indicates that the inference fails, the encoded data stored in the local disk is re-read, and a multimodal inference request is generated and sent to the inference server based on the re-read encoded data.

5. The method according to claim 1, wherein After generating and sending a multimodal reasoning request according to the encoded data, the method further includes: If no inference result is received from the inference server for the multimodal inference request within a preset time interval, the encoded data stored on the local disk is re-read, and a multimodal inference request is generated and sent to the inference server based on the re-read encoded data.

6. The method according to claim 1, characterized in that The step of reading the multimodal data from the local disk into the local memory and encoding the data to obtain the encoded data includes: Read multimodal data from the local disk in binary format and store the binary multimodal data in local memory; Data conversion is performed on the multimodal data in binary format to obtain a multimodal character string, where the multimodal character string is encoded data.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining a guidance text for instructing to perform reasoning on the multimodal data; Generating and sending a multimodal reasoning request according to the encoded data includes: A multimodal reasoning request is generated and sent according to the encoded data and the guidance text.

8. The method according to claim 7, characterized in that The encoded data includes the encoded data of each modality of the multimodal data; The method further comprises: Grouping the coded data of each modal data to obtain multiple groups of coded data, each group of coded data including coded data of at least one modal data; For each set of coded data, construct a request array based on the coded data of at least one modality data and the guide text; The request array constructed for each set of encoded data is saved to the local memory.

9. The method according to claim 8, characterized in that The generating and sending a multimodal reasoning request according to the encoded data and the guidance text includes: For each set of encoded data, generate and send a multimodal reasoning request according to the request array; After persisting the encoded data to the local disk, releasing the multimodal data and the corresponding encoded data in the local memory includes: After persisting the encoded data and the guidance text to the local disk, releasing the multimodal data, the corresponding encoded data, and the guidance text in the local memory; The receiving of the inference result fed back in response to the multimodal inference request includes: For each set of coded data, an inference result fed back in response to the multimodal inference request is received; the inference result includes: an inference result obtained by separately inferring the guidance text in the same set of coded data and each of the coded data.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: generating a first local variable for the multimodal data and a second local variable for the encoded data, wherein a first reference count of the first local variable and a second reference count of the second local variable are both preset values; The releasing of the multimodal data and the corresponding encoded data in the local memory includes: gradually adjusting the first reference count from the preset value to zero, and releasing the multimodal data in the local memory; The second reference count is gradually adjusted from the preset value to zero, and the encoded data in the local memory is released.

11. The method according to claim 10, characterized in that The method further comprises: When it is determined that the first reference count and the second reference count are circularly referenced to each other, the first reference count is gradually adjusted from the preset value to zero while the second reference count is gradually adjusted from the preset value to zero.

12. A request processing device, characterized in that: The device comprises: The data acquisition module is used to read multimodal data from the local disk into the local memory for data encoding to obtain encoded data; a request processing module, configured to generate and send a multimodal reasoning request based on the encoded data; A data processing module, configured to persist the encoded data to a local disk and then release the multimodal data and corresponding encoded data in the local memory, wherein the encoded data in the local disk is used to re-initiate a multimodal inference request based on the encoded data in the event of a request failure; The inference result receiving module is used to receive the inference result fed back to the multimodal inference request.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images