A visual multi-modal large model inference method, device, equipment and medium

Abstract

The application discloses a visual multi-modal large model reasoning method and device, equipment and medium, relates to the technical field of model reasoning, and is used for solving the technical problems of low reasoning efficiency of the existing visual multi-modal large model reasoning method. The method comprises the following steps: inputting to-be-reasoned text and to-be-reasoned pictures into a visual multi-modal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2; wherein the visual multi-modal one-stage execution layer is provided with a text encoder and a picture encoder, and the text encoder and the picture encoder are executed in parallel; inputting the sequence data 1 and the sequence data 2 into a visual multi-modal two-stage execution layer in sequence for processing to obtain a plurality of predicted text vectors; and decoding the plurality of predicted text vectors by using a text decoder to obtain a target reasoning text field. Therefore, the application can improve the model reasoning efficiency by executing operations such as the text encoder and the picture encoder in parallel.

Description

Technical Field

[0001] This application relates to the field of model reasoning technology, and provides a visual multimodal large model reasoning method, apparatus, device and medium. Background Technology

[0002] With the development of large language models such as ChatGPT / Deepseek / Qwen and large visual multimodal models such as Qwen2.5VL / Gemma3, various industries are gradually accepting the use of large models for various applications. In practice, deployment using Nvidia's GPU servers or Huawei Ascend 10 series servers is quite popular, as side-end deployment of large models offers significant advantages such as improved response speed, reduced costs, and flexible deployment. Based on this, Qualcomm, Rockchip, Nvidia, and others are gradually supporting side-end deployment of large models, especially for large language models, where they have some mature solutions. For large visual multimodal models, the industry is also in a phase of rapid development; however, there is no unified and mature solution yet.

[0003] Generally, the application of large models on the side can be divided into two stages: 1. Converting the original model into a side model. 2. Deploying the side model for inference on side devices. For example, Qualcomm has already released some visual multimodal conversion model projects to partners, who need to make custom modifications to obtain the QNNBIN side model. Then, users still need to design, orchestrate, customize, and optimize the QNN SDK for side inference. This clearly does not form a unified framework process for visual multimodal large model inference.

[0004] Therefore, how to provide an efficient inference solution for the side of a large visual multimodal model has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a visual multimodal large model inference method, apparatus, device and medium to solve the technical problem of low inference efficiency in existing visual multimodal large model inference methods.

[0006] On the one hand, a visual multimodal large model inference method is provided, applied to an Android / Linux converged system, the method comprising: The text and image to be reasoned are input into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2. The visual multimodal one-stage execution layer is equipped with a text encoder and an image encoder, which are executed in parallel. Sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during the parallel time. Sequence data 2 is obtained by segmenting the remaining image vectors generated outside the parallel time. The sequence data 1 and the sequence data 2 are sequentially input into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors; wherein, the visual multimodal two-stage execution layer is equipped with an LLM inference model; The multiple predicted text vectors are decoded using a text decoder to obtain the target inference text field.

[0007] Optionally, the visual multimodal one-stage execution layer also includes a vector fusion unit and a slicer. The step of inputting the text and image to be reasoned into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2 includes: During the text encoding process, the text encoder and image encoder, which are executed in parallel in the visual multimodal one-stage execution layer, are used to encode the input text to be reasoned and the image to be reasoned, respectively, to obtain text vectors and image vectors. The vector fusion unit and the image vector are fused and segmented using the vector fusion unit and the slicer in the visual multimodal one-stage execution layer to obtain the sequence data 1; After the text encoding is completed, the image encoder in the visual multimodal one-stage execution layer is used to encode the input images to be inferred, and obtain multiple remaining image vectors. The remaining image vector is sliced ​​using the slicer in the visual multimodal one-stage execution layer to obtain the sequence data 2.

[0008] Optionally, the LLM inference model includes an LLM Prefill executor, an LLMDecode executor, and a sampler. The step of sequentially inputting the sequence data 1 and the sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors includes: The LLM Prefill executor in the LLM inference model is used to process the sequence data 1 and the sequence data 2 sequentially to obtain key-value pairs (KV); wherein the LLM Prefill executor is executed in parallel with the text encoder and the image encoder. The LLM Decode executor in the LLM inference model is used to process the key-value pairs (KV) to obtain multiple candidate text vectors; The sampler in the LLM inference model is used to sample the multiple candidate text vectors to obtain the multiple predicted text vectors.

[0009] Optionally, the visual multimodal two-stage execution layer also includes a QNN high-speed I / O unit. The step of using the LLMDecode executor in the visual multimodal two-stage execution layer to process the key-value pairs (KV) and obtain multiple candidate text vectors includes: The QNN high-speed I / O device is used to transmit the key-value pair (KV) to the LLM Decode executor; The received key-value pairs (KV) are processed using the LLMDecode executor to obtain the multiple candidate text vectors.

[0010] Optionally, after obtaining sequence data 1 and sequence data 2, the method further includes: KV caching is performed using a CPU / DSP data sharing unit; and, The CPU / DSP data sharing unit is used to transmit data between the text encoder and the LLM Prefill executor, between the image encoder and the LLM Prefill executor, and between the LLM Prefill executor and the LLMDecode executor.

[0011] Optionally, after obtaining sequence data 1 and sequence data 2, the method further includes: A global memory manager is used to analyze the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor respectively, and calculate their respective memory spaces; wherein, the memory space is used to store the data flowing in and out of the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor.

[0012] Optionally, before sequentially inputting the sequence data 1 and the sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors, the method further includes: The LLM model is deployed using the Qualcomm QNN BIN model.

[0013] On the one hand, a visual multimodal large model inference device is provided, applied to an Android / Linux converged system, the device comprising: The first execution unit is used to input the text and image to be reasoned into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2. The visual multimodal one-stage execution layer is equipped with a text encoder and an image encoder, which are executed in parallel. The sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during the parallel time. The sequence data 2 is obtained by segmenting the remaining image vectors generated outside the parallel time. The second execution unit is used to input the sequence data 1 and the sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors; wherein, the visual multimodal two-stage execution layer is equipped with an LLM inference model; The decoding unit is used to decode the plurality of predicted text vectors using a text decoder to obtain the target inference text field.

[0014] On one hand, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.

[0015] On the one hand, a storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement any of the methods described above.

[0016] Compared with the prior art, the beneficial effects of this application are as follows: In this application, during visual multimodal large model inference, firstly, the text and image to be inferred can be input into the first-stage execution layer of the visual multimodal model for processing to obtain sequence data 1 and sequence data 2. The first-stage execution layer includes a text encoder and an image encoder, which execute in parallel. Sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during parallel execution, while sequence data 2 is obtained by segmenting the remaining image vectors generated outside of parallel execution. Then, sequence data 1 and sequence data 2 can be sequentially input into the second-stage execution layer of the visual multimodal model for processing to obtain multiple predicted text vectors. The second-stage execution layer includes an LLM inference model. Finally, a text decoder can be used to decode the multiple predicted text vectors to obtain the target inference text field.

[0017] Based on this, in this application, since the inference process is divided into two execution layers and the text encoder and image encoder are executed in parallel, compared with the prior art, this application can accelerate the processing speed through parallelism on the one hand, and further improve the inference efficiency by processing the data difference between sequence data 1 and sequence data 2 through the visual multimodal two-stage execution layer. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application; Figure 2 A schematic diagram of a visual multimodal large model inference method provided in an embodiment of this application; Figure 3 A schematic diagram of a software structure for visual multimodal large model inference provided in an embodiment of this application; Figure 4 A specific schematic diagram of the inference images provided in the embodiments of this application; Figure 5 This is a schematic diagram of a visual multimodal large model inference device provided in an embodiment of this application.

[0020] The diagram is labeled as follows: 10-Visual multimodal large model inference device, 101-Processor, 102-Memory, 103-I / O interface, 104-Database, 50-Visual multimodal large model inference device, 501-First execution unit, 502-Second execution unit, 503-Decoding unit, 504-Data transmission and analysis unit, 505-Model deployment unit. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0022] With the development of large language models such as ChatGPT / Deepseek / Qwen and large visual multimodal models such as Qwen2.5VL / Gemma3, various industries are gradually accepting the use of large models for various applications. In practice, deployment using Nvidia's GPU servers or Huawei Ascend 10 series servers is quite popular, as side-end deployment of large models offers significant advantages such as improved response speed, reduced costs, and flexible deployment. Based on this, Qualcomm, Rockchip, Nvidia, and others are gradually supporting side-end deployment of large models, especially for large language models, where they have some mature solutions. For large visual multimodal models, the industry is also in a phase of rapid development; however, there is no unified and mature solution yet.

[0023] Generally, the application of large models on the side can be divided into two stages: 1. Converting the original model into a side model. 2. Deploying the side model for inference on side devices. For example, Qualcomm has already released some visual multimodal conversion model projects to partners, who need to make custom modifications to obtain the QNNBIN side model. Then, users still need to design, orchestrate, customize, and optimize the QNN SDK for side inference. This clearly does not form a unified framework process for visual multimodal large model inference.

[0024] Based on this, this application provides a visual multimodal large model inference method. In this method, firstly, the text and image to be inferred can be input into the visual multimodal first-stage execution layer for processing to obtain sequence data 1 and sequence data 2. The visual multimodal first-stage execution layer includes a text encoder and an image encoder, which execute in parallel. Sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during parallel execution, while sequence data 2 is obtained by segmenting the remaining image vectors generated outside of parallel execution. Then, sequence data 1 and sequence data 2 can be sequentially input into the visual multimodal second-stage execution layer for processing to obtain multiple predicted text vectors. The visual multimodal second-stage execution layer includes an LLM inference model. Finally, a text decoder can be used to decode the multiple predicted text vectors to obtain the target inference text field. Based on this, in this application, since the inference process is divided into two execution layers and the text encoder and image encoder are executed in parallel, compared with the prior art, this application can accelerate the processing speed through parallelism on the one hand, and further improve the inference efficiency by processing the data difference between sequence data 1 and sequence data 2 through the visual multimodal two-stage execution layer.

[0025] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0026] like Figure 1 The diagram shown illustrates an application scenario provided by an embodiment of this application. This application scenario may include a visual multimodal large model inference device 10.

[0027] The visual multimodal large model inference device 10 can be used for performing visual multimodal large model inference, and can be, for example, an in-vehicle computer, a personal computer (PC), a server, or a laptop. The visual multimodal large model inference device 10 may include one or more processors 101, memory 102, I / O interfaces 103, and a database 104. Specifically, the processor 101 can be a central processing unit (CPU) or a digital processing unit, etc. The memory 102 can be volatile memory, such as random-access memory (RAM); the memory 102 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory 102 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 102 can be a combination of the above-mentioned memories. The memory 102 can store some program instructions of the visual multimodal large model inference method provided in the embodiments of this application. When these program instructions are executed by the processor 101, they can be used to implement the steps of the visual multimodal large model inference method provided in the embodiments of this application, so as to solve the technical problems such as low inference efficiency of existing visual multimodal large model inference methods. The database 104 can be used to store data such as text, images, sequence data 1, sequence data 2, and target inference text fields involved in the solution provided in the embodiments of this application.

[0028] In this embodiment, the visual multimodal large model inference device 10 can obtain inference instructions through the I / O interface 103. Then, the processor 101 of the visual multimodal large model inference device 10 will solve the technical problems such as low inference efficiency of existing visual multimodal large model inference methods according to the program instructions of the visual multimodal large model inference method provided in this embodiment in the memory 102. In addition, text, images, sequence data 1, sequence data 2, and target inference text fields can be stored in the database 104.

[0029] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible scenarios, and this application embodiment does not impose any limitations. Figure 1The functions that the various devices in the application scenarios shown can achieve will be described in subsequent method embodiments, and will not be elaborated on here. Below, the methods of the embodiments of this application will be described in conjunction with the accompanying drawings.

[0030] like Figure 2 The diagram shown is a flowchart of a visual multimodal large model inference method provided in this application embodiment. This method is applied to an Android / Linux converged system and can be used... Figure 1 The visual multimodal large model inference device 10 is used to execute this method. Specifically, the process of this method is described as follows.

[0031] Step 201: Input the text and image to be reasoned into the first-stage execution layer of the visual multimodal system for processing to obtain sequence data 1 and sequence data 2.

[0032] like Figure 3 The diagram shown is a software structure diagram of a visual multimodal large model inference provided in an embodiment of this application. The visual multimodal first-stage execution layer is provided with a text encoder and an image encoder. The text encoder and the image encoder are executed in parallel. Sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during the parallel time. Sequence data 2 is obtained by segmenting the remaining image vectors generated outside the parallel time.

[0033] like Figure 3 As shown, the visual multimodal one-stage execution layer also includes a vector fusion unit and a slicer. Based on this, since the image encoder's runtime is much longer than the text encoder's runtime, when inputting the text and image to be inferred into the visual multimodal one-stage execution layer to obtain sequence data 1 and sequence data 2, firstly, during text encoding, the text encoder and image encoder, which execute in parallel within the visual multimodal one-stage execution layer, encode the input text and image to be inferred respectively to obtain text vectors and image vectors. Then, the vector fusion unit and slicer in the visual multimodal one-stage execution layer fuse and slice the text vectors and image vectors to obtain sequence data 1. Next, after text encoding is complete, the image encoder in the visual multimodal one-stage execution layer continues to encode the input image to be inferred to obtain the remaining image vectors. Finally, the slicer in the visual multimodal one-stage execution layer slices the remaining image vectors to obtain sequence data 2.

[0034] Furthermore, compared to the existing technology's "serial processing" method where text encoding must be completed before image encoding, the parallel execution of text encoder and image encoder in this application significantly accelerates the processing speed and further improves reasoning efficiency.

[0035] Step 202: Input sequence data 1 and sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors.

[0036] like Figure 3 As shown, the visual multimodal two-stage execution layer is equipped with an LLM inference model, which includes an LLMPrefill executor, an LLMDecode executor, and a sampler.

[0037] Based on this, since sequence data has the special characteristics of being continuous and serial, and the image encoder has the characteristic of long execution time, when inputting sequence data 1 and sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors, "sequence data 1" can be executed in advance, and the remaining "sequence data 2" can be executed after the image encoder has finished executing.

[0038] First, the LLM Prefill executor in the LLM inference model can be used to process sequence data 1 and sequence data 2 sequentially (forward propagation) to obtain key-value pairs (KV) in the hidden layer, preparing for the subsequent generation of output candidate text vectors. The LLM Prefill executor is executed in parallel with the text encoder and image encoder. Then, the LLMDecode executor in the LLM inference model is used to process the key-value pairs (KV) to obtain multiple candidate text vectors. That is, after the Prefill stage is completed, output candidate text vectors can be generated one by one. Each time a new candidate text vector is generated, it relies on the previously accumulated KV cache and is iteratively calculated using an autoregressive method. Finally, the sampler in the LLM inference model can be used to sample multiple candidate text vectors to obtain multiple predicted text vectors. That is, the candidate text vector with the highest probability can be selected from the multiple candidate text vectors output by the model as the current output. The sampling strategy can be Top-k sampling, greedy search, etc.

[0039] In this application, the sampler is fixed, and one candidate character vector corresponds to one character. Therefore, sampling the candidate character vector is equivalent to selecting the appropriate next character from the candidate characters. Of course, to speed up the output of large models, multiple characters can also be sampled.

[0040] Step 203: Use a text decoder to decode multiple predicted text vectors to obtain the target inference text field.

[0041] In one possible implementation, such as Figure 3As shown, a "QNN high-speed IO device" can also be set in the visual multimodal two-stage execution layer, so that when the user performs the associated processing of the three models of image encoder, LLM prefill executor and LLM decode executor, the data processing can be selectively performed, such as whether to perform quantization, dequantization and share data blocks.

[0042] Based on this, in this application, when using the LLMDecode executor in the visual multimodal two-stage execution layer to process key-value pairs (KV) and obtain multiple candidate text vectors, a QNN high-speed I / O device can be used to transmit the key-value pairs (KV) to the LLMDecode executor; then, the LLMDecode executor processes the received key-value pairs (KV) to obtain multiple candidate text vectors. Of course, as... Figure 3 As shown, when updating the KV cache, the generated candidate text vectors can also be fed back to the LLM Prefill executor through the QNN high-speed IO device to update the KV cache.

[0043] In one possible implementation, such as Figure 3 As shown, a "CPU / DSP data sharer" can be set in the large model inference architecture.

[0044] Specifically, after obtaining sequence data 1 and sequence data 2, a CPU / DSP data sharing unit can be used for key-value caching; and a CPU / DSP data sharing unit can be used to transfer data between the image encoder and the LLM Prefill executor, and between the LLM Prefill executor and the LLM Decode executor. Furthermore, a CPU / DSP data sharing unit is also used to transfer data processed by the QNN high-speed I / O unit.

[0045] As can be seen, this application can enable efficient data transmission between the CPU and DSP through a "CPU / DSP data sharing device", and in some cases, the CPU and DSP can even share the same address space.

[0046] In one possible implementation, such as Figure 3 As shown, a "global memory manager" can also be set in the large model inference architecture.

[0047] Specifically, after obtaining sequence data 1 and sequence data 2, a global memory manager can be used to analyze the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor respectively, and calculate their respective memory spaces; the memory space is used to store the data flowing in and out of the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor.

[0048] As can be seen, this application can analyze the execution flow and input data through a "global memory manager" to calculate a suitable memory space, thereby avoiding repeated allocation, release, and copying of space. In other words, by obtaining the input data, a unified memory space can be established in advance to reduce the performance impact of frequent memory allocation, release, and copying.

[0049] Based on this, the image encoder can utilize the unified memory space and "CPU / DSP data sharer" mentioned above to reduce CPU / DSP data copy time. Furthermore, by leveraging the unified memory space and the direct memory access capabilities of C++, it can perform data transformation directly in the original space in one step (for example, two reshape operations and one transpose operation in the original Python can be completed in one step in C++), thereby reducing preprocessing time.

[0050] In one possible implementation, in order to improve the convenience of deploying visual multimodal in various scenarios, in this application, before sequentially inputting sequence data 1 and sequence data 2 into the two-stage execution layer of visual multimodal for processing to obtain multiple predicted text vectors, the Qualcomm QNN BIN model can be used to deploy the LLM model.

[0051] Specific experimental examples: like Figure 4 The image shown is a specific schematic diagram of a reasoning image provided in an embodiment of this application.

[0052] The PC card uses the 9728550 board, and then it can be used to... Figure 3 In the large-scale visual multimodal model shown, the input text is "What does this image describe?", and so on. Figure 4 The inference image shown.

[0053] After the visual multimodal large model inference is completed, the output will be "This image depicts a woman and her dog on a beach. The woman is sitting on the ground, interacting with her dog. The dog is wearing a collar and seems to be interacting with or responding to the woman's hand, showing a trusting and friendly relationship. The background is a beach, with waves gently lapping the shore. Under the sunlight, the whole scene appears very peaceful and warm, conveying a theme of harmony and companionship." Furthermore, its initial response time from text and image input to the output of the first character is 900ms, achieving second-level response and greatly expanding its application scenarios. Moreover, its inference speed (Token speed) is 25 toks / s, meeting the needs of current scenarios.

[0054] In summary, this application has the following advantages: (1) It solves the problem of efficient side-end inference for large visual multimodal models in a fusion system based on Qualcomm System on Chip (SoC).

[0055] (2) After the fusion system based on Qualcomm SOC supports visual multimodal large models, the cost and power consumption are reduced compared to the server.

[0056] (3) After the fusion system based on Qualcomm SOC supports visual multimodal large models, it improves the convenience of visual multimodal deployment in various scenarios compared to the server, and also supports more side-end landing scenarios.

[0057] Based on the same inventive concept, embodiments of this application provide a visual multimodal large-scale model inference device 50, such as... Figure 5 As shown, this visual multimodal large model inference device 50 is applied to an Android / Linux converged system and includes: The first execution unit 501 is used to input the text and image to be reasoned into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2. The visual multimodal one-stage execution layer is equipped with a text encoder and an image encoder, which are executed in parallel. Sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during the parallel time. Sequence data 2 is obtained by segmenting the remaining image vectors generated outside the parallel time. The second execution unit 502 is used to input sequence data 1 and sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors; wherein, the visual multimodal two-stage execution layer is equipped with an LLM inference model; Decoding unit 503 is used to decode multiple predicted text vectors using a text decoder to obtain the target inference text field.

[0058] Optionally, the first execution unit 501 is also used for: In the text encoding process, the text encoder and image encoder, which are executed in parallel in the visual multimodal one-stage execution layer, encode the input text to be inferred and the image to be inferred respectively, to obtain text vectors and image vectors. The vector fusion and slicer in the visual multimodal one-stage execution layer are used to fuse and slice the text vectors and image vectors to obtain sequence data 1; After the text encoding is completed, the image encoder in the visual multimodal one-stage execution layer is used to encode the input images to be inferred, and obtain multiple remaining image vectors. The remaining image vectors are sliced ​​using the slicer in the visual multimodal one-stage execution layer to obtain sequence data 2.

[0059] Optionally, the second execution unit 502 is also used for: The LLM Prefill executor in the LLM inference model is used to process sequence data 1 and sequence data 2 sequentially to obtain key-value pairs (KV); the LLM Prefill executor is executed in parallel with the text encoder and the image encoder. The LLMDecode executor in the LLM inference model is used to process key-value pairs (KV) to obtain multiple candidate text vectors; The sampler in the LLM inference model is used to sample multiple candidate text vectors to obtain multiple predicted text vectors.

[0060] Optionally, the second execution unit 502 is also used for: The QNN high-speed I / O unit is used to transmit key-value pairs (KV) to the LLM Decode executor; The LLMDecode executor is used to process the received key-value pairs (KV) to obtain multiple candidate text vectors.

[0061] Optionally, the visual multimodal large model inference device 50 also includes a data transmission and analysis unit 504, used for: KV caching is performed using a CPU / DSP data sharing unit; and, A CPU / DSP data sharing unit is used to transmit data between the image encoder and the LLM Prefill executor, and between the LLM Prefill executor and the LLMDecode executor.

[0062] Optionally, the data transmission and analysis unit 504 is also used for: A global memory manager is used to analyze the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor separately, and calculate their respective memory spaces. The memory space is used to store the data flowing in and out of the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor.

[0063] Optionally, the visual multimodal large model inference device 50 also includes a model deployment unit 505 for: The Qualcomm QNN BIN model is used to deploy the LLM model.

[0064] The visual multimodal large model inference device 50 can be used to perform... Figures 2-3 The method executed by the visual multimodal large model inference device in the illustrated embodiment can be referenced here. Therefore, the functions that each functional module of the visual multimodal large model inference device 50 can achieve can be found in the following examples. Figures 2-3 The embodiments shown are described in detail below.

[0065] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device may perform actions such as... Figures 2-3 The method performed by the visual multimodal large model inference device in the illustrated embodiment.

[0066] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0067] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0068] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A visual multimodal large-scale model inference method, characterized in that, Applied to Android / Linux converged systems, the method includes: The text and image to be reasoned are input into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2. The visual multimodal one-stage execution layer is equipped with a text encoder and an image encoder, which are executed in parallel. Sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during the parallel time. Sequence data 2 is obtained by segmenting the remaining image vectors generated outside the parallel time. The sequence data 1 and the sequence data 2 are sequentially input into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors; wherein, the visual multimodal two-stage execution layer is equipped with an LLM inference model; The multiple predicted text vectors are decoded using a text decoder to obtain the target inference text field.

2. The method as described in claim 1, characterized in that, The visual multimodal one-stage execution layer also includes a vector fusion unit and a slicer. The step of inputting the text and image to be reasoned into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2 includes: During the text encoding process, the text encoder and image encoder, which are executed in parallel in the visual multimodal one-stage execution layer, are used to encode the input text to be reasoned and the image to be reasoned, respectively, to obtain text vectors and image vectors. The vector fusion unit and the image vector are fused and segmented using the vector fusion unit and the slicer in the visual multimodal one-stage execution layer to obtain the sequence data 1; After the text encoding is completed, the image encoder in the visual multimodal one-stage execution layer is used to encode the input images to be inferred, and obtain multiple remaining image vectors. The remaining image vector is sliced ​​using the slicer in the visual multimodal one-stage execution layer to obtain the sequence data 2.

3. The method as described in claim 2, characterized in that, The LLM inference model includes an LLM Prefill executor, an LLMDecode executor, and a sampler. The step of sequentially inputting the sequence data 1 and the sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors includes: The LLM Prefill executor in the LLM inference model is used to process the sequence data 1 and the sequence data 2 sequentially to obtain key-value pairs (KV); wherein the LLM Prefill executor is executed in parallel with the text encoder and the image encoder. The LLM Decode executor in the LLM inference model is used to process the key-value pairs (KV) to obtain multiple candidate text vectors; The sampler in the LLM inference model is used to sample the multiple candidate text vectors to obtain the multiple predicted text vectors.

4. The method as described in claim 3, characterized in that, The visual multimodal two-stage execution layer also includes a QNN high-speed I / O unit. The step of using the LLM Decode executor in the LLM inference model to process the key-value pairs (KV) and obtain multiple candidate text vectors includes: The QNN high-speed I / O device is used to transmit the key-value pair (KV) to the LLM Decode executor; The received key-value pairs (KV) are processed using the LLMDecode executor to obtain the multiple candidate text vectors.

5. The method as described in claim 3, characterized in that, After obtaining sequence data 1 and sequence data 2, the method further includes: KV caching is performed using a CPU / DSP data sharing unit; and, The CPU / DSP data sharing unit is used to transmit data between the image encoder and the LLM Prefill executor, and between the LLM Prefill executor and the LLMDecode executor.

6. The method as described in claim 3, characterized in that, After obtaining sequence data 1 and sequence data 2, the method further includes: A global memory manager is used to analyze the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor respectively, and calculate their respective memory spaces; wherein, the memory space is used to store the data flowing in and out of the text encoder, image encoder, LLM Prefill executor, and LLMDecode executor.

7. The method as described in claim 1, characterized in that, Before sequentially inputting the sequence data 1 and the sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors, the method further includes: The LLM model is deployed using the Qualcomm QNN BIN model.

8. A visual multimodal large-scale model inference device, characterized in that, The device, used in an Android / Linux converged system, includes: The first execution unit is used to input the text and image to be reasoned into the visual multimodal one-stage execution layer for processing to obtain sequence data 1 and sequence data 2. The visual multimodal one-stage execution layer is equipped with a text encoder and an image encoder, which are executed in parallel. The sequence data 1 is obtained by fusing and segmenting the text vectors and image vectors generated during the parallel time. The sequence data 2 is obtained by segmenting the remaining image vectors generated outside the parallel time. The second execution unit is used to input the sequence data 1 and the sequence data 2 into the visual multimodal two-stage execution layer for processing to obtain multiple predicted text vectors; wherein, the visual multimodal two-stage execution layer is equipped with an LLM inference model; The decoding unit is used to decode the plurality of predicted text vectors using a text decoder to obtain the target inference text field.

9. An electronic device, characterized in that, The device includes: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method described in any one of claims 1-7 according to the obtained program instructions.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions for causing a computer to perform the method described in any one of claims 1-7.

Images