Multi-mode large-model edge accelerator system and implementation method, device, equipment and medium thereof

Abstract

The invention provides a multi-mode large model edge accelerator system and an implementation method, device, equipment and medium thereof, and belongs to the technical field of multi-mode large model calculation. The system comprises a multi-mode large model and an acceleration bus; a reconfigurable edge computing accelerator, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and a memory module are hung on the acceleration bus; the reconfigurable edge calculation accelerator performs acceleration calculation on the data processing process of the multi-modal large model; the CPU and the GPU accelerate calculation of the multi-mode large model according to needs, the CPU further performs overall control on an accelerator system, and the GPU further accelerates an image processing task of the accelerator system; and the memory module stores input data, model parameters and output data of the multi-mode large model through a memory chip. According to the method, the CPU, the GPU and the reconfigurable accelerator are flexibly configured according to the computing power requirement, low-delay calculation of a large model is achieved, and the power consumption of the edge heterogeneous accelerator is effectively reduced.

Description

Background Art

[0002] LLM, short for Large Language Model, is a language model built by a deep neural network with more than billions of parameters. It is usually trained using a large amount of unlabeled text using self-supervised learning methods.

[0003] The multimodal large model is a neural network model that can process multiple input types such as text, voice, image, video, etc. It can process multiple data types at the same time, thereby better simulating human multimodal perception and understanding capabilities.

[0004] At present, the relevant multimodal large model reasoning system includes a multimodal accelerated reasoning unit, a search unit and a cache unit. The multimodal accelerated reasoning unit generates reasoning results based on multimodal information reasoning; the search unit searches for reasoning results in the database based on multimodal information; the cache unit saves the reasoning results to the database. The multimodal large model reasoning system as a whole accelerates the calculation of different models (such as visual encoders and LMM, etc.) through different computing nodes, so that the multimodal accelerated reasoning unit has a large hardware area and high power consumption. Although there are currently related fusion architecture systems that adopt heterogeneous forms, such as computing resource pools, memory resource pools, graphics processor resource pools, heterogeneous accelerator resource pools, and storage resource pools built on solid-state drives, each resource pool uses different types of chips and has high performance, but the fusion system still does not involve relevant optimization and design of multimodal large model calculations.

[0005] Therefore, in view of the above-mentioned defects, it is very necessary to provide a multi-modal large-model edge accelerator system and its implementation method, device, equipment, and medium. Summary of the invention

[0006] The above-mentioned multimodal large model accelerates different models separately through different computing nodes, which results in a large hardware area and high power consumption. Although the fusion system using a heterogeneous architecture involves different types of chips but does not have the defect of related applications of multimodal large model calculation, the present invention provides a multimodal large model edge accelerator system and its implementation method, device, equipment, and medium to solve the above-mentioned technical problems.

[0007] In a first aspect, the present invention provides a multimodal large model edge accelerator system, comprising a multimodal large model and an acceleration bus;

[0008] The acceleration bus is equipped with a reconfigurable edge computing accelerator, CPU, GPU, and memory module;

[0009] Reconfigurable edge computing accelerator to accelerate the data processing of multi-modal large models;

[0010] The CPU performs overall control of the accelerator system and accelerates the calculation of multi-modal large models as needed;

[0011] GPU, which accelerates the image processing tasks of the accelerator system and the calculation of multimodal large models as needed;

[0012] The memory module saves the input data, model parameters and output data of the multimodal large model through the storage chip.

[0013] Further, the multimodal large model includes an encoder module, a large model module and a generator module;

[0014] The encoder module encodes each data type and provides it to the large model module;

[0015] The large model module encodes various types of data and processes them uniformly before providing them to the generator module;

[0016] The generator module reconstructs and generates the uniformly processed data according to the corresponding original data type.

[0017] Further, the encoder module includes a text encoder, a speech encoder, an image encoder and a video encoder;

[0018] Each encoder encodes text type, voice type, image type and video type data according to the data type;

[0019] The generator module includes a text generator, a speech generator, an image generator and a video generator. The data processed by the large model module are uniformly reconstructed according to the original data type to generate text, speech, image and video.

[0020] Further, the reconfigurable edge computing accelerator includes a dedicated computing module, a reconfigurable computing module, and a shared computing module;

[0021] Dedicated computing modules and reconfigurable computing modules are used to accelerate the computing of different encoders of the encoder module and different generators of the generator module;

[0022] Shared computing modules to accelerate computing of large model modules.

[0023] Furthermore, the dedicated computing module includes:

[0024] The text computing unit preprocesses text data and performs model calculations through a text processing neural network model to accelerate the text encoder and text generator;

[0025] The speech computing unit pre-processes speech data and performs model calculations through a speech processing neural network model to accelerate the speech encoder and speech generator;

[0026] Image computing unit, which pre-processes image data and performs model calculations through an image processing neural network model to accelerate the image encoder and image generator;

[0027] The video computing unit preprocesses the video data and performs model calculations through the video processing neural network model to accelerate the video encoder and video generator.

[0028] Further, the reconfigurable computing module includes a plurality of reconfigurable computing units;

[0029] Each reconfigurable computing unit performs collaborative computing with the dedicated computing module to supplement computing power, and flexibly configures computing power between the reconfigurable computing module and the dedicated computing module and between each reconfigurable computing unit according to computing power requirements.

[0030] Further, each reconfigurable computing unit includes a first type of reconfigurable computing unit and a second type of reconfigurable computing unit;

[0031] The first type of reconfigurable computing unit is used for data preprocessing;

[0032] The second type of reconfigurable computing unit is used for performing model calculations.

[0033] Furthermore, the shared computing module includes:

[0034] A linear computing unit for accelerating the linear part of a neural network model as a large model;

[0035] A nonlinear calculation unit is used to accelerate the calculation of the nonlinear part of the neural network model as a large model. Further, it also includes an extended function module;

[0036] The extended function modules include clock module and power module;

[0037] A clock module is used to synchronize the data acceleration computing process of the reconfigurable edge computing accelerator, CPU, GPU and memory module on the acceleration bus;

[0038] The power module is used to power the data acceleration computing process of the reconfigurable edge computing accelerator, CPU, GPU and memory module on the acceleration bus.

[0039] In a second aspect, the present invention provides a method for implementing a multi-modal large model edge accelerator system according to the first aspect, comprising the following steps:

[0040] S1. Determine the type of computing task to be processed based on the application scenario of the multimodal large model;

[0041] S2. Analyze the computing power requirements and priorities of each computing task, and allocate computing power to the reconfigurable edge accelerator, CPU, and GPU based on the analysis results;

[0042] S3. Configure the dedicated computing module, reconfigurable computing module and shared computing module of the reconfigurable edge accelerator according to the task type of each computing task;

[0043] S4. Use the configured accelerator system to accelerate the data processing process of the multimodal large model.

[0044] Furthermore, the specific steps of step S1 are as follows:

[0045] S11. Obtain application scenarios of multimodal large models;

[0046] When it is a visual question and answer application scenario, go to step S12;

[0047] When it is a speech recognition application scenario, go to step S13;

[0048] When it is a sentiment analysis application scenario, go to step S14;

[0049] When it is a custom application scenario, go to step S15;

[0050] S12. Determine that the multimodal large model needs to process text and image data type computing tasks, and enter step S2; S13. Determine that the multimodal large model needs to process text and voice data type computing tasks, and enter step S2; S14. Determine that the multimodal large model needs to process text, voice, image and video data type computing tasks, and enter step S2;

[0051] S15. Determine whether the data type calculation task pre-defined in the custom application scenario is the data type calculation task that needs to be processed by the multimodal large model.

[0052] Furthermore, the specific steps of step S2 are as follows:

[0053] S21. Obtain the total computing power requirement of the computing task;

[0054] S22. Determine whether the total computing power requirement is less than the first computing power threshold;

[0055] If yes, go to step S23;

[0056] If not, proceed to step S24;

[0057] S23. The reconfigurable edge computing accelerator of the acceleration system works, while the CPU and GPU do not work;

[0058] S24. The reconfigurable edge computing accelerator of the acceleration system works, and the CPU and GPU are started for collaborative computing according to the specific number, type, priority, response speed requirements and power consumption requirements of computing tasks.

[0059] Furthermore, the specific steps of step S24 are as follows:

[0060] S241. The reconfigurable edge computing accelerator of the acceleration system is started;

[0061] S242. Obtain the total computing power requirement of the computing task, the number of computing tasks, the response speed requirement, and the power consumption requirement; when the total computing power requirement is greater than or equal to the first computing power threshold, less than the second computing power threshold, the number of computing tasks is greater than the number threshold, and the response speed requirement is higher than the speed threshold, proceed to step S243;

[0062] When the total computing power requirement is greater than or equal to the second computing power threshold, the number of computing tasks is greater than the number threshold, the response speed requirement is higher than the speed threshold, and the power consumption requirement is lower than the power consumption threshold, proceed to step S244;

[0063] S243. Start the GPU to perform auxiliary calculations, and proceed to step S245;

[0064] S244. Start the GPU and CPU simultaneously for auxiliary calculation;

[0065] S245. Obtain the priority and computing power requirements of each computing task, and allocate computing resources of the reconfigurable edge accelerator, GPU, and CPU to each task based on the priority and computing power requirements.

[0066] Furthermore, the specific steps of step S245 are as follows:

[0067] S2451. Identify the first computing task with high priority and small computing power requirement among the computing tasks, and assign a reconfigurable edge computing accelerator to the first computing task for collaborative computing;

[0068] S2452. Identify a second computing task with a low priority and high computing power requirement among the computing tasks, assign a GPU to the second computing task for collaborative computing, and assign a CPU to perform collaborative computing when the CPU is started. Further, the specific steps of step S3 are as follows:

[0069] S31. Get each computing task type;

[0070] When it is a single data type computing task, go to step S32;

[0071] When it is a multi-data type computing task, go to step S35;

[0072] S32. Split the single data type task into encoding subtasks, large model calculation subtasks, and generation subtasks to determine whether the dedicated computing module meets the computing power requirements of the single data type computing task;

[0073] If yes, go to step S33;

[0074] If not, proceed to step S34;

[0075] S33. Select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and generation subtask of each single data type computing task, select the shared computing module in the reconfigurable edge computing accelerator to accelerate the large model computing subtask, and enter step S4;

[0076] S34. Select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and the generation subtask of the single data type computing task, and configure the reconfigurable computing unit of the reconfigurable computing module to accelerate the computing of the corresponding task type, select the shared computing module in the reconfigurable edge computing accelerator to accelerate the computing of the large model computing subtask, and enter step S4;

[0077] S35. Split each multi-data type computing task into an encoding subtask, a large model computing subtask, and a generation subtask;

[0078] S36. Determine whether the dedicated computing module meets the computing power requirements of each computing task;

[0079] If yes, go to step S37;

[0080] If not, proceed to step S38;

[0081] S37. Select the computing unit corresponding to the encoding subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform the accelerated computing of each encoding subtask and then provide it to the shared computing module. Then, the large model computing subtask of each computing task is uniformly used in the shared computing module to perform linear and nonlinear calculations and provide them to the dedicated computing module to complete the accelerated calculation. Finally, select the computing unit corresponding to the generation subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform the accelerated calculation of each generation subtask, and enter step S4.

[0082] S38. Select the computing units corresponding to the encoding subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module to perform accelerated calculations of each encoding subtask and provide them to the shared computing module. Then, use the shared computing module to uniformly perform linear and nonlinear calculations on the large model computing subtasks of each computing task and provide them to the dedicated computing module to complete the accelerated calculation. Finally, select the computing units corresponding to the generation subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module to perform accelerated calculations of each generation subtask and enter step S4.

[0083] Furthermore, the specific steps of step S34 are as follows:

[0084] S341. Select a reconfigurable computing unit in the reconfigurable edge computing accelerator that matches the single data type;

[0085] S342. Obtain and compare the computing power requirements of each computing task of a single data type;

[0086] S343. Allocate N1 selected reconfigurable computing units to computing tasks with high computing power requirements, and allocate N2 selected reconfigurable computing units to computing tasks with low computing power requirements, and N1>N2.

[0087] Furthermore, the specific steps of step S38 are as follows:

[0088] S381. Determine the adaptation type of each reconfigurable computing unit of the reconfigurable computing module in the reconfigurable edge computing accelerator, and determine the first reconfigurable computing unit adapted for the encoding task and the second reconfigurable computing unit adapted for the data generation task;

[0089] S382. Determine the computing power requirements of each computing task of multiple data types;

[0090] S383. N3 first reconfigurable computing units are allocated to the encoding subtasks of the computing tasks of the data types with large computing power requirements, and N4 first reconfigurable computing units are allocated to the encoding subtasks of the computing tasks of the data types with small computing power requirements, and N3>N4;

[0091] S384. Assign N5 second reconfigurable computing units to the data generation subtask of the computing task of the data type with high computing power requirements, and assign N6 second reconfigurable computing units to the encoding subtask of the computing task of the data type with low computing power requirements, and N5>N6.

[0092] In a third aspect, the present invention provides an implementation device for a multimodal large-scale edge accelerator system, comprising: a computing task type determination module, for determining the type of computing task to be processed according to an application scenario of a multimodal large model;

[0093] The accelerator system computing power allocation module is used to analyze the computing power requirements and priorities of each computing task, and allocate the computing power of the reconfigurable edge accelerator, CPU, and GPU based on the analysis results;

[0094] The computing power allocation module of each module of the reconfigurable edge accelerator is used to configure the dedicated computing module, reconfigurable computing module and shared computing module of the reconfigurable edge accelerator according to the task type of each computing task; the computing acceleration module is used to use the configured accelerator system to accelerate the data processing process of the multimodal large model.

[0095] Furthermore, the computing task type determination module includes:

[0096] An application scenario acquisition unit, used to acquire application scenarios of a multimodal large model;

[0097] The first computing task determination unit is used in visual question-answering application scenarios to determine whether the multimodal large model needs to process computing tasks of text and image data types;

[0098] The second computing task determination unit is used in speech recognition application scenarios to determine whether the multimodal large model needs to process text and speech data type computing tasks;

[0099] The third computing task determination unit is used in sentiment analysis application scenarios to determine whether the multimodal large model needs to process computing tasks of text, speech, image and video data types;

[0100] The fourth computing task determination unit is used to determine whether a data type computing task pre-defined in a custom application scenario is a data type computing task that needs to be processed by the multimodal large model.

[0101] Furthermore, the accelerator system computing power allocation module includes:

[0102] A total computing power requirement calculation unit is used to obtain the total computing power requirement of a computing task;

[0103] A first total computing power requirement determination unit, configured to determine whether the total computing power requirement is less than a first computing power threshold;

[0104] A first computing power configuration unit is used to configure the reconfigurable edge computing accelerator of the acceleration system to work, and the CPU and GPU to not work when the total computing power demand is less than the first computing power threshold;

[0105] The second computing power configuration unit is used to configure the reconfigurable edge computing accelerator of the acceleration system to work when the total computing power demand is greater than or equal to the first computing power threshold, and to start the CPU and GPU for collaborative computing according to the specific number, type, priority, response speed requirements and power consumption requirements of computing tasks.

[0106] Furthermore, the computing power allocation module of each module of the reconfigurable edge accelerator includes:

[0107] A computing task type acquisition unit, used to acquire each computing task type;

[0108] The computing power requirement judgment unit for a single data type task is used to split the single data type task into an encoding subtask, a large model calculation subtask, and a generation subtask to judge whether the dedicated computing module meets the computing power requirement of the single data type calculation task;

[0109] The dedicated and shared computing module configuration unit for single data type tasks is used to select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to perform accelerated computing for the encoding subtask and generation subtask of each single data type computing task when the dedicated computing module meets the computing power requirements of the single data type computing task, and select the shared computing module in the reconfigurable edge computing accelerator to perform accelerated computing for the large model computing subtask;

[0110] The dedicated, shared and reconfigurable computing module configuration unit for single data type tasks is used to select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and generation subtask of the single data type computing task when the dedicated computing module does not meet the computing power requirements of the single data type computing task, and configure the reconfigurable computing unit of the reconfigurable computing module to accelerate the corresponding task type, and select the shared computing module in the reconfigurable edge computing accelerator to accelerate the large model computing subtask;

[0111] The computing task splitting unit, when used for multi-data type tasks, splits each multi-data type computing task into encoding subtasks, large model computing subtasks, and generation subtasks;

[0112] The computing power requirement judgment unit for most types of tasks is used to judge whether the dedicated computing module meets the computing power requirements of each computing task;

[0113] The dedicated and shared computing module configuration unit for multi-data type tasks is used for the dedicated computing module to meet the computing power requirements of each computing task, select the computing unit corresponding to the encoding subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform accelerated calculations of each encoding subtask and then provide it to the shared computing module, then use the shared computing module to uniformly perform linear and nonlinear calculations on the large model computing subtasks of each computing task and provide them to the dedicated computing module to complete the accelerated calculation, and finally select the computing unit corresponding to the generation subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform accelerated calculations of each generation subtask;

[0114] The dedicated, shared and reconfigurable computing module configuration units for tasks of multiple data types are used when the dedicated computing modules do not meet the computing power requirements of each computing task. The computing units corresponding to the encoding subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module are selected to perform accelerated calculations of each encoding subtask and then provided to the shared computing module. The large model computing subtasks of each computing task are uniformly calculated using the shared computing module for linear and nonlinear calculations and provided to the dedicated computing module to complete the accelerated calculation. Finally, the computing units corresponding to the generation subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module are selected to perform accelerated calculations of each generation subtask.

[0115] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory;

[0116] The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in the first aspect above.

[0117] In a fifth aspect, the present invention provides a storage medium,

[0118] The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0119] The beneficial effects of the present invention are:

[0120] The multimodal large model edge accelerator system and its implementation method, device, equipment, and medium provided by the present invention can accelerate the multimodal large model when processing multiple data types, realize efficient calculation of the multimodal large model in multiple application scenarios, flexibly configure the CPU, GPU, and reconfigurable accelerator according to the computing power requirements, and can realize low-latency calculation of large models under limited computing power, and effectively reduce the power consumption of edge heterogeneous accelerators.

[0121] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect.

[0122] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0124] Figure 1 It is a schematic diagram of the multi-modal large-model edge accelerator system of the present invention.

[0125] Figure 2 It is a schematic diagram of the multimodal large model of the present invention.

[0126] Figure 3 It is a schematic diagram of a reconfigurable edge computing accelerator of the multimodal large model edge accelerator system of the present invention.

[0127] Figure 4 It is a flow chart of the implementation method of the multi-modal large-model edge accelerator system of the present invention.

[0128] Figure 5 It is a schematic diagram of an implementation device of the multi-modal large-model edge accelerator system of the present invention. DETAILED DESCRIPTION

[0129] The implementation method of the multimodal large-model edge accelerator system can be applied to one or more devices, wherein the device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0130] An electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), an interactive network television (IPTV), etc. The network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0131] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0132] See also Figure 1The multimodal large model edge accelerator system shown includes a multimodal large model and an acceleration bus;

[0133] The acceleration bus is equipped with a reconfigurable edge computing accelerator, CPU, GPU, and memory module;

[0134] Reconfigurable edge computing accelerator to accelerate the data processing of multi-modal large models;

[0135] The CPU performs overall control of the accelerator system and accelerates the calculation of multi-modal large models as needed;

[0136] GPU, which accelerates the image processing tasks of the accelerator system and the calculation of multimodal large models as needed;

[0137] The memory module stores the input data, model parameters and output data of the multimodal large model through the storage chip;

[0138] Also includes extended function modules;

[0139] The extended function modules include clock module and power module;

[0140] A clock module is used to synchronize the data acceleration computing process of the reconfigurable edge computing accelerator, CPU, GPU and memory module on the acceleration bus;

[0141] The power module is used to power the data acceleration computing process of the reconfigurable edge computing accelerator, CPU, GPU and memory module on the acceleration bus.

[0142] It should be noted that the CPU may be an x86 CPU, an Arm CPU or a RISC-V CPU, and the memory chip may be a ROM, SRAM, DDR SDRAM, FLASH, HBM or other chip.

[0143] In this embodiment, Figure 2 As shown, the multimodal large model includes an encoder module, a large model module and a generator module;

[0144] The encoder module uses a neural network model to encode each data type and then provides it to the large model module;

[0145] The large model module uses a neural network model to encode various types of data and then processes them uniformly and provides them to the generator module;

[0146] The generator module uses a neural network model to reconstruct and generate the uniformly processed data according to the corresponding original data type;

[0147] The encoder module includes a text encoder, a speech encoder, an image encoder, and a video encoder;

[0148] Each encoder encodes text type, voice type, image type and video type data according to the data type;

[0149] The generator module includes a text generator, a speech generator, an image generator, and a video generator. The data processed by the large model module are reconstructed according to the original data type to generate text, speech, image, and video.

[0150] It should be noted that multimodal large models need to process multiple types of data, such as text, speech, images, and videos. These data have different characteristics, and the preprocessing methods are not exactly the same, and the corresponding neural network models are not exactly the same. For example, text consists of words, phrases, or symbols, and these elements are arranged according to specific grammatical and semantic rules to form sentences and paragraphs. The preprocessing of text data usually involves word segmentation, part-of-speech tagging, data cleaning, text vectorization, etc., which converts the original text into a format that can be processed by the large model. The Transformer-based neural network model has become the mainstream model for text processing due to its advantages in processing long-distance dependencies and capturing deep semantic information. Unlike text data, image data contains rich visual information and has high dimensionality. The preprocessing of image data includes scaling, normalization, feature extraction, data enhancement, etc. The neural networks used for image processing usually include convolutional neural networks (such as ResNet), VisionTransformer, etc.

[0151] like Figure 3 As shown, the reconfigurable edge computing accelerator includes a dedicated computing module, a reconfigurable computing module, and a shared computing module;

[0152] Dedicated computing modules and reconfigurable computing modules accelerate the computation of different encoders of the encoder module and different generators of the generator module, which process different parts of the multimodal large model;

[0153] Shared computing modules accelerate the computing of large model modules that process the same parts as multi-modal large models;

[0154] Specialized computing modules include:

[0155] The text computing unit preprocesses text data and performs model calculations through a text processing neural network model to accelerate the text encoder and text generator;

[0156] The speech computing unit pre-processes speech data and performs model calculations through a speech processing neural network model to accelerate the speech encoder and speech generator;

[0157] Image computing unit, which pre-processes image data and performs model calculations through an image processing neural network model to accelerate the image encoder and image generator;

[0158] The video computing unit pre-processes the video data and performs model calculations through the video processing neural network model to accelerate the video encoder and video generator;

[0159] The reconfigurable computing module includes a plurality of reconfigurable computing units;

[0160] Each reconfigurable computing unit performs collaborative computing with the dedicated computing module to supplement computing power, and flexibly configures computing power between the reconfigurable computing module and the dedicated computing module and between each reconfigurable computing unit according to computing power requirements;

[0161] Each reconfigurable computing unit includes a first type of reconfigurable computing unit and a second type of reconfigurable computing unit;

[0162] The first type of reconfigurable computing unit is used for data preprocessing;

[0163] The second type of reconfigurable computing unit is used to perform model calculations;

[0164] The shared computing modules include:

[0165] A linear computing unit for accelerating the linear part of a neural network model as a large model;

[0166] A nonlinear computing unit is used to accelerate the calculation of the nonlinear part of the neural network model as a large model.

[0167] It should be noted that the dedicated computing module and the reconfigurable computing module are used for accelerating the computing of different parts of various types of data processing, and the shared computing module is used for accelerating the computing of the same part of various types of data processing; specifically, for Figure 2 The multimodal large model includes encoders, generators, and large models of various data types such as text, speech, image, and video; the encoders, generators, and large models are all neural network models; the reconfigurable edge computing accelerator can Figure 2 For example, text data and image data are processed simultaneously, and the encoders and generators of text and images belong to different parts, and the calculation is accelerated through dedicated computing modules and reconfigurable computing modules; as for large models, they belong to the same part, and the calculation is accelerated through shared computing modules.

[0168] Each computing unit of the dedicated computing module of the reconfigurable edge computing accelerator performs preprocessing and generation calculations on the data. For example, the reconfigurable edge computing accelerator needs to process text data and image data at the same time. Since the preprocessing of text data and image data is not exactly the same, for example, word segmentation of text is achieved through text data preprocessing of the text computing unit, and image normalization is achieved through image data preprocessing of the image computing unit. Moreover, the neural network models used by the encoders and generators of text data and image data are not exactly the same. For example, the text encoder is implemented through the text model calculation of the text computing unit, and the image encoder is implemented through the image model calculation of the image computing unit.

[0169] The complexity of text, voice, image, and video data varies, and the computing power required for calculation is different. In multimodal large model calculations, sometimes multiple types of data are processed in parallel, which requires reasonable computing power allocation. Figure 3 The reconfigurable computing modules work together with the dedicated computing modules to supplement computing power and can be flexibly configured according to computing power requirements.

[0170] For example, the reconfigurable edge computing accelerator performs text data preprocessing. If the amount of text data is small and the computing power of the dedicated computing module meets the task requirements, the text data preprocessing of the dedicated computing module is used first. If the amount of text data is large and the computing power of the dedicated computing module is insufficient, the first reconfigurable computing unit of the reconfigurable computing module is used for collaborative computing.

[0171] For example, the reconfigurable edge computing accelerator performs text data preprocessing and image data preprocessing at the same time. If the amount of text data is small, the computing power of the dedicated computing module meets the task requirements, while the amount of image data is large and the computing power of the dedicated computing module is insufficient, then the first reconfigurable computing unit of the reconfigurable computing module only performs image preprocessing collaborative computing. If the amount of text data and image data are both large and the computing power of the dedicated computing module is insufficient, then the first reconfigurable computing unit of the reconfigurable computing module performs text and image preprocessing collaborative computing at the same time, and the number of first reconfigurable computing units used for text and image tasks is proportional to the task computing power requirements.

[0172] The reconfigurable computing module includes a plurality of reconfigurable computing units with different structures, which are suitable for different accelerated computing tasks. The reconfigurable computing module includes a plurality of first reconfigurable computing units and a plurality of second reconfigurable computing units, wherein the first reconfigurable computing units are suitable for data preprocessing and the second reconfigurable computing units are suitable for model calculation.

[0173] When multiple types of data calculations need to be performed simultaneously, such as text and video data preprocessing, M first reconfigurable computing units are allocated for text data preprocessing, and N second reconfigurable computing units are allocated for video data preprocessing, where M <N。

[0174] In addition, in multimodal large model calculations, it is sometimes necessary to simultaneously process multiple single data type tasks, such as multiple text tasks or multiple multi-data type tasks, such as multiple text-image tasks, and to allocate reconfigurable computing units according to the different computing power required by the multiple tasks. Specifically, for example, if two text tasks need to be processed simultaneously, where the length of the first text is greater than that of the second text, then the first text task is allocated X second reconfigurable computing units, and the second text task is allocated Y second reconfigurable computing units, where X>Y.

[0175] The shared computing module is used to accelerate the calculation of the same part of multiple types of data processing in a multi-modal large model, including linear computing units and nonlinear computing units. The linear computing unit is used to accelerate the calculation of the linear part of the neural network model, such as matrix calculation, and the nonlinear unit is used to accelerate the calculation of the nonlinear part of the neural network model, such as nonlinear functions. Large models are accelerated through shared computing modules.

[0176] Large models are generally based on the Transformer structure, which includes self-attention layers, fully connected layers, layer normalization, and residual connections. The multi-head self-attention layer consists of multiple self-attention layers. In each self-attention layer, the input (X) is mapped into three matrices: query (Q), key (K), and value (V), Q = XW Q , K = XW K , V = XW V , where W is the weight. The output of the self-attention layer is Where d is the vector dimension corresponding to the key, and Softmax is a nonlinear function.

[0177] The fully connected layer is a two-layer fully connected neural network with a ReLU function, FFN(x) = ReLU(xW 1 +b 1 )W 2 +b 2 , where FFN is a fully connected layer, x is input, ReLU is a nonlinear function, W is the weight of the neural network, and b is the bias of the neural network. The Q, K, V of the self-attention layer and the W of the fully connected layer are mainly matrix calculations, which are implemented by the linear calculation unit of the shared calculation module; Softmax and ReLU are nonlinear functions, which are implemented by the nonlinear calculation unit of the shared calculation module.

[0178] Based on the above system embodiment, please refer to Figure 4The figure shows a method for implementing a multi-modal large model edge accelerator system in a specific implementation, comprising the following steps:

[0179] S1. Determine the type of computing task to be processed based on the application scenario of the multimodal large model;

[0180] S2. Analyze the computing power requirements and priorities of each computing task, and allocate computing power to the reconfigurable edge accelerator, CPU, and GPU based on the analysis results;

[0181] S3. Configure the dedicated computing module, reconfigurable computing module and shared computing module of the reconfigurable edge accelerator according to the task type of each computing task;

[0182] S4. Use the configured accelerator system to accelerate the data processing process of the multimodal large model.

[0183] In this embodiment, the specific steps of step S1 are as follows:

[0184] S11. Obtain application scenarios of multimodal large models;

[0185] When it is a visual question and answer application scenario, go to step S12;

[0186] When it is a speech recognition application scenario, go to step S13;

[0187] When it is a sentiment analysis application scenario, go to step S14;

[0188] When it is a custom application scenario, go to step S15;

[0189] S12. Determine that the multimodal large model needs to process text and image data type computing tasks, and proceed to step S2;

[0190] S13. Determine that the multimodal large model needs to process text and voice data type computing tasks, and proceed to step S2;

[0191] S14. Determine whether the multimodal large model needs to process text, voice, image and video data type computing tasks, and proceed to step S2;

[0192] S15. Determine the data type calculation task predefined in the custom application scenario as the data type calculation task that the multimodal large model needs to process;

[0193] The specific steps of step S2 are as follows:

[0194] S21. Obtain the total computing power requirement of the computing task;

[0195] S22. Determine whether the total computing power requirement is less than the first computing power threshold;

[0196] If yes, go to step S23;

[0197] If not, proceed to step S24;

[0198] S23. The reconfigurable edge computing accelerator of the acceleration system works, while the CPU and GPU do not work;

[0199] S24. The reconfigurable edge computing accelerator of the acceleration system starts the CPU and GPU for collaborative computing according to the number, type, priority, response speed requirements and power consumption requirements of specific computing tasks;

[0200] The specific steps of step S3 are as follows:

[0201] S31. Get each computing task type;

[0202] When it is a single data type computing task, go to step S32;

[0203] When it is a multi-data type computing task, go to step S35;

[0204] S32. Split the single data type task into encoding subtasks, large model calculation subtasks, and generation subtasks to determine whether the dedicated computing module meets the computing power requirements of the single data type computing task;

[0205] If yes, go to step S33;

[0206] If not, proceed to step S34;

[0207] S33. Select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and generation subtask of each single data type computing task, select the shared computing module in the reconfigurable edge computing accelerator to accelerate the large model computing subtask, and enter step S4;

[0208] S34. Select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and the generation subtask of the single data type computing task, and configure the reconfigurable computing unit of the reconfigurable computing module to accelerate the computing of the corresponding task type, select the shared computing module in the reconfigurable edge computing accelerator to accelerate the computing of the large model computing subtask, and enter step S4;

[0209] S35. Split each multi-data type computing task into an encoding subtask, a large model computing subtask, and a generation subtask;

[0210] S36. Determine whether the dedicated computing module meets the computing power requirements of each computing task;

[0211] If yes, go to step S37;

[0212] If not, proceed to step S38;

[0213] S37. Select the computing unit corresponding to the encoding subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform the accelerated computing of each encoding subtask and then provide it to the shared computing module. Then, the large model computing subtask of each computing task is uniformly used in the shared computing module to perform linear and nonlinear calculations and provide them to the dedicated computing module to complete the accelerated calculation. Finally, select the computing unit corresponding to the generation subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform the accelerated calculation of each generation subtask, and enter step S4.

[0214] S38. Select the computing units corresponding to the encoding subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module to perform accelerated calculations of each encoding subtask and provide them to the shared computing module. Then, use the shared computing module to uniformly perform linear and nonlinear calculations on the large model computing subtasks of each computing task and provide them to the dedicated computing module to complete the accelerated calculation. Finally, select the computing units corresponding to the generation subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module to perform accelerated calculations of each generation subtask and enter step S4.

[0215] In some embodiments, step S24 includes the following steps:

[0216] S241. The reconfigurable edge computing accelerator of the acceleration system is started;

[0217] S242. Obtain the total computing power requirements of the computing tasks, the number of computing tasks, the response speed requirements, and the power consumption requirements;

[0218] When the total computing power requirement is greater than or equal to the first computing power threshold and less than the second computing power threshold, the number of computing tasks is greater than the number threshold, and the response speed requirement is higher than the speed threshold, proceed to step S243;

[0219] When the total computing power requirement is greater than or equal to the second computing power threshold, the number of computing tasks is greater than the number threshold, the response speed requirement is higher than the speed threshold, and the power consumption requirement is lower than the power consumption threshold, proceed to step S244;

[0220] S243. Start the GPU to perform auxiliary calculations, and proceed to step S245;

[0221] S244. Start the GPU and CPU simultaneously for auxiliary calculation;

[0222] S245. Obtain the priority and computing power requirements of each computing task, and allocate computing resources of the reconfigurable edge accelerator, GPU and CPU to each task according to the priority and computing power requirements;

[0223] The specific steps of step S245 are as follows:

[0224] S2451. Identify the first computing task with high priority and small computing power requirement among the computing tasks, and assign a reconfigurable edge computing accelerator to the first computing task for collaborative computing;

[0225] S2452. Identify a second computing task with a low priority and a large computing power requirement among the computing tasks, allocate a GPU for collaborative computing to the second computing task, and allocate a CPU for collaborative computing when the CPU is started.

[0226] In some embodiments, step S34 includes the following steps:

[0227] S341. Select a reconfigurable computing unit in the reconfigurable edge computing accelerator that matches the single data type;

[0228] S342. Obtain and compare the computing power requirements of each computing task of a single data type;

[0229] S343. Assign N1 selected reconfigurable computing units to computing tasks with large computing power requirements, and assign N2 selected reconfigurable computing units to computing tasks with small computing power requirements, and N1>N2;

[0230] In some embodiments, step S38 includes the following steps:

[0231] S381. Determine the adaptation type of each reconfigurable computing unit of the reconfigurable computing module in the reconfigurable edge computing accelerator, and determine the first reconfigurable computing unit adapted for the encoding task and the second reconfigurable computing unit adapted for the data generation task;

[0232] S382. Determine the computing power requirements of each computing task of multiple data types;

[0233] S383. Assign N3 first reconfigurable computing units to the encoding subtask of the computing task of the data type with large computing power requirements, and assign N4 first reconfigurable computing units to the encoding subtask of the computing task of the data type with small computing power requirements, and N3>N4;

[0234] S384. Assign N5 second reconfigurable computing units to the data generation subtask of the computing task of the data type with high computing power requirements, and assign N6 second reconfigurable computing units to the encoding subtask of the computing task of the data type with low computing power requirements, and N5>N6.

[0235] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0236] The following is an embodiment of an implementation device of a multimodal large-scale edge accelerator system provided in an embodiment of the present disclosure. The device and the implementation method of the multimodal large-scale edge accelerator system in the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the implementation device of the multimodal large-scale edge accelerator system, reference can be made to the embodiment of the implementation method of the multimodal large-scale edge accelerator system.

[0237] like Figure 5 As shown, the device comprises:

[0238] A computing task type determination module is used to determine the type of computing task that needs to be processed according to the application scenario of the multimodal large model;

[0239] The accelerator system computing power allocation module is used to analyze the computing power requirements and priorities of each computing task, and allocate the computing power of the reconfigurable edge accelerator, CPU, and GPU based on the analysis results;

[0240] A computing power allocation module for each module of the reconfigurable edge accelerator, used to configure the dedicated computing module, the reconfigurable computing module and the shared computing module of the reconfigurable edge accelerator according to the task type of each computing task;

[0241] The computing acceleration module is used to accelerate the data processing process of the multimodal large model using the configured accelerator system.

[0242] In this embodiment, the computing task type determination module includes:

[0243] An application scenario acquisition unit, used to acquire application scenarios of a multimodal large model;

[0244] The first computing task determination unit is used in the visual question-answering application scenario to determine whether the multimodal large model needs to process computing tasks of text and image data types;

[0245] The second computing task determination unit is used in speech recognition application scenarios to determine whether the multimodal large model needs to process text and speech data type computing tasks;

[0246] The third computing task determination unit is used in sentiment analysis application scenarios to determine whether the multimodal large model needs to process computing tasks of text, speech, image and video data types;

[0247] A fourth computing task determination unit, used to determine that a data type computing task predefined in a custom application scenario is a data type computing task that needs to be processed by the multimodal large model;

[0248] The accelerator system computing power allocation module includes:

[0249] A total computing power requirement calculation unit is used to obtain the total computing power requirement of a computing task;

[0250] A first total computing power requirement determination unit, configured to determine whether the total computing power requirement is less than a first computing power threshold;

[0251] A first computing power configuration unit is used to configure the reconfigurable edge computing accelerator of the acceleration system to work, and the CPU and GPU to not work when the total computing power demand is less than the first computing power threshold;

[0252] The second computing power configuration unit is used to configure the reconfigurable edge computing accelerator of the acceleration system to work when the total computing power demand is greater than or equal to the first computing power threshold, and start the CPU and GPU for collaborative computing according to the specific number, type, priority, response speed requirements and power consumption requirements of computing tasks;

[0253] The computing power allocation modules of each module of the reconfigurable edge accelerator include:

[0254] A computing task type acquisition unit, used to acquire each computing task type;

[0255] The computing power requirement judgment unit for a single data type task is used to split the single data type task into an encoding subtask, a large model calculation subtask, and a generation subtask to judge whether the dedicated computing module meets the computing power requirement of the single data type calculation task;

[0256] The dedicated and shared computing module configuration unit for single data type tasks is used to select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to perform accelerated computing for the encoding subtask and generation subtask of each single data type computing task when the dedicated computing module meets the computing power requirements of the single data type computing task, and select the shared computing module in the reconfigurable edge computing accelerator to perform accelerated computing for the large model computing subtask;

[0257] The dedicated, shared and reconfigurable computing module configuration unit for single data type tasks is used to select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and generation subtask of the single data type computing task when the dedicated computing module does not meet the computing power requirements of the single data type computing task, and configure the reconfigurable computing unit of the reconfigurable computing module to accelerate the corresponding task type, and select the shared computing module in the reconfigurable edge computing accelerator to accelerate the large model computing subtask;

[0258] The computing task splitting unit, when used for multi-data type tasks, splits each multi-data type computing task into encoding subtasks, large model computing subtasks, and generation subtasks;

[0259] The computing power requirement judgment unit for most types of tasks is used to judge whether the dedicated computing module meets the computing power requirements of each computing task;

[0260] The dedicated and shared computing module configuration unit for multi-data type tasks is used for the dedicated computing module to meet the computing power requirements of each computing task, select the computing unit corresponding to the encoding subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform accelerated calculations of each encoding subtask and then provide it to the shared computing module, then use the shared computing module to uniformly perform linear and nonlinear calculations on the large model computing subtasks of each computing task and provide them to the dedicated computing module to complete the accelerated calculation, and finally select the computing unit corresponding to the generation subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform accelerated calculations of each generation subtask;

[0261] The dedicated, shared and reconfigurable computing module configuration units for tasks of multiple data types are used when the dedicated computing modules do not meet the computing power requirements of each computing task. The computing units corresponding to the encoding subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module are selected to perform accelerated calculations of each encoding subtask and then provided to the shared computing module. The large model computing subtasks of each computing task are uniformly calculated using the shared computing module for linear and nonlinear calculations and provided to the dedicated computing module to complete the accelerated calculation. Finally, the computing units corresponding to the generation subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module are selected to perform accelerated calculations of each generation subtask.

[0262] The multimodal large-model edge accelerator system and its implementation method provided by the present invention are modules and algorithm steps of each example described in the embodiments disclosed herein, and can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0263] Those skilled in the art will appreciate that the multimodal large-model edge accelerator system and its implementation method provided by the present invention can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which can be collectively referred to as "circuit", "module" or "system" here.

[0264] In the non-transitory computer-readable storage medium storing the multi-modal large model edge accelerator system and its implementation method, it can be but not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0265] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A multi-modal large-model edge accelerator system, characterized in that: Includes multimodal large models and accelerated buses; The acceleration bus is equipped with a reconfigurable edge computing accelerator, CPU, GPU, and memory module; Reconfigurable edge computing accelerator to accelerate the data processing of multi-modal large models; The CPU performs overall control of the accelerator system and accelerates the calculation of multi-modal large models as needed; GPU, which accelerates the image processing tasks of the accelerator system and the calculation of multimodal large models as needed; The memory module saves the input data, model parameters and output data of the multimodal large model through the storage chip.

2. The multi-modal large model edge accelerator system according to claim 1, characterized in that: The multimodal large model includes an encoder module, a large model module, and a generator module; The encoder module encodes each data type and provides it to the large model module; The large model module encodes various types of data and processes them uniformly before providing them to the generator module; The generator module reconstructs and generates the uniformly processed data according to the corresponding original data type.

3. The multi-modal large model edge accelerator system according to claim 2, characterized in that: The encoder module includes a text encoder, a speech encoder, an image encoder, and a video encoder; Each encoder encodes text type, voice type, image type and video type data according to the data type; The generator module includes a text generator, a speech generator, an image generator and a video generator. The data processed by the large model module are uniformly reconstructed according to the original data type to generate text, speech, image and video.

4. The multi-modal large model edge accelerator system according to claim 3, characterized in that: The reconfigurable edge computing accelerator includes a dedicated computing module, a reconfigurable computing module, and a shared computing module; Dedicated computing modules and reconfigurable computing modules are used to accelerate the computing of different encoders of the encoder module and different generators of the generator module; Shared computing modules to accelerate computing of large model modules.

5. The multi-modal large model edge accelerator system according to claim 4, characterized in that: Specialized computing modules include: The text computing unit preprocesses text data and performs model calculations through a text processing neural network model to accelerate the text encoder and text generator; The speech computing unit pre-processes speech data and performs model calculations through a speech processing neural network model to accelerate the speech encoder and speech generator; Image computing unit, which pre-processes image data and performs model calculations through an image processing neural network model to accelerate the image encoder and image generator; The video computing unit preprocesses the video data and performs model calculations through the video processing neural network model to accelerate the video encoder and video generator.

6. The multi-modal large model edge accelerator system according to claim 4, characterized in that: The reconfigurable computing module includes a plurality of reconfigurable computing units; Each reconfigurable computing unit performs collaborative computing with the dedicated computing module to supplement computing power, and flexibly configures computing power between the reconfigurable computing module and the dedicated computing module and between each reconfigurable computing unit according to computing power requirements.

7. The multi-modal large model edge accelerator system according to claim 6, characterized in that: Each reconfigurable computing unit includes a first type of reconfigurable computing unit and a second type of reconfigurable computing unit; The first type of reconfigurable computing unit is used for data preprocessing; The second type of reconfigurable computing unit is used for performing model calculations.

8. The multi-modal large model edge accelerator system according to claim 4, characterized in that: The shared computing modules include: A linear computing unit for accelerating the linear part of a neural network model as a large model; A nonlinear computing unit is used to accelerate the calculation of the nonlinear part of the neural network model as a large model.

9. The multi-modal large model edge accelerator system according to claim 1, characterized in that: Also includes extended function modules; The extended function modules include clock module and power module; A clock module is used to synchronize the data acceleration computing process of the reconfigurable edge computing accelerator, CPU, GPU and memory module on the acceleration bus; The power module is used to power the data acceleration computing process of the reconfigurable edge computing accelerator, CPU, GPU and memory module on the acceleration bus.

10. A method for implementing a multi-modal large model edge accelerator system according to any one of claims 1 to 9, characterized in that: The steps include: S1. Determine the type of computing task to be processed based on the application scenario of the multimodal large model; S2. Analyze the computing power requirements and priorities of each computing task, and allocate computing power to the reconfigurable edge accelerator, CPU, and GPU based on the analysis results; S3. Configure the dedicated computing module, reconfigurable computing module and shared computing module of the reconfigurable edge accelerator according to the task type of each computing task; S4. Use the configured accelerator system to accelerate the data processing process of the multimodal large model.

11. The method for implementing the multi-modal large model edge accelerator system according to claim 10, characterized in that: The specific steps of step S1 are as follows: S11. Obtain application scenarios of multimodal large models; When it is a visual question and answer application scenario, go to step S12; When it is a speech recognition application scenario, go to step S13; When it is a sentiment analysis application scenario, go to step S14; When it is a custom application scenario, go to step S15; S12. Determine that the multimodal large model needs to process text and image data type computing tasks, and proceed to step S2; S13. Determine that the multimodal large model needs to process text and voice data type computing tasks, and proceed to step S2; S14. Determine whether the multimodal large model needs to process text, voice, image and video data type computing tasks, and proceed to step S2; S15. Determine whether the data type calculation task pre-defined in the custom application scenario is the data type calculation task that needs to be processed by the multimodal large model.

12. The method for implementing the multi-modal large model edge accelerator system according to claim 10, characterized in that: The specific steps of step S2 are as follows: S21. Obtain the total computing power requirement of the computing task; S22. Determine whether the total computing power requirement is less than the first computing power threshold; If yes, go to step S23; If not, proceed to step S24; S23. The reconfigurable edge computing accelerator of the acceleration system works, while the CPU and GPU do not work; S24. The reconfigurable edge computing accelerator of the acceleration system works, and the CPU and GPU are started for collaborative computing according to the specific number, type, priority, response speed requirements and power consumption requirements of computing tasks.

13. The method for implementing the multi-modal large model edge accelerator system according to claim 12, characterized in that: The specific steps of step S24 are as follows: S241. The reconfigurable edge computing accelerator of the acceleration system is started; S242. Obtain the total computing power requirement of the computing task, the number of computing tasks, the response speed requirement, and the power consumption requirement; when the total computing power requirement is greater than or equal to the first computing power threshold, less than the second computing power threshold, the number of computing tasks is greater than the number threshold, and the response speed requirement is higher than the speed threshold, proceed to step S243; When the total computing power requirement is greater than or equal to the second computing power threshold, the number of computing tasks is greater than the number threshold, the response speed requirement is higher than the speed threshold, and the power consumption requirement is lower than the power consumption threshold, proceed to step S244; S243. Start the GPU to perform auxiliary calculations, and proceed to step S245; S244. Start the GPU and CPU simultaneously for auxiliary calculation; S245. Obtain the priority and computing power requirements of each computing task, and allocate computing resources of the reconfigurable edge accelerator, GPU, and CPU to each task based on the priority and computing power requirements.

14. The method for implementing the multi-modal large model edge accelerator system according to claim 13, characterized in that: The specific steps of step S245 are as follows: S2451. Identify the first computing task with high priority and small computing power requirement among the computing tasks, and assign a reconfigurable edge computing accelerator to the first computing task for collaborative computing; S2452. Identify a second computing task with a low priority and a large computing power requirement among the computing tasks, allocate a GPU for collaborative computing to the second computing task, and allocate a CPU for collaborative computing when the CPU is started.

15. The method for implementing the multi-modal large model edge accelerator system according to claim 12, characterized in that: The specific steps of step S3 are as follows: S31. Get each computing task type; When it is a single data type computing task, go to step S32; When it is a multi-data type computing task, go to step S35; S32. Split the single data type task into encoding subtasks, large model calculation subtasks, and generation subtasks to determine whether the dedicated computing module meets the computing power requirements of the single data type computing task; If yes, go to step S33; If not, proceed to step S34; S33. Select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and generation subtask of each single data type computing task, select the shared computing module in the reconfigurable edge computing accelerator to accelerate the large model computing subtask, and enter step S4; S34. Select the corresponding computing unit of the dedicated computing module in the reconfigurable edge computing accelerator to accelerate the encoding subtask and the generation subtask of the single data type computing task, and configure the reconfigurable computing unit of the reconfigurable computing module to accelerate the computing of the corresponding task type, select the shared computing module in the reconfigurable edge computing accelerator to accelerate the computing of the large model computing subtask, and enter step S4; S35. Split each multi-data type computing task into an encoding subtask, a large model computing subtask, and a generation subtask; S36. Determine whether the dedicated computing module meets the computing power requirements of each computing task; If yes, go to step S37; If not, proceed to step S38; S37. Select the computing unit corresponding to the encoding subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform the accelerated computing of each encoding subtask and then provide it to the shared computing module. Then, the large model computing subtask of each computing task is uniformly used in the shared computing module to perform linear and nonlinear calculations and provide them to the dedicated computing module to complete the accelerated calculation. Finally, select the computing unit corresponding to the generation subtask of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator to perform the accelerated calculation of each generation subtask, and enter step S4. S38. Select the computing units corresponding to the encoding subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module to perform accelerated calculations of each encoding subtask and provide them to the shared computing module. Then, use the shared computing module to uniformly perform linear and nonlinear calculations on the large model computing subtasks of each computing task and provide them to the dedicated computing module to complete the accelerated calculation. Finally, select the computing units corresponding to the generation subtasks of each computing task in the dedicated computing module of the reconfigurable edge computing accelerator and the reconfigurable computing units configured with the reconfigurable computing module to perform accelerated calculations of each generation subtask and enter step S4.

16. The method for implementing the multi-modal large model edge accelerator system according to claim 15, characterized in that: The specific steps of step S34 are as follows: S341. Select a reconfigurable computing unit in the reconfigurable edge computing accelerator that matches the single data type; S342. Obtain and compare the computing power requirements of each computing task of a single data type; S343. Allocate N1 selected reconfigurable computing units to computing tasks with high computing power requirements, and allocate N2 selected reconfigurable computing units to computing tasks with low computing power requirements, and N1>N2.

17. The method for implementing the multi-modal large model edge accelerator system according to claim 15, characterized in that: The specific steps of step S38 are as follows: S381. Determine the adaptation type of each reconfigurable computing unit of the reconfigurable computing module in the reconfigurable edge computing accelerator, and determine the first reconfigurable computing unit adapted for the encoding task and the second reconfigurable computing unit adapted for the data generation task; S382. Determine the computing power requirements of each computing task of multiple data types; S383. N3 first reconfigurable computing units are allocated to the encoding subtasks of the computing tasks of the data types with large computing power requirements, and N4 first reconfigurable computing units are allocated to the encoding subtasks of the computing tasks of the data types with small computing power requirements, and N3>N4; S384. Assign N5 second reconfigurable computing units to the data generation subtask of the computing task of the data type with high computing power requirements, and assign N6 second reconfigurable computing units to the encoding subtask of the computing task of the data type with low computing power requirements, and N5>N6.

18. A device for implementing a multi-modal large-scale edge accelerator system, characterized in that: include: A computing task type determination module is used to determine the type of computing task that needs to be processed according to the application scenario of the multimodal large model; The accelerator system computing power allocation module is used to analyze the computing power requirements and priorities of each computing task, and allocate the computing power of the reconfigurable edge accelerator, CPU, and GPU based on the analysis results; A computing power allocation module for each module of the reconfigurable edge accelerator, used to configure the dedicated computing module, the reconfigurable computing module and the shared computing module of the reconfigurable edge accelerator according to the task type of each computing task; The computing acceleration module is used to accelerate the data processing process of the multimodal large model using the configured accelerator system.

19. An electronic device, characterized in that: including a processor and a memory; The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in any one of claims 10 to 17.

20. A storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in any one of claims 10 to 17.

Images