Computing system, model processing method, electronic equipment and storage medium

Abstract

The invention discloses a calculation system, a model processing method, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and the system comprises a storage and calculation integrated module and a storage and calculation separation module. Wherein the storage and calculation integrated module comprises a storage and calculation integrated array, and the storage and calculation separation module comprises a storage unit and a calculation unit. And under the condition that the storage and calculation separation module and the storage and calculation integrated module cooperatively execute the calculation task, the calculation unit is configured to call the first parameter of the storage unit and the second parameter of the storage and calculation integrated array so as to complete the calculation operation of the calculation task. According to the technical scheme, the parameters are called in a cross-module mode through the calculation unit, cooperative operation of low-precision and high-precision parameters is achieved, and parameter redundant storage of coupling submodules in a multi-modal model can be reduced. Meanwhile, the double modules cooperate to adapt to calculation requirements of different precision and different modes, the data interaction efficiency and the calculation parallelism degree are improved, and the reasoning precision, the energy consumption and the delay are effectively balanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a computing system and model processing method, an electronic device, and a storage medium. BACKGROUND

[0002] A multi-modal large model extends the generality and reasoning generation capability of a language large model to multiple modalities (such as images, videos, etc.), improves the perception interaction capability of the large model, and provides a more intelligent perception and decision scheme for robot applications. Compared with the language large model, the multi-modal large model adds multiple modal processing and inter-modal fusion links to the calculation, and the computing power requirements of different modal reasoning differ greatly, and the computing mode and resource scheduling are more complex.

[0003] The implementation of the multi-modal large model relies on the cooperation of hardware and software, and the processing is divided into specific stages. On the hardware side, there are mainly two architectures of memory and calculation separation and memory and calculation integration: the memory and calculation separation architecture increases the bandwidth through advanced packaging such as high bandwidth memory (HBM), or shortens the memory and calculation distance through near-memory calculation, improves the data transmission speed, and reduces the energy consumption; the memory and calculation integration architecture completes the calculation in the memory, has no data transfer, and has low delay, but the memory and calculation accuracy and coupling are limited, and only support text single modality. On the software side, the inference acceleration engine reduces the calculation amount through model sparsification and compression, and combines hardware to share the load through data parallelism and pipeline parallelism.

[0004] However, a single hardware architecture cannot balance performance, energy efficiency, and generality, a hybrid architecture lacks a sharing mechanism, resulting in insufficient cooperation and resource redundancy, and the cooperation between hardware and algorithms is insufficient to adapt to the differences between various modalities and handle coupled modules, and finally cannot achieve the best balance between speed, energy consumption, and accuracy of multi-modal large model reasoning, thereby affecting the inference effect of the multi-modal large model. SUMMARY

[0005] Therefore, the present application provides a computing system and model processing method, an electronic device, and a storage medium to improve the inference effect of the multi-modal large model.

[0006] In a first aspect, the present application provides a computing system, comprising: a memory and calculation integrated module comprising a memory and calculation integrated array; a memory and calculation separated module comprising a storage unit and a calculation unit; in a case where the memory and calculation separated module and the memory and calculation integrated module cooperatively perform a calculation task, the calculation unit is configured to call first parameters of the storage unit and second parameters of the memory and calculation integrated array to complete an operation operation of the calculation task.

[0007] The aforementioned computing system, through the collaborative retrieval of parameters from different modules using a dual-architecture approach, balances the high bandwidth and low power consumption of in-memory computing with the high precision and flexibility of in-memory computing separation, thus resolving the contradictions between bandwidth, energy efficiency, and accuracy inherent in single-architecture systems. Simultaneously, it reduces redundant parameter storage in coupled models, adapts to the diverse computational needs of large multimodal models, and improves overall resource utilization and computational efficiency.

[0008] In some embodiments of this application, the in-memory computing array of the in-memory computing module is integrated into the storage unit of the in-memory computing separation module.

[0009] In the aforementioned computing system, the in-memory computing array is integrated into the storage unit of the in-memory computing separate module, which can shorten the data transmission path, reduce cross-module handling latency and energy consumption, and is especially suitable for multi-modal high-frequency parameter calling scenarios. At the same time, it facilitates resource coordination, reduces conflicts, simplifies hardware structure, reduces chip area, and lowers deployment costs.

[0010] In some embodiments of this application, the computing unit includes: an inverse quantization unit configured to perform fusion processing on a first parameter of the storage unit and a second parameter of the in-memory computing array to obtain fused parameters; the computing unit is further configured to execute the computing task according to the fused parameters.

[0011] In the aforementioned computing system, the dequantization unit integrates parameters of different precision, breaking down the collaborative barrier caused by inconsistent parameter precision in the dual-architecture system and ensuring computational continuity. It leverages both in-memory computing with low-precision parameters to reduce energy consumption and in-memory computing with high-precision parameters to supplement precision, adapting to the mixed high- and low-precision requirements in multimodal computing, reducing resource redundancy, and improving collaborative computing efficiency.

[0012] In some embodiments of this application, the precision of the first parameter is higher than that of the second parameter.

[0013] In the aforementioned computing system, the first parameter has higher precision than the second parameter, allowing in-memory computing to focus on low-precision computation to leverage its low power consumption and high bandwidth advantages. Separation of in-memory and computation balances high and low precision requirements to meet the needs of high-precision applications such as image processing. It precisely adapts to the precision differences between different modalities in large multimodal models, enabling the dual architectures to perform their respective functions, balancing energy efficiency and precision, and reducing resource waste.

[0014] In some embodiments of this application, when the storage-computation separation module performs the computation task independently, the computation unit of the storage-computation separation module is further configured to call the first parameter.

[0015] In the aforementioned computing system, when the in-memory computing module is executed independently, it invokes the high-precision first parameter, enabling it to independently handle high-precision modal tasks such as image and video processing, fully leveraging its flexibility and accuracy advantages. It does not rely on an integrated in-memory computing module, avoiding resource coupling, improving the processing efficiency of single-architecture tasks, and adapting to multimodal scenarios requiring high-precision independent processing.

[0016] In some embodiments of this application, when the in-memory computing module performs computing tasks independently, the in-memory computing array is also configured to invoke the second parameter.

[0017] In the aforementioned computing system, when the in-memory compute unit executes independently, it calls the low-precision second parameter, enabling it to independently and efficiently process low-precision tasks such as text processing, maximizing its low-power and high-bandwidth characteristics. This eliminates the need to occupy in-memory compute separation resources, avoiding resource idleness, improving the processing efficiency of single-architecture tasks, and adapting to multimodal scenarios with independent low-precision requirements.

[0018] In some embodiments of this application, the computing unit further includes: a multiply-accumulate processor configured to perform multiply-accumulate calculation tasks of different precisions; and a first special function unit configured to perform non-multiply-accumulate calculation tasks of the storage-computation separation module.

[0019] In the aforementioned computing system, the multiply-accumulate processor and the first special function unit have clearly defined roles: the former efficiently handles high-frequency multiply-accumulate tasks, while the latter focuses on non-multiply-accumulate operators, avoiding resource conflicts. This adapts to the differentiated computing needs of different modes and stages in multimodal large models, improving the computational efficiency and task adaptability of the in-memory / computation separation module.

[0020] In some embodiments of this application, the computing unit further includes a shared weight cache unit; the shared weight cache unit is configured to store weights for executing the computing task; the computing unit is further configured to execute the computational operation of the computing task according to the weights.

[0021] In the aforementioned computing system, a shared weight cache allows multiple multiply-add processors to share weights, reducing redundant reads and thus lowering transmission power consumption and bandwidth requirements. It supports parallel computation with weight reuse, adapting to scenarios where repeated weights are frequently used in large multimodal models, saving storage resources and accelerating computation.

[0022] In some embodiments of this application, the in-memory computing module further includes: a second special function unit configured to perform non-multiply-accumulate computation tasks of the in-memory computing array.

[0023] In the aforementioned computing system, the second special function unit handles in-memory computation non-multiply-accumulate tasks, avoiding the consumption of multiply-accumulate resources and allowing it to focus on core computations. This enhances the independent working capability of in-memory computation, eliminating the need for in-memory computation separation, and enabling it to independently complete low-precision modal full-process computations. It complements the first special function unit, improving the system's adaptability to multimodal non-multiply-accumulate requirements.

[0024] In some embodiments of this application, the in-memory computing module further includes: a control unit, which, when the in-memory computing module performs a computing task independently, schedules corresponding storage resources and computing resources according to the computing task to perform the computational operation of the computing task through the storage resources and computing resources; and a buffer and conflict unit, configured to coordinate resource conflicts in the in-memory computing array when scheduling the storage resources and computing resources.

[0025] In the aforementioned computing system, the control unit flexibly schedules resources and modes to adapt to independent or collaborative scenarios, thereby improving utilization; the buffer and conflict unit coordinates resource conflicts, prioritizes computing, and reduces latency.

[0026] In some embodiments of this application, the computing system further includes a shared lexical and key-value cache unit, configured to store intermediate results generated by the computing unit and the in-memory computing array performing computational operations when the storage-computation separation module and the storage-computation integrated module jointly perform computing tasks.

[0027] In the aforementioned computing system, a shared cache centrally stores intermediate results from the dual-architecture collaboration, reducing redundant storage and saving resources. The dual architectures directly read intermediate results, reducing cross-module transmission latency and energy consumption, and adapting to multimodal, high-frequency intermediate result reuse scenarios. This accelerates collaboration efficiency, ensures data synchronization, and improves the coherence and overall speed of multimodal inference.

[0028] In some embodiments of this application, the computing system further includes: a task allocation unit, configured to parse the computing task, identify the modality type corresponding to the computing task, and dynamically switch the working modes of the in-memory computing module and the in-memory computing separation module according to the modality type; the working modes include a mode of independently executing computing tasks and a mode of collaboratively executing computing tasks.

[0029] In the aforementioned computing system, the task allocation unit parses tasks and dynamically switches between dual-architecture modes, matching the optimal architecture combination for different modalities to maximize the advantages of the dual architecture. During multimodal concurrency, it coordinates parallel processing, reduces inference latency, and adapts to the needs of different stages of multimodal processing without hardware refactoring, thereby improving system flexibility and multimodal inference efficiency.

[0030] Secondly, this application also provides a model processing method applied to the computing system described in the first aspect. The method includes: when a storage-computation separation module and a storage-computation integrated module are jointly executing a computing task, the computing unit of the storage-computation separation module retrieves the first parameter of the storage unit in the storage-computation separation module and the second parameter of the storage-computation integrated array in the storage-computation integrated module to complete the computational operation of the computing task.

[0031] In some embodiments of this application, the computing unit of the storage-computing separation module is further configured to invoke the first parameter when the storage-computing separation module performs the computing task independently.

[0032] In the above method, the storage and computation are separated and the first parameter is called separately without relying on the storage and computation resources, avoiding coordination overhead, improving the processing efficiency and adaptability of single-modal high-precision tasks, and enriching the processing modes of multi-modal tasks.

[0033] In some embodiments of this application, the in-memory computing array is further configured to invoke the second parameter when the in-memory computing module performs the computing task independently.

[0034] In the above method, the in-memory computing calls the second parameter separately without occupying the resources of the in-memory computing separation, reducing resource idleness and coordination delay, improving the processing efficiency of single-modal low-precision tasks, and optimizing the energy consumption of multi-modal tasks.

[0035] Thirdly, this application provides an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the second aspect of the preceding embodiments.

[0036] Fourthly, this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in the second aspect of the preceding embodiment.

[0037] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the second aspect of the embodiments described above.

[0038] As can be seen from the above technical solutions, the computing system, model processing method, electronic device, and storage medium of this application relate to the field of artificial intelligence. The system includes an in-memory computing module and a separate in-memory computing module. The in-memory computing module includes an in-memory computing array, and the separate in-memory computing module includes a storage unit and a computing unit. When the separate in-memory computing module and the in-memory computing module collaboratively execute computing tasks, the computing unit is configured to retrieve a first parameter from the storage unit and a second parameter from the in-memory computing array to complete the computational operations. By applying the technical solution of this application, the computing unit retrieves parameters across modules, enabling collaborative computation of low-precision and high-precision parameters, which can reduce redundant parameter storage in coupled sub-modules in multimodal models. Simultaneously, the dual-module collaboration adapts to different precision and modal computational needs, improving data interaction efficiency and computational parallelism, and effectively balancing inference accuracy, energy consumption, and latency.

[0039] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This paper shows a schematic diagram of the structure of a computing system provided in an embodiment of this application; Figure 2 This illustration shows a hybrid architecture diagram of in-memory computing and in-memory computing separation tightly coupled according to an embodiment of this application; Figure 3 This illustration shows an architecture diagram of tightly coupled in-memory computing and in-memory computing separation in a collaborative working scenario provided by an embodiment of this application; Figure 4 A flowchart illustrating the model processing method provided in an embodiment of this application is shown; Figure 5 This paper illustrates a schematic diagram of the model deployment process provided in an embodiment of this application. Figure 6 This illustration shows a schematic diagram of the multimodal large model structure analysis and dependency relationships provided in an embodiment of this application; Figure 7 A schematic block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0043] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0044] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0045] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. In this disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression. In some embodiments, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc., can be used interchangeably. In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”. The prefixes such as "first" and "second" in the embodiments of this disclosure are only for distinguishing different descriptive objects and do not constitute restrictions on the position, order, priority, number or content of the descriptive objects. For the description of the descriptive objects, please refer to the description in the claims or the context of the embodiments. The use of prefixes should not constitute unnecessary restrictions.

[0046] In the embodiments disclosed herein, "multiple" refers to two or more. In the embodiments disclosed herein, terms such as “import”, “input”, and “read in” can be used interchangeably.

[0047] In some embodiments, devices, etc., can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as “device”, “equipment”, “circuit”, “network element”, “node”, “function”, “unit”, “section”, “system”, “network”, “chip”, “chip system”, “entity”, and “subject” can be used interchangeably.

[0048] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", "subscriber station", "mobile unit", "subscriber unit", "wireless unit", "remote unit", "mobile device", "wireless device", "wireless communication device", "remote device", "mobile subscriber station", "access terminal", "mobile terminal", "wireless terminal", "remote terminal", "handset", "useragent", "mobile client", and "client" can be used interchangeably.

[0049] With the rapid development of large-scale model technology, multimodal large-scale models extend the versatility and reasoning generation capabilities of language large-scale models to multiple modalities, enhancing the perception and interaction capabilities of large-scale models and providing more intelligent perception and decision-making methods for applications such as robotics. The computation of multimodal large-scale models adds multiple modal processing and intermodal fusion to the computation of language large-scale models, and the computational requirements for intermodal reasoning differ significantly, making the computational model and resource scheduling more complex. In particular, coupled models exist in multimodal large-scale model reasoning, meaning that different modal reasoning and the same modality are processed at different stages using the same model, but their computational characteristics differ significantly, leading to conflicts between acceleration and deployment methods.

[0050] In related technologies, accelerating inference for large multimodal models can be achieved through hardware acceleration and software optimization. Hardware acceleration can increase bandwidth through packaging technologies such as High Bandwidth Memory (HBM), or shorten the memory-computation distance using near-memory computing architectures to improve data transmission speed and reduce power consumption. In-memory computing architectures complete computation within memory, eliminating data transfer and achieving low latency and high power consumption; however, due to limitations in in-memory accuracy and strong coupling between memory and computation, it currently only supports large single-modal models based on text. Software optimization includes inference acceleration engines for large models that reduce computational load through model sparsity and compression, and distribute the computational load based on hardware characteristics using data parallelism and pipelined parallelism strategies.

[0051] However, for multimodal large model inference, the above approach has the following problems: single hardware architecture is difficult to balance performance, energy efficiency and versatility; hybrid architecture lacks a sharing mechanism, resulting in insufficient coordination and resource redundancy; and hardware and algorithm coordination is insufficient.

[0052] Among these challenges, single-hardware architectures struggle to balance performance, energy efficiency, and versatility. Compute-in-memory (CIM) architectures suffer from bandwidth constraints and power consumption issues. These problems are exacerbated by the large number of parameters in multimodal, large-scale models and the differentiated processing required for various modalities. This is particularly true for single-user request scenarios and edge-side inference scenarios, where single-batch inference is more severely affected by bandwidth, while edge devices are more sensitive to power consumption and heat dissipation. Compute-in-memory (CIM) architectures face limitations in computing power and accuracy. Because parameters are fixed in the computation array, computing resources cannot be reused across layers. In single-batch scenarios, the computing resources used by each layer in a multi-layer model are interdependent, leading to a high overall idle rate. Furthermore, CIM architectures suffer from computational errors due to low-precision computation and simulation calculations, making them unsuitable for high-precision modal processing such as image processing.

[0053] The lack of sharing mechanisms in hybrid architectures leads to insufficient collaboration and resource redundancy. Some new architectures adopt a hybrid approach that integrates multiple architectures to improve overall performance. However, different architectures operate independently within the system, meaning computation and storage are not shared. This is especially true for in-memory compute hybrid architectures, where the low precision of the in-memory compute array necessitates high-precision computations from other architectures, which cannot directly utilize the low-precision parameters of the in-memory compute array. Furthermore, the coupling of models in multimodal processing prevents model decomposition, requiring different architectures to deploy identical sub-models, resulting in storage redundancy. The large number of parameters in multimodal large models exacerbates the problems of redundant storage area and cost.

[0054] As shown in the table below, due to insufficient hardware and algorithm collaboration, multimodal data processing differs significantly from single-text data processing in terms of quantization accuracy, modal structure, and parallelism. Therefore, different architectures and acceleration methods should be adopted for optimization and acceleration of each modality to achieve the best results. Single architectures and acceleration methods oriented towards single modalities cannot take into account the characteristics of all modalities, especially the coupled modules in large multimodal models, and cannot achieve the best balance of speed, energy consumption, and accuracy by integrating all modalities.

[0055]

[0056] To address the aforementioned issues, some embodiments of this application provide a computing system that is a hybrid architecture that tightly couples in-memory computing and in-memory computing separation. This system leverages the high bandwidth and low power consumption advantages of the in-memory computing architecture and the flexibility and high precision advantages of the in-memory computing separation architecture to resolve the contradictions between bandwidth, energy efficiency, flexibility, and precision. Simultaneously, it enhances the synergy between the in-memory computing and in-memory computing separation architectures and reduces resource redundancy caused by the coupled model.

[0057] Figure 1 This is a schematic diagram of the structure of a computing system provided in an embodiment of this application. Figure 1 As shown, in some embodiments, the computing system 100 includes a memory-computing module 101 and a memory-computing separation module 102. The memory-computing module 101 includes a memory-computing array 1011; the memory-computing separation module 102 includes a storage unit 1021 and a computing unit 1022.

[0058] When the storage-computing separation module 102 and the storage-computing integrated module 101 work together to perform computing tasks, the computing unit 1022 of the storage-computing separation module 102 is configured to retrieve the first parameter of the storage unit 1021 and the second parameter of the storage-computing integrated array 1011 to complete the computational operation of the computing task.

[0059] In other words, the computing system is a hybrid architecture with tight coupling between in-memory computing (in-memory computing module 101) and in-memory computing separation (in-memory computing separation module 102). The in-memory computing separation module 102 can access the storage resources of the in-memory computing module 102. When the in-memory computing module 101 and the in-memory computing separation module 102 work together, the in-memory computing separation module 102 can obtain parameters from both its own storage unit 1021 and the storage resources of the in-memory computing module 101. For example, low-precision parameters are taken from the storage resources of the in-memory computing module 101, while high-precision parameters are taken from the storage unit 1021 of the in-memory computing separation module 102.

[0060] In one implementation, both the in-memory computing array 1011 and the storage unit 1021 are non-volatile, fast-access storage devices, such as memristors.

[0061] This embodiment of the invention discloses a computing system that tightly couples in-memory computing and in-memory computing separation. This system can collaboratively retrieve parameters from the in-memory computing module 101 (high bandwidth, low power consumption) and the in-memory computing separation module 102 (high precision, high flexibility), resolving the performance, energy efficiency, and accuracy contradictions inherent in single-architecture systems. This reduces data transmission energy consumption and inference latency while meeting the requirements of high-precision modalities such as image processing. It also reduces redundant storage in coupled models, adapts to multimodal large-model inference, effectively improves resource utilization efficiency, and controls hardware costs.

[0062] For example, such as Figure 2 As shown, the computing-in-memory (CIM) array of the in-memory computing module 101 is used to share low-precision weights, and the storage unit of the storage-computing separation module 102 is a storage-class memory (SCM) array, which is used to store high-precision weights.

[0063] like Figure 2 As shown, in some embodiments, the in-memory computing array 1011 of the in-memory computing module 101 is integrated into the storage unit 1021 of the in-memory computing separation module 102. For example... Figure 2 As shown, in some examples, the storage unit may integrate a storage-class memory array of storage separation module 101 and a storage-computing integrated array of storage module 102.

[0064] In this embodiment of the disclosure, the in-memory computing array 1011 is integrated into the storage unit 1021 of the in-memory computing separation module 102, which can shorten the data transmission path, reduce latency, reduce energy consumption, and improve energy efficiency; resources within the same unit are easily coordinated, which can quickly respond to tasks and avoid conflicts; it can also simplify the hardware structure, reduce the chip area, reduce costs, and is more suitable for edge devices such as service robots and smart cameras.

[0065] In some embodiments, the precision of the first parameter is higher than that of the second parameter. That is, the in-memory computing module 101 can perform low-precision calculations, and the in-memory computing separate module 102 can perform both low-precision and high-precision calculations.

[0066] The dual architecture design of the aforementioned in-memory computing module 101 and in-memory computing separate module 102 can support collaborative work or independent work.

[0067] In collaborative operation, i.e., when the in-memory computing separation module 102 and the in-memory computing integrated module 101 work together to perform computing tasks, the computing unit 1022 is configured to retrieve the first parameter of the storage unit 1021 and the second parameter of the in-memory computing integrated array 1011 to complete the computational operation. Since the first and second parameters have different precisions, the computing unit 1022 needs to perform fusion processing on the two sets of parameters after retrieving them.

[0068] For the fusion processing of the first parameter and the second parameter, in some embodiments, such as Figure 2 As shown, the computation unit 1022 also includes a dequantization unit 10221. The dequantization unit 10221 is configured to perform a fusion process on the first parameter of the storage unit 1021 and the second parameter of the in-memory computing array 1011 to obtain fused parameters. In other words, the dequantization unit 10221 restores these two parameters to high-precision weights. Correspondingly, the computation unit 1022 is configured to perform computation tasks based on the fused parameters.

[0069] In this embodiment of the present disclosure, the inverse quantization unit 10221 in the computing unit 1022 can merge the high-precision first parameter of the storage-computation separation module 102 and the low-precision second parameter of the storage-computation integrated module 101 into a unified high-precision weight. This not only breaks the collaborative computing barrier caused by the inconsistency of parameter precision between the two architectures and ensures the continuity of the operation logic, but also reduces data transmission and storage energy consumption by relying on the low-precision parameters of the storage-computation integrated module. At the same time, it uses the high-precision parameters of the storage-computation separation to supplement the precision to meet the high-precision inference requirements of multimodal such as images and videos. Furthermore, it eliminates the need to deploy parameters separately for different precisions, reduces resource redundancy, and effectively improves the collaborative computing efficiency of the two architectures.

[0070] In some embodiments, when the storage-computation separation module 102 and the storage-computation integration module 101 perform computing tasks independently, both the storage-computation separation module 102 and the storage-computation integration module 101 call parameters within their own storage resources.

[0071] In some embodiments, when the in-memory computing module 101 performs computing tasks independently, the in-memory computing array 1011 is also configured to call its own second parameter.

[0072] In some embodiments, when the storage-computation separation module 102 performs the computation task alone, the computation unit 1022 of the storage-computation separation module 102 is also configured to call its own first parameter.

[0073] In this embodiment, when the dual architectures operate independently, each calls its own storage resource parameters. The in-memory computing module 101 can independently and efficiently process low-precision tasks such as text using its own low-precision parameters, maximizing low power consumption and high bandwidth characteristics. The in-memory computing separation module 102 can independently call its own high-precision parameters to handle high-precision tasks such as image processing, highlighting its flexibility and accuracy advantages. At the same time, the two are independent of each other and can process different tasks in parallel or independently undertake single-modal tasks adapted to themselves, avoiding resource contention and idleness, and improving the system's task adaptation flexibility and overall resource utilization.

[0074] For example, such as Figure 2 As shown, the storage unit 1021 includes a memory-computing array 1011 based on the memory-computing module 101. The storage unit 1021 and the computing unit 1022 constitute a memory-computing separation architecture (memory-computing separation module). The memory-computing separation architecture can read the storage resources of the memory-computing architecture (memory-computing module). The memory-computing architecture performs low-precision calculations with errors, while the memory-computing separation architecture performs both low-precision and high-precision calculations. The two architectures support independent and collaborative operation.

[0075] When operating independently, the dual architectures use different storage resources: in-memory computing (in-memory computing) uses the parameters within the in-memory computing array, while storage-compute separation uses a dedicated storage array. Figure 2 The dual architectures can process different tasks and computations in parallel using the memory array (within the storage-level memory array). In collaborative operation, the storage-compute separation architecture can access the parameters in the storage-compute integrated architecture. The low-precision part of the parameters in the storage-compute separation computation is taken from the storage-compute integrated array, while the high-precision part of the parameters is taken from the dedicated storage array, thereby achieving partial resource sharing between the two architectures.

[0076] Based on the above computing system, in some embodiments, for coupled model processing in multimodal large models, the common parameters in the model are stored in the in-memory computing module 101 and called by the dual architecture of the in-memory computing module 101 and the in-memory computing separation module 102, thereby reducing resource redundancy.

[0077] For example, the computing systems of the in-memory computing module 101 and the in-memory computing separation module 102 under different configurations can implement 10 working modes, as shown in the table below, to meet the multimodal and multi-stage computing needs of multimodal large models. Modes 1-4 realize architecture collaborative computing for independent models, and modes 5-10 realize dual architecture collaborative computing for coupled models.

[0078]

[0079] In some embodiments, the computing unit 1022 further includes a multiply-accumulate processor 10222 and a first special function unit 10223. The multiply-accumulate processor is configured to perform multiply-accumulate calculation tasks of different precisions; the first special function unit 10223 is configured to perform non-multiply-accumulate calculation tasks of the storage-computation separation module 102.

[0080] For example, such as Figure 2 As shown, the computation unit, also known as the Arithmetic Logic Unit (ALU), can include a Multiply Accumulate Core (MAC Core) and a Special Function Unit (SFU). The MAC Core can be used to perform low-precision and high-precision calculations, while the SFU handles operators other than multiply-accumulate calculations.

[0081] In this embodiment, the multiply-accumulate processor 10222 in the computing unit 1022 focuses on multiply-accumulate calculations of different precisions, which can efficiently handle core tasks such as high-frequency matrix operations in large models; the first special function unit 10223 is dedicated to handling non-multiply-accumulate operators, avoiding the occupation of multiply-accumulate resources. This not only improves the efficiency of multiply-accumulate calculations (adapting to low and high precision requirements), but also efficiently handles diverse non-multiply-accumulate operations, reduces resource conflicts, and flexibly adapts to the differentiated computing needs of different modes and stages in multimodal large models.

[0082] In one implementation, the multiply-accumulate processor 10222 contains a matrix multiply-accumulate unit that supports integer and floating-point calculations, and can perform low-precision and high-precision calculations.

[0083] Based on the above embodiments, such as Figure 2 As shown, the computing unit 1022 shares the storage resources in the in-memory computing array 1011 with the in-memory computing module 101. When the two architectures process shared sub-models in a multimodal large model, the low-precision parameters of the model are stored in the in-memory computing array 1011, the low-precision part of the high-precision parameters is stored in the in-memory computing array 1011, and the high-precision part is stored in a dedicated array (…). Figure 3In the medium-level memory array, the computing unit 1022 reads two parts of parameters and restores them to high-precision weights through the dequantization unit 10221.

[0084] In some embodiments, the computing unit 1022 further includes a shared weight cache unit 10224 (Shared WCache). The shared weight cache unit 10224 is configured to store the weights for performing the computation tasks. Thus, the weights acquired by the computing unit 1022 and stored in the shared weight cache 10224 are shared by multiple multiply-accumulate processors 10222, allowing the weights to be simultaneously sent to multiple multiply-accumulate processors 10222 for parallel computation with weight reuse. Correspondingly, the computing unit is also configured to perform computational operations on the computation tasks according to the weights.

[0085] In this embodiment, the shared weight cache unit 10224 stores the computation weights and is shared by multiple multiply-accumulate processors 10222, enabling parallel computation with reused weights: on the one hand, it reduces repeated reading of weights, reducing data transmission bandwidth and energy consumption; on the other hand, it supports multiple processors to call the same weight at the same time, greatly improving the parallelism of multiply-accumulate computation, especially suitable for high-frequency use scenarios of repeated weights (such as Transformer multi-head attention weights) in multimodal large models, which not only reduces storage redundancy and saves hardware resources, but also significantly speeds up the computation.

[0086] In some embodiments, the computing unit 1022 further includes other caches 10225. Caches 10225 can be used to store key data other than weights during the operation of the computing unit 102, to supplement the function of the shared weight cache unit 10224 and improve overall computing efficiency. For example, caches 10225 can store input feature data (such as text or image features of a multimodal model), temporary calculation results between the MAC Core and SFU, etc.

[0087] In some embodiments, the in-memory computing array 1011 is a multi-functional unit that can be used for in-memory computing, storage, and dual functions of in-memory computing and storage.

[0088] In some embodiments, the in-memory computing module 101 further includes a control unit 1013 and a buffer / conf unit 1014. The control unit 1013 manages the operating mode of the in-memory computing array 1011, such as independent or collaborative operation. The control unit 1013 is configured to schedule corresponding storage and computing resources according to the computing task when the in-memory computing module 101 is executing a computing task independently, so as to perform the computational operation of the computing task through the storage and computing resources. The buffer / conf unit 1014 is configured to coordinate resource conflicts in the scheduling of storage and computing resources within the in-memory computing array 1011.

[0089] In other words, the in-memory computing array 1011 is a multi-functional unit that can be used for in-memory computing, storage, and dual functions of in-memory computing and storage. Its working mode is managed by the control unit 1013, and its resource conflicts are managed by the buffer and conflict unit 1014.

[0090] In this embodiment, the control unit 1013 can flexibly manage the operating modes of the in-memory computing array 1011 (in-memory computing, storage, dual-function, etc.) and schedule storage and computing resources as needed, enabling the array to accurately adapt to independent or collaborative working scenarios and improve resource utilization. The buffer and conflict resolution unit 1014 coordinates resource conflicts (such as when in-memory computing and storage are performed simultaneously), prioritizing core computing tasks and reducing latency. The combination of these two components allows the in-memory computing module to efficiently respond to the dynamic demands of multimodal tasks while avoiding resource contention and enhancing operational stability.

[0091] In some embodiments, the in-memory computing module 101 further includes a second special function unit 1012. The second special function unit 1012 is configured to perform non-multiply-accumulate computation tasks of the in-memory computing array 1011. For example, the special function unit (SFU) is used to supplement other computations performed by the in-memory computing array 1011 in the in-memory computing module 101.

[0092] In one implementation, when the in-memory computing array 1011 is configured for storage function, the storage-computing separation module reads the storage parameters within the array. The low-precision parameters within the array can be used independently or combined with the parameters in the storage-level memory array to form high-precision parameters.

[0093] In one implementation, when the in-memory computing array 1011 is configured for in-memory computing, the in-memory computing array 1011 performs matrix multiplication and addition calculations with the input and the storage weights in the array.

[0094] In one implementation, the in-memory computing array 1011 functions as a dual-function array for both in-memory computing and storage, alternating between storage and data retrieval. The buffer and conflict resolution unit 1014 determines the occupancy status of the in-memory computing array. When a conflict arises between storage and data retrieval, in-memory computing is prioritized, followed by data retrieval after the computation is complete. The control unit 1013 configures and schedules storage and computing resources on an array-by-array basis, and schedules special function units (SFUs) to supplement other computations.

[0095] like Figure 4As shown, in some embodiments, the computing system 100 further includes a shared token and key-value cache unit 103 (shared token / KV cache unit). The shared token and key-value cache unit 103 is configured to store intermediate results generated by the computational operations performed by the computing unit 1022 and the in-memory computing array 1011 when the in-memory computing separation module 102 and the in-memory computing module 101 are collaboratively executing computational tasks. That is, the shared token and key-value cache unit can be used to store intermediate results shared by the computing unit 1022 and the in-memory computing array during computation, such as encoded tokens, decoded tokens, and KV caches.

[0096] In this embodiment, the shared token and key-value cache unit 103 centrally stores the intermediate results (such as encoded tokens, KV caches, etc.) of the dual-architecture collaborative computation. This avoids the storage-computation separation module and the storage-computation integrated module storing duplicate intermediate data, significantly reducing storage redundancy and saving hardware resources. Intermediate results can be read directly from the shared cache without cross-module transmission, reducing data transfer latency and energy consumption. It can also accelerate the collaborative efficiency of the dual-architecture, ensure real-time synchronization of intermediate results, avoid computational errors caused by data inconsistency, and improve the coherence and overall speed of multimodal reasoning.

[0097] In some embodiments, the computing system 100 further includes a task allocation unit 104. The task allocation unit 104 is configured to parse computing tasks, identify the modal type corresponding to the computing tasks, and dynamically switch the operating modes of the in-memory computing module 101 and the in-memory computing separation module 102 according to the modal type. The operating modes include a mode for independently executing computing tasks and a mode for collaboratively executing computing tasks.

[0098] For example, during multimodal large model inference, the task allocation unit 104 can parse tasks in real time, switch the working mode of the dual architecture (in-memory computing module 101 and in-memory computing separation module 102) according to the task allocation table, and allocate tasks to the dual architecture. The task allocation table may include information such as task type, module parallelism, and token parallelism.

[0099] In this embodiment of the disclosure, the task allocation unit 104 can parse the computation task and identify the corresponding modal type, and then dynamically switch the independent or collaborative working mode of the in-memory computing module and the in-memory computing separation module.

[0100] Based on the computing system 100 described above, some embodiments of this application also provide a model processing method, which can be applied to the computing system 100 provided in the above embodiments. For example... Figure 5 As shown, the method includes step S101.

[0101] S101. When the storage-computing separation module and the storage-computing integrated module work together to perform computing tasks, the computing unit of the storage-computing separation module retrieves the first parameter of the storage unit in the storage-computing separation module and the second parameter of the storage-computing integrated array in the storage-computing integrated module to complete the computational operation of the computing task.

[0102] In this embodiment, when a dual-architecture system collaboratively executes a computational task, the computation unit retrieves the high-precision first parameter from the storage unit of the in-memory computing module and the low-precision second parameter from the in-memory computing module array to complete the calculation. This approach leverages the characteristics of the low-precision parameters of the in-memory computing module to reduce data transmission energy consumption and storage redundancy, while ensuring computational accuracy through the high-precision parameters of the in-memory computing module. This effectively resolves the contradiction between accuracy and energy efficiency in a single architecture. Furthermore, it eliminates the need to deploy duplicate parameters separately for multimodal tasks, allowing for flexible adaptation to differentiated needs such as low-precision text computation and high-precision image computation in large multimodal models. This reduces cross-module data transfer delays and significantly improves the computational efficiency and resource utilization efficiency of multimodal tasks.

[0103] In some embodiments, when the storage-computation separation module performs the computation task independently, the computation unit of the storage-computation separation module is also configured to invoke the first parameter.

[0104] In some embodiments, when the in-memory computing module performs the computing task independently, the in-memory computing array is also configured to invoke the second parameter.

[0105] It should be noted that the foregoing explanation of the system embodiments also applies to the methods of the embodiments of this disclosure, and the principles are the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0106] Figure 5 The model deployment flowchart provided for the embodiments of this application is as follows: Figure 6 As shown, the model deployment process may include steps S201-S204. This embodiment implements model deployment based on the aforementioned computing system 100 and model processing method.

[0107] S201, Model Analysis and Characteristic Analysis.

[0108] Model analysis and characteristic analysis refer to decomposing a large multimodal model into basic modules so that the computational system can perform computational tasks on the large multimodal model.

[0109] like Figure 7As shown, in some embodiments, model parsing and feature analysis include multimodal model structure deconstruction and dependency graph construction. Multimodal model structure deconstruction involves analyzing the input and output modalities (images, text, video, etc.) and model structure of the large multimodal model. Based on functional and computational requirements, the large multimodal model is decomposed into independent modules, including input processing modules (visual encoders, text encoders, modality transducers, etc.), inference and generation modules (Transformers, Diffusion, etc.), and output processing modules (visual decoders, text decoders, task heads, etc.). Dependency graph construction involves drawing data flow and control flow graphs between modules based on the input and output of different modalities, clarifying the input and output dependencies between modules. The parameter scale and original parameter accuracy of each module are obtained, and the computational and storage requirements of each module are analyzed.

[0110] S202, Hybrid precision quantization of hardware sensing and multimodal sensing.

[0111] In one implementation, step S202 includes parallelism analysis within each module and quantization precision matching for each module. This step is to place the first parameter and the second parameter in the in-memory computing module and the in-memory computing separate module for subsequent calculations.

[0112] Among them, the parallelism analysis within each module includes the hardware-aware quantization process, which adds latency loss, computational loss and power consumption loss on the basis of quantization accuracy, so that the quantization scheme matches the hardware characteristics of the in-memory computing module and the in-memory computing separation module, and the combination of software and hardware achieves the best comprehensive performance in terms of speed, power consumption and accuracy.

[0113] For speed and power consumption calculations, some modules have no dependencies between tokens, so parallel computing can be used to reduce latency and power consumption. Therefore, we analyze the parallelism of token processing in each module and calculate the latency and power consumption of parallel processing.

[0114] Quantization accuracy matching for each module involves quantizing all modules of the large model, with all parameters within each module using the same quantization accuracy, while different modules can use different quantization accuracies. In structural analysis, all modules are categorized into independent modules and coupled modules based on their model-sharing relationships; coupled modules share the same model.

[0115] For example, the Large Language Model (LLM) is shared by image encoding, text encoding, image reasoning, and text reasoning. The shared model can use different quantization precisions in different modules (e.g., the LLM is quantized to an 8-bit integer in the text reasoning module and to a 16-bit floating-point number in the image reasoning module).

[0116] For example, the quantization process for an independent module is as follows: For various computing modes in a hybrid architecture, the quantization loss L_quant, simulation loss L_cim, computing power L_power, and computing latency L_time are calculated under different precision support conditions for different computing modes. The overall loss from quantitative deployment is calculated as L = L_quant + αL_power + βL_time + γL_cim, where α, β, and γ are balancing coefficients, which are added when the calculation mode is based on in-memory computing with errors. The calculation mode and quantization accuracy that minimize the overall quantitative deployment loss are selected as the matching scheme, and the parameters are quantized according to the quantization accuracy.

[0117] The quantization process for the coupled module is as follows: The coupled modules each use different computation modes and support precision to calculate the total loss. Since the models of each module are coupled, the high-precision computation module needs to be compatible with the computation mode and computation precision of the low-precision computation module.

[0118] For example, as shown in the table below, LLM1-LLM4 are coupled modules. LLM1 uses low-precision calculation with error for INT8, LLM2 uses high-precision calculation with error for FP8+INT8(CIM), LLM3 uses low-precision calculation for INT8, and LLM4 uses high-precision calculation for FP8+INT8. The total quantitative deployment loss is the sum of the losses of all coupled modules. The scheme with the minimum total quantitative deployment loss is selected as the calculation mode and quantization precision for the mutually coupled modules. The parameters of each module are quantized according to the quantization precision, with high-precision quantization compatible with low-precision quantization. Here, INT8 stands for 8-bit integer, and FP8 stands for 8-bit floating-point number.

[0119]

[0120] The hybrid quantization implementation examples are shown in the table below, where N is the number of input semantic tokens, M is the number of input images, and K is the number of tokens converted from each image.

[0121] Understandably, low-precision calculations can be performed by in-memory computing modules or in-memory computing separate modules, while high-precision calculations can be performed by in-memory computing separate modules.

[0122]

[0123] S203, Hybrid Architecture Mapping and Resource Allocation.

[0124] Hybrid architecture mapping and resource allocation can include parallel design between modules and hybrid architecture parameter mapping.

[0125] In some embodiments, hybrid architecture mapping includes parallel design of input-output independent modules based on large model structure parsing. For example, in the encoding phase, text encoders and image encoders can be executed in parallel, and LLM modules can be executed in parallel.

[0126] In some embodiments, the hybrid architecture parameter mapping includes mapping the model parameters of each module in CIM (Centered In-Memory Array) and SCM (Storage Class Memory Array), and using differentiated parameter mapping according to the attributes and computing modes of the modules to obtain the first parameter and the second parameter.

[0127] For independent modules, the parameters of each module are mapped separately according to the calculation mode. Module parameters for low-precision calculations with errors are mapped to the CIM array, while those for low-precision and high-precision calculations are mapped to the SCM array. In high-precision calculations with errors, the higher bits are mapped to the SCM array, and the lower bits are mapped to the CIM array. For coupled modules, the parameters of coupled modules are mapped uniformly. Low-precision calculations with errors and low-precision calculations are mapped to the CIM space, while those for high-precision calculations with errors and high-precision calculations are mapped to the SCM space, and the lower bits are mapped to the CIM space.

[0128] For example, the hybrid architecture parameter mapping is shown in the table below:

[0129] S204, Deployment Verification and Task Assignment.

[0130] Deployment verification and task allocation use test cases to evaluate all types of tasks that the computing system needs to process at the same time. Based on the parallelism within modules, the parallelism between modules, and the model parameter mapping mode, the system working mode is matched, and a runtime task allocation table is generated, including the resource calls and working modes of the in-memory computing module and the in-memory computing separation module.

[0131] For example, tokens can be used in parallel within a module, CIM can employ pipelined computation, and computational units can be reused with weights to improve computational speed. Another example is that modules can be used in parallel, employing a dual-architecture pattern to improve computational speed, and resources can be scheduled based on model parameter mapping.

[0132] An exemplary task allocation implementation is shown in the table below. When a task changes during model inference, the runtime task allocation table is consulted, system resources are dynamically allocated based on the task type, and the working mode is changed to achieve optimal latency, power consumption, and accuracy for multimodal large model inference.

[0133]

[0134] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0135] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like ​ As shown, the electronic device 700 includes a calculator 701, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 702 or loaded from memory 708 into RAM (Random Access Memory) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The calculator 701, ROM 702, and RAM 703 are interconnected via bus 704. An I / O (Input / Output) interface 705 is also connected to bus 704.

[0137] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; memory 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] Calculator 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of calculator 701 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processors, controllers, microcontrollers, etc. Calculator 701 performs the various methods and processes described above, such as model processing methods. For example, in some embodiments, model processing methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as memory 708. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by calculator 701, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the calculator 701 may be configured to perform the aforementioned model processing method by any other suitable means (e.g., by means of firmware).

[0139] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0144] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0145] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0146] As can be seen from the above technical solutions, the computing system, model processing method, electronic device, and storage medium of this application relate to the field of artificial intelligence. The system includes an in-memory computing module and a separate in-memory computing module. The in-memory computing module includes an in-memory computing array, and the separate in-memory computing module includes a storage unit and a computing unit. When the separate in-memory computing module and the in-memory computing module collaboratively execute computing tasks, the computing unit is configured to retrieve a first parameter from the storage unit and a second parameter from the in-memory computing array to complete the computational operations. By applying the technical solution of this application, the computing unit retrieves parameters across modules, enabling collaborative computation of low-precision and high-precision parameters, which can reduce redundant parameter storage in coupled sub-modules in multimodal models. Simultaneously, the dual-module collaboration adapts to different precision and modal computational needs, improving data interaction efficiency and computational parallelism, and effectively balancing inference accuracy, energy consumption, and latency.

[0147] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A computing system, characterized in that, include: In-memory computing module, including in-memory computing array; The storage-compute separation module includes a storage unit and a computing unit; When the storage-computing separation module and the storage-computing integrated module work together to perform computing tasks, the computing unit is configured to retrieve the first parameters of the storage unit and the second parameters of the storage-computing integrated array to complete the computational operations of the computing task.

2. The computing system according to claim 1, characterized in that, The in-memory computing module's in-memory computing array is integrated into the storage unit of the in-memory computing separation module.

3. The computing system according to claim 1 or 2, characterized in that, The computing unit includes: The dequantization unit is configured to perform a fusion process on the first parameter of the storage unit and the second parameter of the in-memory computing array to obtain fused parameters; The computing unit is also configured to perform the computing task according to the fusion parameters.

4. The computing system according to claim 1 or 2, characterized in that, The precision of the first parameter is higher than that of the second parameter.

5. The computing system according to claim 4, characterized in that, When the storage-computation separation module performs the computation task independently, the computation unit of the storage-computation separation module is also configured to call the first parameter.

6. The computing system according to claim 4, characterized in that, When the in-memory computing module performs the computing task independently, the in-memory computing array is also configured to invoke the second parameter.

7. The computing system according to claim 1 or 2, characterized in that, The computing unit further includes: The multiply-accumulate processor is configured to perform multiply-accumulate calculation tasks of varying precision. The first special function unit is configured to perform non-multiply-accumulate computation tasks of the storage-computation separation module.

8. The computing system according to claim 1 or 2, characterized in that, The computing unit further includes a shared weight cache unit; the shared weight cache unit is configured to store the weights for performing the computing task; The computing unit is also configured to perform computational operations on the computing task according to the weights.

9. The computing system according to claim 1 or 2, characterized in that, The in-memory computing module also includes: The second special function unit is configured to perform non-multiply-accumulate computation tasks of the in-memory computing array.

10. The computing system according to claim 1, characterized in that, The in-memory computing module also includes: The control unit, when the in-memory computing module performs a computing task independently, schedules the corresponding storage and computing resources according to the computing task, so as to perform the computing operation of the computing task through the storage and computing resources; The buffer and conflict unit is configured to coordinate resource conflicts in the in-memory computing array that are used to schedule the storage resources and the computing resources.

11. The computing system according to claim 1, characterized in that, Also includes: The shared lexical and key-value caching unit is configured to store intermediate results generated by the computational unit and the in-memory array performing computational operations when the in-memory separation module and the in-memory integrated module are performing computational tasks in collaboration.

12. The computing system according to claim 1, characterized in that, Also includes: The task allocation unit is configured to parse the computing task, identify the modality type corresponding to the computing task, and dynamically switch the working modes of the in-memory computing module and the in-memory computing separation module according to the modality type; the working modes include a mode of independently executing computing tasks and a mode of collaboratively executing computing tasks.

13. A model processing method, characterized in that, The method includes: When the storage-computing separation module and the storage-computing integrated module work together to perform computing tasks, the computing unit of the storage-computing separation module retrieves the first parameter of the storage unit in the storage-computing separation module and the second parameter of the storage-computing integrated array in the storage-computing integrated module to complete the computational operation of the computing task.

14. The method according to claim 13, characterized in that, Also includes: When the storage-computation separation module performs the computation task independently, the computation unit of the storage-computation separation module is also configured to call the first parameter.

15. The method according to claim 13, characterized in that, Also includes: When the in-memory computing module performs the computing task independently, the in-memory computing array is also configured to invoke the second parameter.

16. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 13-15.

17. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 13-15.

18. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 13-15.

Images