Model data processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a model data processing method and device, computer equipment and a storage medium. Comprising the steps that a pre-training model and model data corresponding to the pre-training model are acquired, and the pre-training model is a model under a state space model structure; determining a first operator needing data processing acceleration in the pre-training model according to the model data; the first operator is decoupled to obtain a second operator corresponding to the first operator, and a hardware processing unit corresponding to the second operator is different from a hardware processing unit corresponding to the first operator; acquiring to-be-processed data corresponding to the second operator, and determining discretization parameters corresponding to the to-be-processed data; and performing data processing on the to-be-processed data and the discretization parameters according to the second operator to obtain a data processing result. Therefore, the second operator decoupled through the first operator of the model can be loaded by other hardware processing units, discretization processing is carried out on the to-be-processed data, the calculation amount of the model is reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a model data processing method, device, computer equipment and storage medium. Background Art

[0002] In the field of artificial intelligence, model data reasoning is a key link, and the model's data processing speed will directly affect the model's data processing efficiency. In the process of data processing, the model often involves multiple types of operators, which are used to process data. Operators usually perform continuous calculations during data processing, which leads to low computing efficiency of the hardware architecture running the model.

[0003] In related technologies, the data processing efficiency of the model is improved by using technologies such as operator fusion, quantization, model pruning, and low-rank decomposition. However, these technologies have limited data processing acceleration effects in the data processing of models with certain structures, such as the limited data processing acceleration effect of the visual state space model under the state space model structure (SSM).

[0004] Therefore, for models under the state-space model structure, the acceleration effect of data processing for such models in related technologies is low, and the data processing speed of such models cannot be significantly improved, resulting in slow data processing speed and low efficiency for such models. Summary of the invention

[0005] The embodiments of the present application provide a model data processing method, apparatus, computer equipment and storage medium, which can significantly improve the data processing speed and processing efficiency of the model under the state-space model structure.

[0006] In order to achieve the above-mentioned purpose, an embodiment of the present application provides a model data processing method on the one hand, comprising: Acquire a pre-trained model and model data corresponding to the pre-trained model, wherein the pre-trained model is a model under a state space model structure; Determine, according to the model data, a first operator in the pre-trained model that needs to be accelerated in data processing; Performing decoupling processing on the first operator to obtain a second operator corresponding to the first operator, wherein a hardware processing unit corresponding to the second operator is different from a hardware processing unit corresponding to the first operator; Acquire the data to be processed corresponding to the second operator, and determine the discretization parameters corresponding to the data to be processed; The data to be processed and the discretization parameters are processed according to the second operator to obtain a data processing result.

[0007] In order to achieve the above-mentioned purpose, an embodiment of the present application provides a model data processing device, including: A first acquisition module is used to acquire a pre-trained model and model data corresponding to the pre-trained model, wherein the pre-trained model is a model under a state space model structure; A determination module, configured to determine a first operator in the pre-trained model that needs to be accelerated in data processing according to the model data; a decoupling module, configured to perform decoupling processing on the first operator to obtain a second operator corresponding to the first operator, wherein the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; A second acquisition module, used to acquire the data to be processed corresponding to the second operator, and determine the discretization parameters corresponding to the data to be processed; A processing module is used to perform data processing on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0008] In some embodiments, the decoupling module is used to: Determine the operation logic corresponding to each sub-operator in the first operator; Determine the type of hardware processing unit corresponding to each sub-operator according to the operation logic; Each of the sub-operators is decoupled according to the type of the hardware processing unit to obtain a second operator.

[0009] In some embodiments, the decoupling module is used to: Determine the operation logic corresponding to each sub-operator in the first operator; Modify each sub-operator according to the preset calculation library and the operation logic corresponding to each sub-operator to generate a target sub-operator corresponding to each sub-operator; A second operator corresponding to the first operator is generated according to the target sub-operator corresponding to each sub-operator.

[0010] In some embodiments, the processing module is used to: Determine a state transfer matrix, an input control matrix, and an output control matrix corresponding to the second operator; Determine a discrete step length matrix according to the discretization parameter, multiply the discrete step length matrix and the state transfer matrix, and perform exponential processing on each element in the multiplied matrix to obtain a discrete state transfer matrix; Determine an input matrix corresponding to the current moment according to the data to be processed, and multiply the input matrix, the discrete step size matrix and the input control matrix to obtain a discrete input control matrix; Obtaining a state matrix corresponding to the previous moment, and multiplying the state matrix and the discrete state transfer matrix and adding the matrix to the discrete input control matrix to obtain a target state matrix corresponding to the current moment; Multiplying the target state matrix and the output control matrix to obtain an output matrix corresponding to the current moment; The data processing result corresponding to the data to be processed is determined according to the output matrix corresponding to the current moment.

[0011] In some embodiments, the processing module is used to: Determine the output matrix corresponding to the input matrix at other times in the data to be processed; The output matrix corresponding to the current moment and the output matrix corresponding to the other moments are stacked to obtain a first target output matrix corresponding to the data to be processed, where the first target output matrix is ​​a data processing result corresponding to the data to be processed.

[0012] In some embodiments, the processing module is used to: Performing data format conversion on the second operator and other operators corresponding to the pre-trained model to obtain a target pre-trained model in a target data format; Determining a third operator corresponding to the second operator in the target pre-trained model; The data to be processed and the discretization parameters are processed according to the third operator to obtain a data processing result.

[0013] In some embodiments, the processing module is used to: Determine a discrete step size matrix, a state transfer matrix, an input control matrix and an output control matrix corresponding to the third operator; Performing block accumulation processing on the input matrix corresponding to the data to be processed to obtain a first processing matrix, and performing one-dimensional convolution processing on the first processing matrix and the state transfer matrix to obtain a target state matrix; Multiplying the input matrix, the discrete step size matrix and the input control matrix to obtain a first target input matrix; Performing block accumulation processing on the first target input matrix to obtain a second target input matrix, and determining an output matrix to be processed according to the second target input matrix, the target state matrix and a preset state space matrix whose elements are all zero; One-dimensional convolution processing is performed on the output matrix to be processed and the output control matrix to obtain a second target output matrix, where the second target output matrix is ​​a data processing result corresponding to the data to be processed.

[0014] In some embodiments, the processing module is used to: The input matrix corresponding to the data to be processed is processed in blocks to obtain multiple sub-matrices; The cumulative sum matrix corresponding to each sub-matrix is ​​determined according to the time step of each sub-matrix, and the cumulative sum matrix corresponding to each sub-matrix is ​​subjected to matrix stacking processing to obtain a first processing matrix.

[0015] In some embodiments, the processing module is used to: Determine a submatrix corresponding to each time step in the first target input matrix; The sub-matrices corresponding to each time step are accumulated to obtain a second target input matrix.

[0016] In order to achieve the above-mentioned purpose, on the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the model data processing method provided by the embodiment of the present application.

[0017] In order to achieve the above-mentioned objectives, on the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the model data processing method provided in the embodiment of the present application when executing the computer program.

[0018] In an embodiment of the present application, a pre-trained model and model data corresponding to the pre-trained model are obtained, where the pre-trained model is a model under a state-space model structure; a first operator in the pre-trained model that needs to be accelerated in data processing is determined according to the model data; the first operator is decoupled to obtain a second operator corresponding to the first operator, and the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; data to be processed corresponding to the second operator is obtained, and discretization parameters corresponding to the data to be processed are determined; data processing is performed on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0019] In this way, the first operator that needs data processing acceleration is determined by the model data of the pre-trained model under the state-space model structure, and then the first operator is decoupled to obtain the second operator. The second operator can be applied to other hardware processing units relative to the first operator, so that the computing power of other hardware processing units can be used to accelerate the operation of the second operator, and the data to be processed by the second operator is discretized by using discretization parameters. In this way, the second operator avoids directly processing the data to be processed as a whole, but processes the discretized data, which can reduce the second operator's influence on the overall data to be processed. Complex calculations are performed, thereby improving the processing speed of the second operator for the data to be processed. Therefore, compared with the limited data acceleration processing effect of the model acceleration technology in the related technology on the model under the state-space model structure, the present application can use the first operator to decouple to obtain the second operator, and then load the second operator through other hardware units, and discretize the data to be processed by the second operator, which can greatly improve the data processing speed of the second operator, thereby greatly improving the data processing speed of the pre-trained model of the entire state-space model. Therefore, the data processing acceleration effect of the pre-trained model of the state-space model is more obvious than the related technology.

[0020] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of the system framework corresponding to the model data processing method provided in the embodiment of the present application; Figure 2 It is a flowchart of the model data processing method provided in the embodiment of the present application; Figure 3 It is a flowchart diagram of the steps included in step 230 provided in the embodiment of the present application; Figure 4 is another flow chart of the steps included in step 230 provided in an embodiment of the present application; Figure 5 It is a comparison chart of the number of operators corresponding to the pre-trained model and the target pre-trained model provided in the embodiment of the present application; Figure 6 is another flow chart of the model data processing method provided in an embodiment of the present application; Figure 7 is a structural schematic diagram of a model data processing device provided in an embodiment of the present application; Figure 8 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0024] It should be noted that in the specific implementation of this application, model-related data is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0025] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The model data processing method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The model data processing method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the model data processing method, etc., but is not limited to the above forms.

[0027] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0028] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations: State Space Model (SSM): A state space model is a mathematical model of a dynamic system that describes the dynamic behavior of a system through state variables. A state space model mainly consists of two equations: the state equation and the observation equation.

[0029] Visual State Space Model: It is the application of state space model in the field of vision. It regards the visual system (such as target recognition, tracking, scene understanding and other tasks in computer vision) as a dynamic system, which also contains state equations and observation equations.

[0030] Vmamba model: It is a model corresponding to the state-space model structure. It is mainly inspired by the ability of state-space models in long sequence modeling, and introduces the relevant concepts of state-space models into the visual field. Its core contains a series of visual state space (VSS) blocks, which use unique computational methods to process visual data.

[0031] Operator: In the context of a model, an operator is an abstract representation of an operation on data. It can be seen as a function or a transformation rule that acts on input data and produces corresponding output. An operator receives data in a specific format (such as vectors, tensors, etc.) as input, and outputs processed data after a series of predefined calculation steps.

[0032] Discretization: Discretization is a process of converting continuous objects (such as functions, space, time, data, etc.) into discrete forms. It has wide applications in many fields such as mathematics and computer science. The core idea is to divide or approximate continuous quantities into discrete units through specific methods, so that complex continuous problems can be represented and calculated in a more manageable discrete form.

[0033] ONNX (Open Neural Network Exchange): ONNX is an open format for representing deep learning models. Its main purpose is to provide a unified model representation standard that enables easy model conversion and interoperability between different deep learning frameworks (such as PyTorch, TensorFlow, etc.). For example, a model trained in PyTorch can be converted to the ONNX format and then deployed and inferred on other inference engines that support ONNX (such as ONNX Runtime).

[0034] Other related terms that need to be explained will be introduced in the following text.

[0035] First, let’s introduce the technical problems existing in related technologies: In the field of artificial intelligence, model data reasoning is a key link, and the model's data processing speed will directly affect the model's data processing efficiency. In the process of data processing, the model often involves multiple types of operators, which are used to process data. Operators usually perform continuous calculations during data processing, which leads to low computing efficiency of the hardware architecture running the model.

[0036] In related technologies, the data processing efficiency of the model is improved by using technologies such as operator fusion, quantization, model pruning, and low-rank decomposition. However, these technologies have limited data processing acceleration effects in the data processing of models with certain structures, such as the limited data processing acceleration effect of the visual state space model under the state space model structure (SSM).

[0037] Therefore, for models under the state-space model structure, the acceleration effect of data processing for such models in related technologies is low, and the data processing speed of such models cannot be significantly improved, resulting in slow data processing speed and low efficiency for such models.

[0038] In order to solve the above technical problems, the embodiments of the present application provide a model data processing method, device, computer equipment and storage medium.

[0039] Among them, the first operator that needs data processing acceleration is determined by the model data of the pre-trained model under the state-space model structure, and then the first operator is decoupled to obtain the second operator. The second operator can be applied to other hardware processing units relative to the first operator, so that the computing power of other hardware processing units can be used to accelerate the operation of the second operator, and the data to be processed by the second operator is discretized by using discretization parameters. In this way, the second operator avoids directly processing the data to be processed as a whole, but processes the discretized data, which can reduce the second operator's response to the overall data to be processed. Complex calculations are performed, thereby improving the processing speed of the second operator for the data to be processed. Therefore, compared with the limited data acceleration processing effect of the model acceleration technology in the related technology on the model under the state-space model structure, the present application can use the first operator to decouple to obtain the second operator, and then load the second operator through other hardware units, and discretize the data to be processed by the second operator, which can greatly improve the data processing speed of the second operator, thereby greatly improving the data processing speed of the pre-trained model of the entire state-space model. Therefore, the data processing acceleration effect of the pre-trained model of the state-space model is more obvious than the related technology.

[0040] The model data processing method, device, computer equipment and storage medium will be introduced in detail in the following text.

[0041] See also Figure 1 , Figure 1 1 is a schematic diagram of a system framework corresponding to the model data processing method provided in the embodiment of the present application. The model data processing method provided in the embodiment of the present application can be applied to the system framework.

[0042] It includes a terminal 140, the Internet 130, a gateway 120, a server 110, and the like.

[0043] The terminal 140 or the server 110 may be a device for executing the model data processing method.

[0044] The terminal 140 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to image classification, target detection, semantic segmentation, etc. In addition, it can be a single device or a combination of multiple devices. For example, multiple desktop computers are interconnected through a local area network, share a display, etc. to work together, and together constitute a terminal 140. The terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0045] The server 110 refers to a computer system that can provide certain services to the terminal 140. Compared with the ordinary terminal 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0046] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0047] The model data processing method in the embodiment of the present application can be applied to a variety of scenarios, such as image classification, target detection, semantic segmentation, etc. The scenario to which the model data processing method in the present application is applied is not limited here.

[0048] See also Figure 2 , Figure 2 It is a flow chart of the model data processing method provided in the embodiment of the present application.

[0049] The model data processing method provided in the embodiment of the present application may include the following steps: Step 210: obtaining a pre-trained model and model data corresponding to the pre-trained model, where the pre-trained model is a model under a state space model structure; Step 220: Determine the first operator in the pre-trained model that needs to be accelerated in data processing according to the model data; Step 230: Decouple the first operator to obtain a second operator corresponding to the first operator, where the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; Step 240: Obtain the data to be processed corresponding to the second operator, and determine the discretization parameters corresponding to the data to be processed; Step 250: Perform data processing on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0050] Steps 210 to 250 will be described in detail below.

[0051] In step 210, a pre-trained model and model data corresponding to the pre-trained model are obtained, where the pre-trained model is a model under a state-space model structure.

[0052] In the present application, the pre-trained model may be a model under a state space model structure, wherein the pre-trained model includes model data, the model data may include data of multiple operators corresponding to the pre-trained model, and the corresponding multiple operators in the pre-trained model may be determined through the model data.

[0053] Pre-trained models are mainly trained on the architecture of the Graphics Processing Unit (GPU), such as the VMamba model. If you want to migrate to other architectures for inference, you need to decouple and convert some operators in the pre-trained model so that these operators are suitable for loading and running on other hardware processing units, such as the Central Processing Unit (CPU).

[0054] In step 220, a first operator in the pre-trained model that needs data processing acceleration is determined based on the model data.

[0055] Based on the model data, multiple operators corresponding to the pre-trained model can be determined. Since the pre-trained model is trained on the architecture of the graphics processor, most of these operators are suitable for running and loading on the graphics processor. When the graphics processor is busy with tasks and has a large amount of computational load, the processing speed of the data to be processed input into the pre-trained model will be slower.

[0056] Among the multiple operators of the pre-trained model, some operators are built based on the functions of the pytorch framework, so they support loading and running on different hardware processing units. PyTorch has an abstract device management mechanism, which abstracts and encapsulates different hardware processing units (such as CPU, GPU). At the bottom layer, PyTorch defines the torch.device class, which can be used to specify the device where the tensor is located. For example, you can use torch.device("cpu") or torch.device("cuda:0") (assuming that there is a GPU device and the number is 0) to specify that the tensor is operated on the CPU or GPU respectively. This abstraction allows the operator to be independent of specific hardware devices. As long as the device supports the device abstraction layer interface of PyTorch, the operator can run on it.

[0057] However, in the pre-trained model, some custom operators or operators that are not built based on the PyTorch framework often rely on graphics processors for processing and cannot be directly loaded and processed by other hardware processing units. These operators are determined by model data and determined as the first operators.

[0058] In order to improve the data processing speed of the first operator during the calculation process, the present application can perform subsequent processing on the first operator based on the model data in the pre-trained model to accelerate the data processing, so as to achieve the migration of at least part of the sub-operators in the first operator to other hardware processing units for loading, such as migrating to the central processing unit for loading and processing, so as to reduce the computing pressure of the graphics processing unit, and achieve parallel processing of the data that needs to be processed by the pre-trained model by the graphics processing unit and the central processing unit, so as to improve the processing speed and efficiency of the data by the pre-trained model.

[0059] For another example, without changing the calculation purpose and calculation method of the first operator, the number of sub-operators in the first operator can be reduced, thereby obtaining a second operator with fewer sub-operators, which can greatly reduce the data that needs to be processed in the intermediate steps of the data processing process. These will be described later.

[0060] In step 230, the first operator is decoupled to obtain a second operator corresponding to the first operator, and the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator.

[0061] Among them, the first operator can be a CrossScanTriton operator, which is an operator for scanning operations across different dimensions. When processing images or other multi-dimensional data, traditional linear scanning may not be sufficient to capture sufficient contextual information. CrossScanTriton uses the Triton accelerator to achieve efficient cross-dimensional scanning, which can scan data from different directions or paths to more comprehensively understand and process the spatial relationship of the data.

[0062] Use of the CrossScanTriton operator in pre-trained models: The CrossScanTriton operator is used in pre-trained models to implement bidirectional scanning or multi-directional scanning, helping the pre-trained model to obtain broader contextual information when processing visual data and enhancing the model's understanding of the relationship between different locations in the image.

[0063] The CrossScanTriton operator is implemented and runs based on the Triton accelerator, and the Triton accelerator is implemented based on the graphics processor. If the CrossScanTriton operator is to be migrated to other hardware processing units to implement operations, the CrossScanTriton operator needs to be decoupled to obtain the corresponding decoupled second operator. The number of second operators can be one, two or more, so that the second operator can be applied to other hardware processing units, such as a central processing unit.

[0064] The first operator may be a CrossMergeTriton operator, which is used to merge or fuse features from different scanning directions or different levels. By utilizing the parallel computing capability of the Triton accelerator, this operator can efficiently process and merge data, ensuring that information from different sources can be effectively integrated together to form a more representative feature representation.

[0065] In visual tasks, pre-trained models need to integrate information from different scanning paths or features from different levels. The CrossMergeTriton operator can help pre-trained models perform this complex feature fusion while maintaining efficiency, thereby improving the performance of the model. The CrossMergeTriton operator is suitable for loading and running on a graphics processor, and needs to be decoupled to obtain the corresponding decoupled second operator. The number of second operators can be one, two, or more, so that the second operator can be applied to other hardware processing units, such as a central processing unit.

[0066] The first operator may be a SelectiveScanOflex operator, which is an implementation of SelectiveScan that can dynamically select a scanning path or method based on the characteristics of the input data or the requirements of the model. For example, it can choose to scan from left to right, from right to left, or bidirectionally, or perform selective scanning in different dimensions. Different from the traditional fixed scanning path, this operator can select different scanning methods based on the characteristics of the input data or the requirements of the model to better capture important information in the data.

[0067] In a pre-trained model, such as the VMamba model, the SelectiveScanOflex operator can be used to enhance the flexibility of the model, enabling it to adjust the scanning strategy according to different input images or task requirements, thereby improving the generalization ability and efficiency of the model in different scenarios. The SelectiveScanOflex operator is suitable for loading and running on a graphics processor, and needs to be decoupled to obtain the corresponding decoupled second operator. The number of second operators can be one, two, or more, so that the second operator can be applied to other hardware processing units, such as a central processing unit.

[0068] See also Figure 3 , Figure 3 230 provided in an embodiment of the present application includes steps. In some implementations, decoupling the first operator to obtain a second operator corresponding to the first operator includes the following steps: Step 301: Determine the operation logic corresponding to each sub-operator in the first operator; Step 302: Determine the type of hardware processing unit corresponding to each sub-operator according to the operation logic; Step 303: Decouple each sub-operator according to the type of hardware processing unit to obtain a second operator.

[0069] Steps 301 to 303 are described in detail below.

[0070] In step 301, the operation logic corresponding to each sub-operator in the first operator is determined.

[0071] The first operator includes multiple sub-operators, and the operation logic of each sub-operator can be analyzed, wherein some operation logics are based on the graphics processor, such as parallel calculation logic, and some operation logics are based on the central processing unit, such as precise calculation logic.

[0072] In step 302, the type of hardware processing unit corresponding to each sub-operator is determined according to the operation logic.

[0073] The type of hardware processing unit corresponding to each sub-operator can be determined according to the operation logic, that is, the hardware processing units corresponding to some sub-operators are graphics processors, and the hardware processing units corresponding to some sub-operators are central processing units.

[0074] In step 303, each sub-operator is decoupled according to the type of hardware processing unit to obtain a second operator.

[0075] Then, each sub-operator is decoupled according to the type of hardware processing unit to obtain the second operator. Among them, the loading of some sub-operators must rely on the graphics processor, and this type of operator maintains the original operation and loading mode. However, the operation and loading of some sub-operators can be performed by the graphics processor or the central processing unit, and they are determined as the second operator. Subsequently, these two different types of sub-operators can be loaded on the graphics processor and the central processing unit respectively, so as to realize the transfer of all the computing tasks of the original first operator on the graphics processor to the central processing unit, thereby improving the data processing efficiency.

[0076] See also Figure 4 , Figure 4 is another flow chart of the steps included in step 230 provided in an embodiment of the present application. In some implementations, the first operator is decoupled to obtain a second operator corresponding to the first operator, including: Step 401, determining the operation logic corresponding to each sub-operator in the first operator; Step 402: modify each sub-operator according to the preset calculation library and the operation logic corresponding to each sub-operator to generate a target sub-operator corresponding to each sub-operator; Step 403: Generate a second operator corresponding to the first operator according to the target sub-operator corresponding to each sub-operator.

[0077] Steps 401 to 403 will be explained in detail below.

[0078] In step 401, the operation logic corresponding to each sub-operator in the first operator is determined.

[0079] Among them, the operation logic corresponding to each sub-operator in the first operator can be determined, and the input, output and specific functions of each sub-operator in the first operator can be specifically analyzed, such as whether it is used for matrix multiplication, convolution operation, or other specific mathematical operations, so as to determine the operation logic of each sub-operator.

[0080] In step 402, each sub-operator is modified according to a preset calculation library and the operation logic corresponding to each sub-operator to generate a target sub-operator corresponding to each sub-operator.

[0081] Then, each sub-operator is modified according to the preset calculation library and the operation logic corresponding to each sub-operator to generate the target sub-operator corresponding to each sub-operator. The preset calculation library is specifically a preset calculation library that can be defined under the PyTorch framework, which contains a variety of basic mathematical operators, such as operators for basic arithmetic operations such as addition, subtraction, multiplication, and division.

[0082] After determining the operation logic of each sub-operator, a target sub-operator with the same function and role as the original sub-operator can be constructed through the preset computing library. The target sub-operator has the same data input, data processing logic, role and function as the original sub-operator, but is defined through the PyTorch framework. This is to achieve the conversion of atomic operators and obtain better compatibility of the target sub-operator with other hardware processing units.

[0083] In step 403, a second operator corresponding to the first operator is generated according to the target sub-operator corresponding to each sub-operator.

[0084] Then, the second operator corresponding to the first operator is generated according to the target sub-operator corresponding to each sub-operator. For example, the target sub-operators corresponding to each sub-operator can be integrated according to the overall operation logic of the first operator, for example, the inputs and outputs of different target sub-operators are connected according to the overall operation logic of the first operator to generate the second operator.

[0085] The hardware processing unit of the first operator and the hardware processing unit of the second operator may be different. For example, the hardware processing unit of the first operator may be a graphics processor, while the hardware processing unit of the second operator may be a central processing unit.

[0086] During the data processing of the pre-trained model, the second operator corresponding to the first operator of the pre-trained model can be handed over to the central processing unit for execution, which can share the computing tasks of the graphics processor and improve the computing efficiency of the graphics processor. At the same time, the central processing unit and the graphics processor simultaneously process the data to be processed corresponding to the pre-trained model, which can improve the data processing speed and processing efficiency.

[0087] In step 240, the data to be processed corresponding to the second operator is obtained, and the discretization parameters corresponding to the data to be processed are determined.

[0088] The data to be processed corresponding to the second operator may be continuous data, such as a continuous matrix corresponding to continuous image data. Discretization parameters corresponding to the data to be processed may be determined, and the discretization parameters may be used to discretize the continuous data to be processed.

[0089] Discretization processing usually involves converting continuous data or operations into discrete form. In deep learning models, this means mapping continuous weights, activation values, or other parameters from a continuous real number space to a discrete space. For example, converting floating-point weights to fixed-point numbers, or approximating continuous activation functions (such as Sigmoid or ReLU) as piecewise linear functions or step functions. This conversion can simplify some originally complex calculation operations, thereby reducing the number of required operators. For example, for the discretization of some activation functions, if the continuous Sigmoid function is approximated as a step function, the complex mathematical operations originally used to calculate the Sigmoid function can be simplified to simple comparison and assignment operations, and a dedicated activation function operator may not be required, or a simpler logical judgment operator can be used instead, thereby reducing the operators in the model.

[0090] In step 250, data processing is performed on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0091] The data to be processed and the discretized parameters can be input into the second operator for data processing, thereby obtaining a data processing result. Alternatively, the pre-trained model can be converted into the ONNX data format to obtain a converted target pre-trained model, and then a third operator corresponding to the second operator is determined in the target pre-trained model, and the data to be processed and the discretized parameters are processed according to the third operator to obtain a data processing result.

[0092] In some implementations, performing data processing on the data to be processed and the discretization parameter according to the second operator to obtain a data processing result includes: (1.1) Determine the state transfer matrix, input control matrix and output control matrix corresponding to the second operator; (1.2) Determine the discrete step length matrix according to the discretization parameters, multiply the discrete step length matrix and the state transfer matrix, and perform index processing on each element in the multiplied matrix to obtain the discrete state transfer matrix; (1.3) Determine the input matrix corresponding to the current moment according to the data to be processed, and multiply the input matrix, discrete step length matrix and input control matrix to obtain a discrete input control matrix; (1.4) Obtain the state matrix corresponding to the previous moment, multiply the state matrix and the discrete state transfer matrix, and add them to the discrete input control matrix to obtain the target state matrix corresponding to the current moment; (1.5) Multiply the target state matrix and the output control matrix to obtain the output matrix corresponding to the current moment; (1.6) Determine the data processing result corresponding to the data to be processed based on the output matrix corresponding to the current moment.

[0093] Among them, the state transfer matrix, input control matrix and output control matrix corresponding to the second operator are determined. The state transfer matrix corresponding to the second operator is represented as A(d_in,n), the input control matrix is ​​represented as B (b,l,n) and the output control matrix is ​​represented as C(b,l,n). These matrices are operator units inside the second operator.

[0094] The original data corresponding to the data to be processed can be image data, b represents the size of the image bachsize, l represents the number of image blocks, d_in represents the feature dimension of each matrix, and n is a hyperparameter.

[0095] The state transition matrix describes how the state changes over time. The input control matrix describes how external input affects the change of the state. The output control matrix describes how the state matrix processed internally by the second operator is mapped to the output space.

[0096] Then, the discrete step length matrix is ​​determined according to the discretization parameters, which can be expressed as delta(b,l,d_in). The discrete step length matrix is ​​used to discretize the data to be processed. Then, the discrete step length matrix and the state transfer matrix are multiplied, and each element in the multiplied matrix is ​​indexed to obtain the discrete state transfer matrix, which can be expressed as deltaA(b,l,d_in,n).

[0097] Then, the input matrix corresponding to the current moment is determined according to the data to be processed, and the input matrix, discrete step length matrix and input control matrix are multiplied to obtain the discrete input control matrix. The input matrix is ​​represented as u(b,l,d_in), and then the dimensions of the three matrices, input matrix u(b,l,d_in), input control matrix B (b,l,n) and discrete state transfer matrix deltaA (b,l,d_in,n), are multiplied to obtain the discrete input control matrix deltaB_u (b,l,d_in,n).

[0098] By obtaining the state matrix corresponding to the previous moment, the state matrix is ​​the matrix generated by the second operator when processing the input matrix of the previous moment in the data to be processed, which can be expressed as X(t-1), where t-1 represents the previous moment. Then, the state matrix is ​​multiplied by the discrete state transfer matrix and added to the discrete input control matrix to obtain the target state matrix corresponding to the current moment, which can be specifically expressed as: X(t) = deltaAx(t-1) + deltaB_u(t). Among them, the target state matrix is ​​X(t), t is the current moment, deltaA is the discrete state transfer matrix, the discrete input control matrix is ​​deltaB, and the input matrix is ​​u(t).

[0099] Then multiply the target state matrix and the output control matrix to get the output matrix corresponding to the current moment. That is, perform a dot product operation on the target state matrix X(t) (or expressed as X(b,d_in,n)) and the output control matrix C(b,l,n) to get the output matrix corresponding to the current moment, expressed as y(b,n).

[0100] Finally, the data processing result corresponding to the data to be processed is determined according to the output matrix corresponding to the current moment. Specifically, the output matrix corresponding to the input matrix at other moments in the data to be processed is determined, and the output matrix corresponding to the current moment and the output matrix corresponding to other moments are stacked to obtain the first target output matrix corresponding to the data to be processed. The first target output matrix is ​​the data processing result corresponding to the data to be processed. The first target output matrix is ​​represented as y(b,l,n).

[0101] From the above, it can be seen that in the present application, by discretizing the data to be processed, it is avoided to perform calculations on the entire data to be processed like the first operator, which involves more processing steps. This can greatly reduce the amount of calculation. Therefore, the second operator has fewer operator units than the first operator, and thus has less calculation and faster calculation speed. The second operator can be applied to the central processing unit for loading, which can reduce the amount of calculation of the graphics processing unit, thereby improving the data processing speed and processing efficiency of the pre-trained model for the data to be processed as a whole.

[0102] In some implementations, performing data processing on the data to be processed and the discretization parameter according to the second operator to obtain a data processing result includes: (2.1) Convert the data format of the second operator and other operators corresponding to the pre-trained model to obtain the target pre-trained model in the target data format; (2.2) determining a third operator corresponding to the second operator in the target pre-trained model; (2.3) The data to be processed and the discretization parameters are processed according to the third operator to obtain a data processing result.

[0103] The data format of the pre-trained model can be converted, for example, into a target pre-trained model in the ONNX data format. During the conversion process, the data format conversion of the second operator and other operators is involved.

[0104] The pre-trained model can be a VMamba model. The VMamba model is trained on a graphics processor. If you want to migrate it to other hardware processing units for reasoning (such as CPU), you often need to convert it into ONNX format as middleware, that is, the target pre-trained model, and then perform data processing on other hardware processing units.

[0105] Then, the third operator corresponding to the second operator is determined in the target pre-trained model. The third operator is actually the corresponding operator in the ONNX data format, and its function is the same as the second operator. Finally, the data to be processed and the discretized parameters are processed according to the third operator to obtain the data processing result.

[0106] In some implementations, performing data processing on the data to be processed and the discretization parameter according to the third operator to obtain a data processing result includes: (2.3.1) Determine the discrete step length matrix, state transfer matrix, input control matrix and output control matrix corresponding to the third operator; (2.3.2) Perform block accumulation processing on the input matrix corresponding to the data to be processed to obtain a first processing matrix, and perform one-dimensional convolution processing on the first processing matrix and the state transfer matrix to obtain a target state matrix; (2.3.3) Multiply the input matrix, discrete step size matrix and input control matrix to obtain the first target input matrix; (2.3.4) performing block accumulation processing on the first target input matrix to obtain a second target input matrix, and determining a to-be-processed output matrix according to the second target input matrix, the target state matrix, and a preset state space matrix whose elements are all zero; (2.3.5) Perform one-dimensional convolution processing on the output matrix to be processed and the output control matrix to obtain a second target output matrix, which is the data processing result corresponding to the data to be processed.

[0107] Among them, the discrete step matrix, state transfer matrix, input control matrix and output control matrix corresponding to the third operator can be understood as the corresponding operator unit inside the third operator. The discrete step matrix is ​​represented as dts, the state transfer matrix is ​​represented as As, the input control matrix is ​​represented as Bs, and the output control matrix is ​​represented as Cs. The input matrix corresponding to the data to be processed can also be determined, and the input matrix is ​​represented as Us.

[0108] These matrix data can be converted to float data type to ensure calculation accuracy.

[0109] The input matrix corresponding to the processed data is then processed in blocks and accumulated to obtain a first processing matrix, and the first processing matrix and the state transfer matrix are processed in one dimension to obtain a target state matrix. Specifically, the input matrix corresponding to the processed data can be processed in blocks to obtain multiple sub-matrices, and then the cumulative sum matrix corresponding to each sub-matrix is ​​determined according to the time step of each sub-matrix, and the cumulative sum matrix corresponding to each sub-matrix is ​​processed by matrix stacking to obtain the first processing matrix.

[0110] For example, within the total time step L corresponding to the input matrix, the input matrix is ​​processed in blocks with cumsum_chunk_size as the step size to obtain multiple sub-matrices. cumsum_chunk_size is set to 6 to improve the computing efficiency on the central processing unit. The specific representation of the sub-matrix corresponding to each time step is x_chunk =dts[cumsum_chunk_size*i,cumsum_chunk_size*(i+1)]. Then the cumulative sum matrix corresponding to the sub-matrix of each time step is calculated, represented as item_chunk. If the sub-matrix x_chunk is the first block, the cumulative sum is calculated directly; otherwise, it is accumulated on the basis of the cumulative sum corresponding to the previous time step. The cumulative sum matrices item_chunk corresponding to each time step are stacked together to obtain the first processing matrix ts.

[0111] Then, the first processing matrix and the state transfer matrix are subjected to one-dimensional convolution to obtain the target state matrix. For example, the F.conv1d function of the PyTorch computing library is used to perform a one-dimensional convolution operation on the first processing matrix ts, where the first processing matrix ts is processed as (chunk_L*B, G*D, 1), and the state transfer matrix As is processed as (-1, 1, 1), and the number of convolution groups groups is set to G*D. This is actually a one-dimensional convolution of the first processing matrix ts and the state transfer matrix As to obtain the convolution result, and then each element of the convolution result is indexed to obtain the state matrix Ats to be processed. Then, the value of the last time step of the state matrix Ats to be processed is normalized to avoid zero division errors. The specific operation is: scale = Ats[-1].detach() + 1e-30, and then the normalized target state matrix rAts is obtained by dividing the state matrix Ats to be processed by this scale value.

[0112] After that, the input matrix, discrete step matrix and input control matrix are multiplied to obtain the first target input matrix. For example, the input matrix Us and the discrete step matrix dts are element-by-element multiplied to obtain the intermediate discrete input matrix duts, and then the torch.matmul function is used to perform matrix multiplication of the intermediate discrete input matrix duts and the input control matrix Bs to obtain the first target input matrix dtBus.

[0113] The first target input matrix is ​​then subjected to block accumulation processing to obtain a second target input matrix, and an output matrix to be processed is determined according to the second target input matrix, the target state matrix, and a preset state space matrix whose elements are all zero.

[0114] Specifically, determine the submatrix corresponding to each time step in the first target input matrix, accumulate the submatrix corresponding to each time step, and obtain the second target input matrix. For example, take out the submatrix corresponding to each time step from the first target input matrix dtBus, and the submatrix is ​​represented as x_chunk =dts[cumsum_chunk_size*i, cumsum_chunk_size*(i+1)]. If the submatrix x_chunk is the submatrix corresponding to the first time step, directly calculate the accumulation; if not, continue to accumulate based on the accumulation result of the previous submatrix to obtain the second target input matrix dtBus_cumsum.

[0115] Then, the target state matrix rAts is element-wise multiplied with the second target input matrix dtBus_cumsum to obtain the second processing matrix hs_tmp, and the second processing matrix hs_tmp is multiplied with a preset state space matrix initialized to all zeros (such as the state space variable matrix hprefix matrix) to obtain the third processing matrix. The state matrix to be processed Ats is multiplied with the preset state space matrix to obtain the fourth processing matrix, and finally the third processing matrix and the fourth processing matrix are added to obtain the output matrix to be processed hs.

[0116] Finally, the output matrix to be processed and the output control matrix are subjected to one-dimensional convolution to obtain the second target output matrix, which is the data processing result corresponding to the data to be processed. For example, the F.conv1d function is used to perform a one-dimensional convolution operation on the output control matrix Cs and the output matrix to be processed hs. The output control matrix Cs is processed as (1, chunk_L*B*G*N, 1), and the output matrix to be processed hs is processed as (chunk_L*B*G*D, N, 1). The number of convolution groups groups is set to chunk_L*B*G, thereby obtaining the first output matrix ys.

[0117] If the third operator has a corresponding instruction matrix Ds, the instruction matrix Ds is multiplied by the input matrix us and then added to the first output matrix ys to obtain the final second target output matrix oys. If the third operator does not have an instruction matrix Ds, the first output matrix ys is used as the second target output matrix oys.

[0118] From the above, it can be seen that by dividing the input matrix into blocks and then stacking them into an integral matrix, and then calculating the integral matrix, many calculation steps can be reduced. Therefore, compared with the first operator, the third operator has fewer calculation units, but it can achieve the same function as the first operator, which improves the processing speed of the data to be processed, thereby improving the data processing speed and processing efficiency of the pre-trained model.

[0119] In some implementations, in the target pre-trained model, three optimization methods, namely constant folding, dead code elimination, and operator fusion, can also be used to effectively reduce the computational complexity and runtime resource overhead when the target pre-trained model processes data, thereby improving the overall execution efficiency.

[0120] For example, in the constant folding optimization method, during the export of the target pre-trained model, constant expressions such as torch.tensor([2, 3]) * torch.tensor([4, 5]) in the model are calculated in advance as [8, 15] to avoid repeated calculations at runtime. In addition, in the FLOPs calculation function, for the expression flops = 9* B * L * D * N of static parameters (such as B, L, D, N), if the parameters are known (such as B=1, L=256, etc.), they can be directly folded into constant values, thereby reducing the amount of calculation in the function call. In addition, for overlapping matrix operations (such as bias addition and activation functions), they are folded in advance by optimizing logic before inference to further improve performance.

[0121] For another example, this application uses dead code elimination technology to eliminate redundant logic in the code corresponding to the target pre-trained model to reduce resource waste. For example, in the main function, there are multiple branch logics to handle different tasks, such as args.inference_time_pytorch for calculating the inference time of the pre-trained model and args.inference_time_onnx for calculating the inference time of the target pre-trained model; if some branches are not enabled or will not be triggered (such as args.inference_time_simply_onnx for calculating the inference time of a special operator of the target pre-trained model is not set), the relevant logic can be directly removed through static analysis. There are also some imported modules such as selective_scan_cuda_oflex or selective_scan_cuda_core that may not be actually called. The import logic of these unused modules can be deleted through analysis during compilation. In addition, in some breakpoint check logic (such as pdb.set_trace()), it can be removed by dead code elimination in the production environment to avoid affecting the inference performance. By detecting and removing unused parameters, branches, and debugging code, this optimization method significantly reduces the invalid overhead at runtime.

[0122] For another example, this application uses operator fusion technology to merge multiple continuous operations in the target pre-trained model into a single efficient operator to further improve the reasoning efficiency. For example, some continuous tensor operations, such as flatten, transpose, and flip, the original code implements these operations in multiple steps, and after optimization, they can be merged into a custom operator custom_operator to reduce the operation overhead on the CPU. Similarly, matrix operations such as ys[:,0:2]+ys[:,2:4].flip(dims=[-1]) and transpose can be fused through kernels to reduce the storage and operation times of intermediate tensors. During the reasoning process of the target pre-trained model, the post-processing logic after the reasoning output (such as normalization or activation function) can be directly embedded in the target pre-trained model, and the integration of reasoning and post-processing can be achieved through the operator fusion of the target pre-trained model. In addition, in some matrix calculations, multiple torch.einsum operations, such as torch.einsum('bdl,bdnl,bdl->bdln') and torch.einsum('bdl,dn->bdln'), can be fused by optimizing the computational graph. This operator fusion optimization method significantly reduces the number of operator calls and the memory usage of intermediate tensors at runtime, which is of great value for the inference deployment of the target pre-trained model.

[0123] See also Figure 5 , Figure 5 : is a comparison chart of the number of operators corresponding to the pre-trained model and the target pre-trained model provided in the embodiment of the present application. Figure 5 It can be seen that in the present application, after optimizing the first operator and other operators in the pre-trained model, the number of operators of the target pre-trained model is significantly smaller, and the computational inference speed of the target pre-trained model is higher.

[0124] In an embodiment of the present application, a pre-trained model and model data corresponding to the pre-trained model are obtained, where the pre-trained model is a model under a state-space model structure; a first operator in the pre-trained model that needs to be accelerated in data processing is determined according to the model data; the first operator is decoupled to obtain a second operator corresponding to the first operator, and the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; data to be processed corresponding to the second operator is obtained, and discretization parameters corresponding to the data to be processed are determined; data processing is performed on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0125] In this way, the first operator that needs data processing acceleration is determined by the model data of the pre-trained model under the state-space model structure, and then the first operator is decoupled to obtain the second operator. The second operator can be applied to other hardware processing units relative to the first operator, so that the computing power of other hardware processing units can be used to accelerate the operation of the second operator, and the data to be processed by the second operator is discretized by using discretization parameters. In this way, the second operator avoids directly processing the data to be processed as a whole, but processes the discretized data, which can reduce the second operator's influence on the overall data to be processed. Complex calculations are performed, thereby improving the processing speed of the second operator for the data to be processed. Therefore, compared with the limited data acceleration processing effect of the model acceleration technology in the related technology on the model under the state-space model structure, the present application can use the first operator to decouple to obtain the second operator, and then load the second operator through other hardware units, and discretize the data to be processed by the second operator, which can greatly improve the data processing speed of the second operator, thereby greatly improving the data processing speed of the pre-trained model of the entire state-space model. Therefore, the data processing acceleration effect of the pre-trained model of the state-space model is more obvious than the related technology.

[0126] See also Figure 6 , Figure 6 : is another flow chart of the model data processing method provided in the embodiment of the present application. The model data processing method may include the following steps: Step 501: Obtain a pre-trained model and model data corresponding to the pre-trained model, where the pre-trained model is a model under a state space model structure; Step 502: Determine the first operator in the pre-trained model that needs to be accelerated in data processing according to the model data; Step 503: Determine the operation logic corresponding to each sub-operator in the first operator; Step 504: modify each sub-operator according to the preset calculation library and the operation logic corresponding to each sub-operator to generate a target sub-operator corresponding to each sub-operator; Step 505: Generate a second operator corresponding to the first operator according to the target sub-operator corresponding to each sub-operator; Step 506: Perform data format conversion on the second operator and other operators corresponding to the pre-trained model to obtain a target pre-trained model in a target data format; Step 507: Determine a third operator corresponding to the second operator in the target pre-trained model; Step 508: Perform data processing on the data to be processed and the discretization parameters according to the third operator to obtain a data processing result; Step 509, determining the discrete step length matrix, state transfer matrix, input control matrix and output control matrix corresponding to the third operator; Step 510: Perform block accumulation processing on the input matrix corresponding to the data to be processed to obtain a first processing matrix, and perform one-dimensional convolution processing on the first processing matrix and the state transfer matrix to obtain a target state matrix; Step 511, multiplying the input matrix, the discrete step length matrix and the input control matrix to obtain a first target input matrix; Step 512: performing block accumulation processing on the first target input matrix to obtain a second target input matrix, and determining a to-be-processed output matrix according to the second target input matrix, the target state matrix, and a preset state space matrix whose elements are all zero; Step 513: Perform one-dimensional convolution processing on the output matrix to be processed and the source control matrix to obtain a second target output matrix, where the second target output matrix is ​​the data processing result corresponding to the data to be processed.

[0127] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above model data processing method, which will not be repeated here.

[0128] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of a model data processing device provided in an embodiment of the present application. The model data processing device can execute the above-mentioned model data processing method.

[0129] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0130] The model data processing device 600 comprises: A first acquisition module 610 is used to acquire a pre-trained model and model data corresponding to the pre-trained model, where the pre-trained model is a model under a state space model structure; A determination module 620 is used to determine a first operator in the pre-trained model that needs to be accelerated in data processing according to the model data; A decoupling module 630 is used to perform decoupling processing on the first operator to obtain a second operator corresponding to the first operator, where the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; A second acquisition module 640 is used to acquire the data to be processed corresponding to the second operator and determine the discretization parameters corresponding to the data to be processed; The processing module 650 is used to perform data processing on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0131] In some implementations, the decoupling module 630 is configured to: Determine the operation logic corresponding to each sub-operator in the first operator; Determine the type of hardware processing unit corresponding to each sub-operator according to the operation logic; Each sub-operator is decoupled according to the type of hardware processing unit to obtain a second operator.

[0132] In some implementations, the decoupling module 630 is configured to: Determine the operation logic corresponding to each sub-operator in the first operator; Modify each sub-operator according to the preset calculation library and the operation logic corresponding to each sub-operator to generate the target sub-operator corresponding to each sub-operator; A second operator corresponding to the first operator is generated according to the target sub-operator corresponding to each sub-operator.

[0133] In some embodiments, the processing module 650 is configured to: Determine a state transfer matrix, an input control matrix, and an output control matrix corresponding to the second operator; Determine a discrete step length matrix according to the discretization parameters, multiply the discrete step length matrix and the state transfer matrix, and perform index processing on each element in the multiplied matrix to obtain a discrete state transfer matrix; Determine the input matrix corresponding to the current moment according to the data to be processed, and multiply the input matrix, the discrete step length matrix and the input control matrix to obtain a discrete input control matrix; Get the state matrix corresponding to the previous moment, multiply the state matrix and the discrete state transfer matrix, and add them to the discrete input control matrix to get the target state matrix corresponding to the current moment; Multiply the target state matrix and the output control matrix to obtain the output matrix corresponding to the current moment; The data processing result corresponding to the data to be processed is determined according to the output matrix corresponding to the current moment.

[0134] In some embodiments, the processing module 650 is configured to: Determine the output matrix corresponding to the input matrix at other times in the data to be processed; The output matrix corresponding to the current moment and the output matrices corresponding to other moments are stacked to obtain a first target output matrix corresponding to the data to be processed. The first target output matrix is ​​the data processing result corresponding to the data to be processed.

[0135] In some embodiments, the processing module 650 is configured to: Performing data format conversion on the second operator and other operators corresponding to the pre-trained model to obtain a target pre-trained model in a target data format; Determining a third operator corresponding to the second operator in the target pre-trained model; The data to be processed and the discretization parameters are processed according to the third operator to obtain a data processing result.

[0136] In some embodiments, the processing module 650 is configured to: Determine a discrete step size matrix, a state transfer matrix, an input control matrix, and an output control matrix corresponding to the third operator; Perform block accumulation processing on the input matrix corresponding to the data to be processed to obtain a first processing matrix, and perform one-dimensional convolution processing on the first processing matrix and the state transfer matrix to obtain a target state matrix; Multiplying the input matrix, the discrete step length matrix and the input control matrix to obtain a first target input matrix; The first target input matrix is ​​processed by block accumulation to obtain a second target input matrix, and an output matrix to be processed is determined according to the second target input matrix, the target state matrix and a preset state space matrix whose elements are all zero; One-dimensional convolution processing is performed on the output matrix to be processed and the output control matrix to obtain a second target output matrix, which is a data processing result corresponding to the data to be processed.

[0137] In some embodiments, the processing module 650 is configured to: The input matrix corresponding to the data to be processed is processed in blocks to obtain multiple sub-matrices; The cumulative sum matrix corresponding to each submatrix is ​​determined according to the time step of each submatrix, and the cumulative sum matrix corresponding to each submatrix is ​​subjected to matrix stacking processing to obtain a first processing matrix.

[0138] In some embodiments, the processing module 650 is configured to: Determine the submatrix corresponding to each time step in the first target input matrix; The sub-matrices corresponding to each time step are accumulated to obtain the second target input matrix.

[0139] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the above model data processing method, which will not be repeated here.

[0140] In an embodiment of the present application, a first acquisition module 610 acquires a pre-trained model and model data corresponding to the pre-trained model, where the pre-trained model is a model under a state-space model structure; a determination module 620 determines a first operator in the pre-trained model that requires data processing acceleration based on the model data; a decoupling module 630 performs decoupling processing on the first operator to obtain a second operator corresponding to the first operator, and a hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; a second acquisition module 640 acquires the data to be processed corresponding to the second operator, and determines the discretization parameters corresponding to the data to be processed; and a processing module 650 performs data processing on the data to be processed and the discretization parameters based on the second operator to obtain a data processing result.

[0141] In this way, the first operator that needs data processing acceleration is determined by the model data of the pre-trained model under the state-space model structure, and then the first operator is decoupled to obtain the second operator. The second operator can be applied to other hardware processing units relative to the first operator, so that the computing power of other hardware processing units can be used to accelerate the operation of the second operator, and the data to be processed by the second operator is discretized by using discretization parameters. In this way, the second operator avoids directly processing the data to be processed as a whole, but processes the discretized data, which can reduce the second operator's influence on the overall data to be processed. Complex calculations are performed, thereby improving the processing speed of the second operator for the data to be processed. Therefore, compared with the limited data acceleration processing effect of the model acceleration technology in the related technology on the model under the state-space model structure, the present application can use the first operator to decouple to obtain the second operator, and then load the second operator through other hardware units, and discretize the data to be processed by the second operator, which can greatly improve the data processing speed of the second operator, thereby greatly improving the data processing speed of the pre-trained model of the entire state-space model. Therefore, the data processing acceleration effect of the pre-trained model of the state-space model is more obvious than the related technology.

[0142] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned model data processing method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0143] See also Figure 8 , Figure 8 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes: The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the model data processing method of the embodiment of this application; Input / output interface 703, used to implement information input and output; Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 705 that transmits information between various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704); The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0144] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned model data processing method is implemented.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The model data processing method, model data processing device, computer equipment and storage medium provided in the embodiments of the present application are as follows: in the embodiments of the present application, a pre-trained model and model data corresponding to the pre-trained model are obtained, and the pre-trained model is a model under a state-space model structure; a first operator in the pre-trained model that needs to be accelerated in data processing is determined according to the model data; the first operator is decoupled to obtain a second operator corresponding to the first operator, and the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; the data to be processed corresponding to the second operator is obtained, and the discretization parameters corresponding to the data to be processed are determined; data processing is performed on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

[0147] In this way, the first operator that needs data processing acceleration is determined by the model data of the pre-trained model under the state-space model structure, and then the first operator is decoupled to obtain the second operator. The second operator can be applied to other hardware processing units relative to the first operator, so that the computing power of other hardware processing units can be used to accelerate the operation of the second operator, and the data to be processed by the second operator is discretized by using discretization parameters. In this way, the second operator avoids directly processing the data to be processed as a whole, but processes the discretized data, which can reduce the second operator's influence on the overall data to be processed. Complex calculations are performed, thereby improving the processing speed of the second operator for the data to be processed. Therefore, compared with the limited data acceleration processing effect of the model acceleration technology in the related technology on the model under the state-space model structure, the present application can use the first operator to decouple to obtain the second operator, and then load the second operator through other hardware units, and discretize the data to be processed by the second operator, which can greatly improve the data processing speed of the second operator, thereby greatly improving the data processing speed of the pre-trained model of the entire state-space model. Therefore, the data processing acceleration effect of the pre-trained model of the state-space model is more obvious than the related technology.

[0148] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0150] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0152] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0153] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0158] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model data processing method, characterized in that: include: Acquire a pre-trained model and model data corresponding to the pre-trained model, wherein the pre-trained model is a model under a state space model structure; Determine, according to the model data, a first operator in the pre-trained model that needs to be accelerated in data processing; Performing decoupling processing on the first operator to obtain a second operator corresponding to the first operator, wherein a hardware processing unit corresponding to the second operator is different from a hardware processing unit corresponding to the first operator; Acquire the data to be processed corresponding to the second operator, and determine the discretization parameters corresponding to the data to be processed; The data to be processed and the discretization parameters are processed according to the second operator to obtain a data processing result.

2. The model data processing method according to claim 1, characterized in that: The decoupling the first operator to obtain a second operator corresponding to the first operator includes: Determine the operation logic corresponding to each sub-operator in the first operator; Determine the type of hardware processing unit corresponding to each sub-operator according to the operation logic; Each of the sub-operators is decoupled according to the type of the hardware processing unit to obtain a second operator.

3. The model data processing method according to claim 1, characterized in that: The decoupling the first operator to obtain a second operator corresponding to the first operator includes: Determine the operation logic corresponding to each sub-operator in the first operator; Modify each sub-operator according to the preset calculation library and the operation logic corresponding to each sub-operator to generate a target sub-operator corresponding to each sub-operator; A second operator corresponding to the first operator is generated according to the target sub-operator corresponding to each sub-operator.

4. The model data processing method according to claim 3, characterized in that: The step of performing data processing on the data to be processed and the discretization parameter according to the second operator to obtain a data processing result includes: Determine a state transfer matrix, an input control matrix, and an output control matrix corresponding to the second operator; Determine a discrete step length matrix according to the discretization parameter, multiply the discrete step length matrix and the state transfer matrix, and perform exponential processing on each element in the multiplied matrix to obtain a discrete state transfer matrix; Determine an input matrix corresponding to the current moment according to the data to be processed, and multiply the input matrix, the discrete step size matrix and the input control matrix to obtain a discrete input control matrix; Obtaining a state matrix corresponding to the previous moment, and multiplying the state matrix and the discrete state transfer matrix and adding the matrix to the discrete input control matrix to obtain a target state matrix corresponding to the current moment; Multiplying the target state matrix and the output control matrix to obtain an output matrix corresponding to the current moment; The data processing result corresponding to the data to be processed is determined according to the output matrix corresponding to the current moment.

5. The model data processing method according to claim 4, characterized in that: The step of determining the data processing result corresponding to the data to be processed according to the output matrix corresponding to the current moment includes: Determine the output matrix corresponding to the input matrix at other times in the data to be processed; The output matrix corresponding to the current moment and the output matrix corresponding to the other moments are stacked to obtain a first target output matrix corresponding to the data to be processed, where the first target output matrix is ​​a data processing result corresponding to the data to be processed.

6. The model data processing method according to claim 3, characterized in that: The step of performing data processing on the data to be processed and the discretization parameter according to the second operator to obtain a data processing result includes: Performing data format conversion on the second operator and other operators corresponding to the pre-trained model to obtain a target pre-trained model in a target data format; Determining a third operator corresponding to the second operator in the target pre-trained model; The data to be processed and the discretization parameters are processed according to the third operator to obtain a data processing result.

7. The model data processing method according to claim 6, characterized in that: The step of performing data processing on the data to be processed and the discretization parameter according to the third operator to obtain a data processing result includes: Determine a discrete step size matrix, a state transfer matrix, an input control matrix and an output control matrix corresponding to the third operator; Performing block accumulation processing on the input matrix corresponding to the data to be processed to obtain a first processing matrix, and performing one-dimensional convolution processing on the first processing matrix and the state transfer matrix to obtain a target state matrix; Multiplying the input matrix, the discrete step size matrix and the input control matrix to obtain a first target input matrix; Performing block accumulation processing on the first target input matrix to obtain a second target input matrix, and determining an output matrix to be processed according to the second target input matrix, the target state matrix and a preset state space matrix whose elements are all zero; One-dimensional convolution processing is performed on the output matrix to be processed and the output control matrix to obtain a second target output matrix, where the second target output matrix is ​​a data processing result corresponding to the data to be processed.

8. The model data processing method according to claim 7, characterized in that: The step of performing block-by-block accumulation processing on the input matrix corresponding to the data to be processed to obtain a first processing matrix includes: The input matrix corresponding to the data to be processed is processed in blocks to obtain multiple sub-matrices; The cumulative sum matrix corresponding to each sub-matrix is ​​determined according to the time step of each sub-matrix, and the cumulative sum matrix corresponding to each sub-matrix is ​​subjected to matrix stacking processing to obtain a first processing matrix.

9. The model data processing method according to claim 7, characterized in that: The step of performing block accumulation processing on the first target input matrix to obtain a second target input matrix includes: Determine a submatrix corresponding to each time step in the first target input matrix; The sub-matrices corresponding to each time step are accumulated to obtain a second target input matrix.

10. A model data processing device, characterized in that: include: A first acquisition module is used to acquire a pre-trained model and model data corresponding to the pre-trained model, wherein the pre-trained model is a model under a state space model structure; A determination module, configured to determine a first operator in the pre-trained model that needs to be accelerated in data processing according to the model data; a decoupling module, configured to perform decoupling processing on the first operator to obtain a second operator corresponding to the first operator, wherein the hardware processing unit corresponding to the second operator is different from the hardware processing unit corresponding to the first operator; A second acquisition module, used to acquire the data to be processed corresponding to the second operator, and determine the discretization parameters corresponding to the data to be processed; A processing module is used to perform data processing on the data to be processed and the discretization parameters according to the second operator to obtain a data processing result.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the model data processing method according to any one of claims 1 to 9.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the model data processing method according to any one of claims 1 to 9 is implemented.

Images