Data scheduling method, system and equipment

Abstract

The embodiment of the invention discloses a data scheduling method, system and equipment. The method comprises the following steps: acquiring a user request; calling a corresponding target model object according to a user request; and scheduling data of the sub-function module of the target model object to a corresponding target hardware unit according to a preset corresponding relationship between execution priorities of each sub-function module of the model object and the hardware unit, so as to process a user request by using available resources of the target hardware unit.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a data scheduling method, system and device. Background Art

[0002] As the influence of artificial intelligence (AI) big models in the industry gradually increases, more and more local big models are exposed to the vision of electronic equipment providers. However, big models have more parameters and complex model structures, and they need to occupy more electronic equipment hardware resources during model operation. In the process of calling big models for task processing, how to reasonably use the hardware resources of electronic equipment, improve the operation efficiency of big models, and increase the task response speed is an urgent problem to be solved in the process of using big models for task processing. Summary of the invention

[0003] The technical solution of this application is implemented as follows:

[0004] The present application provides a data scheduling method, which is applied to a first device. The method includes:

[0005] Get user request;

[0006] According to the user's request, call the corresponding target model object;

[0007] According to the preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware units, the data of the sub-function modules of the target model object are dispatched to the corresponding target hardware units, so as to utilize the available resources of the target hardware units to process the user request;

[0008] Among them, each sub-functional module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset corresponding relationship is determined based on the execution speed of each sub-functional module on different hardware units.

[0009] The embodiment of the present application provides a data scheduling system, including: an application module, a model service module and multiple hardware units; wherein:

[0010] The application module is used to obtain user requests and call the corresponding target model object according to the user request;

[0011] A model service module, for dispatching data of the sub-function modules of the target model object to the corresponding target hardware unit according to the preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware unit, so as to process the user request by utilizing the available resources of the target hardware unit;

[0012] A hardware unit, used to run the data of the sub-function modules of the target model object;

[0013] Among them, each sub-functional module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset corresponding relationship is determined based on the execution speed of each sub-functional module on different hardware units.

[0014] The present application provides a data scheduling device, including:

[0015] A memory for storing executable data scheduling instructions;

[0016] The processor is used to implement the data scheduling method provided in the embodiment of the present application when executing the executable data scheduling instruction stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are configured to execute the steps of the above-mentioned data scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a data scheduling method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the architecture of a local multimodal multi-model service system provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the structure of a data scheduling system provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the composition structure of a data scheduling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0024] In the following description, reference is made to “some embodiments\other embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments\other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0025] In the following description, the terms "first\second\" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] Based on the problems existing in the related art, the embodiment of the present application provides a data scheduling method, which can be applied to a first device, which can be a desktop computer, a laptop computer, a local server, a cloud server, etc. Figure 1 FIG. 1 is a flow chart of a data scheduling method provided in an embodiment of the present application, and the method comprises the following steps:

[0028] S101: Obtain user request.

[0029] In some embodiments, the user request may be triggered by a user clicking or touching a control or content in a human-computer interaction interface corresponding to the first device. The human-computer interaction interface may be an interface of an application software running on the first device, such as a chat interface of a chat software, a search interface in a map navigation software, etc.

[0030] In some embodiments, within a preset time period, the user requests obtained may include multiple, different user requests may come from the same application software, for example, multiple user requests may all come from chat software; different user requests may also come from different application software, for example, multiple user requests may come from chat software, map navigation software, etc.

[0031] S102: Call the corresponding target model object according to the user request.

[0032] It should be noted that the target model object can be a large model or other artificial intelligence model. For example, the target model object can include a large language model, such as a bidirectional transformer model (Bidirectional Encoder Representations from Transformers, BERT), a generative pre-trained transformation model (Generative Pre-Trained Transformer, GPT), etc. The following description takes the target model object as a large model.

[0033] In some embodiments, after obtaining the user request, the user request can be analyzed to determine the processing task corresponding to the user request and the task type corresponding to the processing task, and then determine the target model object that can process the task corresponding to the task type, and call the target model object.

[0034] Exemplarily, if it is determined that the task type corresponding to the user request is a raw image task, the target model object can be determined to be a large model of the raw image class; if it is determined that the task type corresponding to the user request is a raw text task, the target model object can be determined to be a large model of the raw text class.

[0035] In some embodiments, the determined target model objects may include multiple ones. In this case, any one of the multiple target models may be called; or the target model object with the least number of parameters may be called according to the number of parameters of the model.

[0036] S103. According to the preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware units, the data of the sub-function modules of the target model object are dispatched to the corresponding target hardware units to process the user request using the available resources of the target hardware units.

[0037] In some embodiments, the various sub-functional modules of the target model object are used to perform inference operations on the data requested by the corresponding user. The sub-functional modules of the model object may be sub-modules with complete functions of the model object. For example, if the model object is a large language model, its corresponding sub-functional modules may include a natural language encoding sub-module, a text prediction sub-module, etc.

[0038] In some embodiments, the sub-functional modules of the model object can be divided according to the functions to be implemented, or according to the number of network layers of the model. For example, if the model object is a deep neural network including 100 network layers, the model object can be divided into 5 sub-functional modules with the model parameter weight file corresponding to 20 layers as one sub-functional module.

[0039] In some embodiments, the hardware unit may be a hardware processing unit of the first device, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), etc.

[0040] In some embodiments, the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit may be pre-established, and the preset correspondence may be determined based on the execution speed of each sub-function module on different hardware units. Through the preset correspondence, the order of the execution speeds of different sub-function modules on different hardware units of the first device may be determined. For example, if the hardware unit includes a CPU, a GPU, and an NPU, in the preset relationship, the hardware unit execution priority of the natural language encoding submodule of the large language model may be from high to low: CPU, NPU, GPU, that is, the execution speed of the natural language encoding submodule on the CPU is the fastest, the execution speed on the NPU is the second fastest, and the execution speed on the GPU is the slowest.

[0041] In some embodiments, the target model object includes multiple sub-functional modules, and the execution priorities of the hardware units corresponding to different sub-functional modules may be different. The hardware units with the highest execution priority corresponding to different sub-functional modules can be determined based on the preset relationship, and then the data of each sub-functional module can be scheduled to the corresponding hardware unit with the highest execution priority, wherein the data of the sub-functional module can include a weight file of the model parameters corresponding to the sub-functional module.

[0042] In an embodiment of the present application, a user request is obtained; according to the user request, a corresponding target model object is called; according to the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, the data of the sub-function module of the target model object is dispatched to the corresponding target hardware unit, so as to utilize the available resources of the target hardware unit to process the user request. In this way, through the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, the target hardware unit that can make each sub-function module of the target model object execute the fastest can be determined from the multiple hardware units of the first device, and the data of the sub-function module can be further dispatched to the corresponding target hardware computing unit to process the user request, which can improve the operating efficiency of the target model object and the response speed to the user request.

[0043] In some embodiments of the present application, in the process of creating a preset correspondence between each sub-functional module of a model object and the execution priority of a hardware unit, a first task type corresponding to a reference sub-functional module can be determined; data of the reference sub-functional module can be scheduled to different hardware units to obtain reference execution speeds of user requests for the first task type for different hardware units; and a preset correspondence can be established based on each reference execution speed.

[0044] In some embodiments, the reference sub-function module is any one of the sub-function modules. The reference sub-function module can be any one of the sub-function modules corresponding to the multiple model objects, and the multiple model objects can all be large models.

[0045] In some embodiments, the first task type may be the type of task that the reference sub-function module can handle. For example, if the reference sub-function module can be used to handle the task of generating pictures, then its corresponding first task type may be a picture generation task; if the reference sub-function module can be used to handle the task of generating text, then its corresponding first task type may be a text generation task.

[0046] In some embodiments, the data of the reference sub-function module may include a model parameter weight file of the reference sub-function module, and the model parameter weight file of the reference sub-function module may be scheduled to each hardware unit of the first device respectively to detect the reference execution speed of different hardware units in processing user requests of the first task type.

[0047] In some embodiments, when the hardware unit of the first device includes a CPU, an NPU, and a GPU, the user request data and the data of the reference sub-function module of the same first task type can be sequentially dispatched to the CPU, the NPU, and the GPU, thereby obtaining the CPU 、 The NPU and the GPU process a reference execution speed of a user request of the first task type based on data of the reference sub-functional module.

[0048] In some embodiments, the reference execution speeds may be sorted according to their sizes, thereby obtaining a preset correspondence between the reference sub-functional modules and the execution priorities of the hardware units, wherein the hardware unit with a faster reference execution speed has a higher execution priority.

[0049] In some embodiments, the above-mentioned process of steps S201 to S203 can be executed on each sub-functional module of multiple model objects in turn, so as to obtain a preset correspondence between different sub-functional modules and the execution priorities of each hardware unit. After that, the preset correspondences are integrated to obtain the preset correspondence between the execution priorities of each sub-functional module of multiple model objects and each hardware unit.

[0050] In other embodiments, a second task type corresponding to a complete model object can also be determined; the data of the model object is scheduled to different hardware units to obtain the execution speeds of different hardware units in processing user requests of the second task type based on the model object, and a preset correspondence between the model object and the execution priority of each hardware unit is established according to each execution speed. That is, the established preset correspondence may include the preset correspondence between the model object and the execution priority of each hardware unit, and may also include the preset correspondence between the sub-functional modules of the model object and the execution priority of each hardware unit.

[0051] In some embodiments, the preset correspondence between the execution priority of the sub-functional modules and each hardware unit can be represented in the form of a preset correspondence table. For example, as shown in Table 1, the hardware units of the first device include a CPU, an integrated graphics card (iGPU), a dedicated graphics processing unit (dGPU) and an NPU, a text translation module Translator, a text decoding module Text encoder, a variational auto-encoder decoder (VAE decoder), a super-resolution reconstruction module Super Resolution, and an image segmentation module Segmentation are all sub-functional modules of the model object, and a U-shaped network (UNET) and a large language model (Large Language Model, LLM) is a model object, where 1, 2, 3, and 4 represent the execution priorities of each hardware unit in processing the corresponding sub-functional module or model object task. The larger the number, the lower the execution priority. For example, for the model object LLM, the corresponding CPU execution priority is 4, the iGPU execution priority is 2, the dGPU execution priority is 1, and the NPU execution priority is 3. That is, the priorities of each hardware unit in processing the tasks corresponding to the model object LLM are dGPU, iGPU, NPU, and CPU in descending order.

[0052] Table 1 Preset correspondence table of model object / sub-function module and hardware unit execution priority

[0053]

[0054] In some embodiments, after creating a preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware unit, the preset correspondence can be subsequently updated. For example, when a new model object is added, or a new sub-function module of the model object is added, the data of the new model object or the new sub-function module and the user-requested data of the corresponding task type can be scheduled to different hardware units for testing to obtain the execution speeds of the different hardware units, thereby obtaining the newly added preset correspondence; or, after the hardware unit or the parameters of the hardware unit of the first device are updated, the updated hardware unit can be re-tested for the execution speed of the user-requested data of the sub-function module or the model object corresponding to the task type, thereby obtaining an updated preset correspondence.

[0055] In some embodiments, the above steps S201 to S203 can be performed during the process of testing the hardware unit of the first device before the first device leaves the factory, or can be performed when the first device is used for the first time after leaving the factory, or can be performed during normal use of the first device, that is, the present application does not limit the timing of creating the preset correspondence table.

[0056] It can be understood that by determining the task type corresponding to the reference sub-function module, scheduling the data of the reference sub-function module to different hardware units, determining the execution speed of user requests for the task type by different hardware units, and establishing a preset correspondence between the reference sub-function module and the execution priority of each hardware unit based on each execution speed, it is convenient to directly determine the target hardware unit corresponding to the user request based on the preset correspondence when obtaining the user request of the task type corresponding to the reference sub-function module in the future, thereby improving the response speed to the user request.

[0057] In some embodiments of the present application, in the process of scheduling the data of the sub-functional modules of the target model object to the corresponding target hardware units according to the preset correspondence between the execution priorities of the various sub-functional modules of the model object and the hardware units, the current available resources of each hardware unit can be obtained; based on the available resources and the preset correspondence, the target hardware units corresponding to the specific sub-functional modules are determined; and the data of the specific sub-functional modules and the data requested by the user are scheduled to the corresponding target hardware units.

[0058] In some embodiments, the currently available resources of the hardware unit may include the number of cores, the number of available threads, the available cache space, etc. of the hardware unit. In implementation, the operation data of each hardware unit of the first device may be monitored in real time or periodically to obtain the available resources corresponding to each hardware unit; or the current operation status data of each hardware unit may be detected upon obtaining a user request to determine the currently available resources of the hardware.

[0059] It should be noted that the specific sub-functional modules are one or more sub-functional modules used to perform inference calculations on data requested by the user.

[0060] In some embodiments, in addition to determining the computing power unit that can make each sub-functional module execute at the fastest speed based on the preset correspondence between each sub-functional module and the execution priority of the hardware unit based on the model object, the current resource usage of each computing power unit can also be considered, that is, a comprehensive decision is made based on the current available resources of the hardware computing power unit and the preset correspondence to determine the target hardware unit corresponding to the specific sub-functional module.

[0061] In some embodiments, the execution speed ranking result for a specific sub-functional module among multiple hardware units of the first device can be determined based on the preset correspondence between the various sub-functional modules of the model object and the execution priority of the hardware unit, and then the target hardware unit corresponding to the specific sub-functional module can be further determined based on the current available resources of each hardware unit.

[0062] Exemplarily, if the multiple hardware units of the first device include a CPU, a GPU and an NPU, according to the preset correspondence between the sub-functional modules of the model object and the execution priority of the hardware units, the execution speed of the specific sub-functional module A is sorted from fast to slow as GPU, NPU and CPU, and the current available resources of the GPU account for 10% of all available resources of the GPU, the current available resources of the NPU account for 80% of all available resources of the NPU, and the current available resources of the CPU account for 50% of all available resources of the CPU. In this case, the NPU can be determined as a target hardware unit for the specific sub-functional module A, and the CPU can also be determined as the target hardware unit corresponding to the specific sub-functional module A.

[0063] In some embodiments, when the specific sub-functional modules include multiple ones, if there is an execution order for each specific sub-functional module, the target hardware unit corresponding to each specific sub-functional module can be determined according to the execution order of each specific sub-functional module; when the specific sub-functional modules are independent of each other and there is no execution order constraint, the target hardware unit corresponding to each specific sub-functional module can be determined at the same time.

[0064] In some embodiments, the data of a specific sub-functional module may include a model parameter weight file corresponding to the specific sub-functional module, and the data requested by the user may include the data content of the processing task corresponding to the user request, which may include text, voice, pictures, videos, etc.

[0065] In some embodiments, after determining the target hardware unit corresponding to each specific sub-functional module, the data of the specific sub-functional module and the data of the user request can be scheduled to the corresponding target hardware unit, so that the target hardware unit calculates the processing task corresponding to the user request based on the data of the specific sub-functional module, obtains a response result for the request, and feeds the response result back to the user.

[0066] It can be understood that based on the current available resources of each hardware unit and the preset correspondence between each sub-functional module of the model object and the execution priority of the hardware unit, a comprehensive decision is made based on the theoretical execution speed priority of the hardware unit and the actual operation of the hardware unit, so that the target hardware unit corresponding to the specific sub-functional module is determined to be more reasonable, thereby improving the data execution speed of each specific sub-functional module to respond to user requests more quickly.

[0067] In some embodiments of the present application, after obtaining the user request, i.e., step S101, if it is determined that the currently available resources of the hardware units of the first device cannot complete the processing of the user request, a second device with currently available hardware resources is determined; the data of the user request and the data of the corresponding sub-functional module are sent to the second device, so that the user request is processed by the hardware units of the second device.

[0068] In some embodiments, the second device may be an electronic device that establishes a communication connection with the first device. For example, the second device may be a remote server, a cloud server, etc.

[0069] In some embodiments, if it is determined that the current available resources of the various hardware units of the first device are unable to calculate the data requested by the user based on the data of the sub-function module of the target model object corresponding to the user request, it can be determined whether there are available hardware resources on the second device that has established a communication connection with the first device. If it is determined that the second device has available hardware resources and the hardware resources can complete the calculation of the data requested by the user based on the data of the target model object, the data requested by the user and the data of the corresponding sub-function module can be sent to the second device, and the user request can be processed by the hardware resources of the second device.

[0070] In some embodiments, the hardware resources of the second device may include currently available resources of the hardware unit of the second device, such as the currently available number of cores, available number of threads, available memory space, etc. of the hardware unit. After obtaining the data requested by the user and the data of the corresponding sub-function module, the second device may calculate the data requested by the user based on the data of the sub-function module, thereby obtaining a response result for the user request, and sending the response result to the first device.

[0071] In some embodiments, the second device may also have a pre-established correspondence table between the various sub-functional modules of the model object and the execution priorities of the hardware units. The second device can call the corresponding model object locally based on the obtained user request, and schedule the sub-functional module data of the model object to its corresponding target hardware unit according to the correspondence table, thereby processing the user request.

[0072] It can be understood that when the currently available resources of the hardware unit of the first device are unable to complete the processing of the user request, by sending the data of the user request and the data of the corresponding sub-functional module to the second device for processing, the available hardware resources of the second device that is communicatively connected to the first device can be reasonably utilized, so that the user request can be processed in a timely manner and the user experience can be improved.

[0073] In some embodiments of the present application, the user request carries task processing indication information. After obtaining the user request, i.e., step S101, if it is determined that the task processing indication information in the user request indicates that the current user request is to be processed by the target second device, the data of the current user request is sent to the target second device so that the current user request is processed by the hardware unit of the target second device.

[0074] In some embodiments, the target second device may be a specific electronic device that can establish a communication connection with the first device and is indicated by the task processing instruction information.

[0075] In some embodiments, when it is determined that the task processing indication information indicates that the current user request is to be processed through the target second device, if the target second device is already in a connected state with the first device, the data requested by the current user and the data of the corresponding sub-function module can be directly sent to the target second device for processing; if the target second device and the first device have not established a communication connection, a communication connection between the two electronic devices can be established first, and then the data requested by the current user and the data of the corresponding sub-function module can be sent to the target second device for processing.

[0076] It can be understood that when the task processing indication information carried in the user request indicates that the current user request is processed through the target second device, the user's personalized experience can be met by sending the data of the current user request to the target second device so that the current user request is processed through the second device.

[0077] In some embodiments of the present application, the first device includes a model management module, which can also determine the target model object corresponding to the user request after obtaining the user request, that is, step S101; if it is determined that the management function of the model management module has passed the verification, obtain the decryption key of the target model object stored in the local device through the model management module; use the decryption key to decrypt the encrypted data of the target model object to obtain the data of each sub-functional module of the target model object, so as to perform inference calculations on the data requested by the user based on the data of each sub-functional module of the target model object.

[0078] In some embodiments, the task type of the corresponding task can be determined based on a user request, and the corresponding target model object can be determined based on the task type. The data of the target model object can be pre-stored in a storage area of ​​the first device. The data of the target model object can be encrypted, so the encrypted data of the target model object needs to be decrypted before calling the target model object.

[0079] In some real-time, the model management module can be used to manage model objects, and the management functions of the model management module may include encrypting encrypted model object data, calling model objects, etc. Therefore, before calling the target model object, the management function of the model management module may be verified first, and if the model management module passes the verification, the decryption key of the target model object stored in the local device may be obtained through the model management module.

[0080] In some embodiments, the model management module can be used to calculate a check code based on the local timestamp, salt value and the model's unique security decryption key through the Secure Hash Algorithm 256 (SHA256) to verify whether the management function of the model management module is normal. If the model management module can generate a salt value and can generate a check code based on the local timestamp, salt value and the model's unique security decryption key through the SHA256 algorithm, it can be proved that the management function of the model management module is normal.

[0081] In some embodiments, when it is determined that the management function of the model management module is normal, the decryption key of the target model object can be obtained through the model management module, and the encrypted data of the target model object can be decrypted using the decryption function of the model management module to obtain the data of each sub-functional module of the target model object.

[0082] In some embodiments, after obtaining the data of each sub-functional module of the target model object, the data of each sub-functional module of the target model object can also be loaded into the memory to realize the call of the target model object. After calling the target model object, the data of the sub-functional modules of the target model object can be used to perform inference calculations on the data requested by the user.

[0083] It can be understood that by verifying the management function of the model management module, it can be ensured that the data decryption, model calling and other functions of the model management module are normal, so that the encrypted data of the target model object can be smoothly decrypted through the model management module, and the target model object can be called to realize the sub-function module of the target model object to perform inference operations on the data requested by the user.

[0084] In some embodiments of the present application, the user request includes multiple requests. After obtaining the user request, that is, step S101, the target model object corresponding to each user request and the sub-functional module of each target model object can also be determined; according to the correspondence between each sub-functional module of each target model object and the execution priority of the hardware unit, and the available resources of the hardware unit, the execution order of the multiple user requests is determined.

[0085] In some embodiments, each user request may be obtained at the same time, or may be obtained within a preset time period, for example, within 100 milliseconds or 1 second. According to the task type of the task corresponding to each user request, the model object that can handle the task corresponding to the task type can be determined as the target model object, thereby obtaining the target model object corresponding to each user request.

[0086] In some embodiments, the execution order of multiple user requests can be determined by combining the preset correspondence between each sub-function module of each target model object and the execution priority of the hardware unit, and the current available resources of each hardware unit. The correspondence between each sub-function module of each target model object and the execution priority of the hardware unit can be obtained from the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit.

[0087] In some embodiments, the execution order of user requests includes parallel processing and serial processing. If the target model objects corresponding to the multiple user requests are different, and the target hardware units corresponding to the multiple sub-function modules are different according to the corresponding relationship between the execution priority of each sub-function module of each target model object and the hardware unit, and the available resources of the hardware unit, then the data requested by each user can be dispatched to different hardware units for parallel processing.

[0088] In some embodiments, if multiple user requests correspond to the same target model object, or even if the target model objects are different, but the target hardware units corresponding to the sub-functional modules of the target model objects are the same, each user request can be processed serially according to the order in which the user requests arrive, or the complexity of the tasks corresponding to the user requests.

[0089] It can be understood that, in the case where multiple user requests are included, the sub-function modules of the target model objects corresponding to each user request are determined respectively, and the execution order of multiple user requests is determined based on the correspondence between each sub-function module of each target model object and the execution priority of the hardware unit, as well as the available resources of the hardware unit. This can achieve parallel or serial processing of multiple tasks and ensure timely response to multiple user requests.

[0090] In some embodiments of the present application, after obtaining the user request, that is, step S101, the text content in the user request can also be converted into feature information, which can include input data for different types of models. The feature information can be identified by a feature vector. For example, the text content in the user request can be converted into a prompt word token suitable for the input of a raw text model, and each token is used as a feature vector; for a raw image model, information such as image size and resolution can be added after the prompt word to form a feature vector corresponding to the raw image model. Information such as image size and resolution can be determined by the raw image model.

[0091] In some embodiments, the first model object may be a text model, a picture model, or other types of large models, and the first model object and the second model object are different types of model objects. After converting the text content in the user request data into feature information, if it is determined that the target model object to be called is the first model object, the feature information can be input into the first model object to process the user request through the first model object.

[0092] In some embodiments, if the first model object is a text model object, the second model object may be a picture model object, and if the first model object is a picture model object, the second model object may be a text model object. When it is determined that the target model object to be called is the second model object, the feature information may be input into the second model object to process the user request through the second model object.

[0093] It can be understood that by converting the text content in the user request into input data for different types of models, different model objects can support input of the same modality, thereby avoiding the need to convert the user request input into the corresponding input of different model objects for different types of model objects, thereby improving the execution efficiency of user requests.

[0094] In an embodiment of the present application, a user request is obtained; according to the user request, a corresponding target model object is called; according to the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, the data of the sub-function module of the target model object is dispatched to the corresponding target hardware unit, so as to utilize the available resources of the target hardware unit to process the user request. In this way, through the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, the target hardware unit that can make each sub-function module of the target model object execute the fastest can be determined from the multiple hardware units of the first device, and the data of the sub-function module can be further dispatched to the corresponding target hardware computing unit to process the user request, which can improve the operating efficiency of the target model object and the response speed to the user request.

[0095] Next, the implementation process of the application embodiment in the actual application scenario is introduced.

[0096] like Figure 2 As shown, it is a schematic diagram of the architecture of a local multi-modal multi-model service system provided by this application, such as Figure 2 As shown, the local multimodal multi-model service system 200 includes a Windows Universal Application Platform (Universal Windows Platform, UWP) APP (Application) 201, a service communication module 202, a client model service module 203 (equivalent to the "model service module" in other embodiments), a hardware unit module 204, a model security verification module 205 and a remote hardware unit module 206.

[0097] In some embodiments, the service communication module 202 includes a communication interface 2021 for models of different versions (such as version 1.0, version 1.5, other versions, etc.); the client model service module 203 includes a service interface 2031, a task queue 2032, an AI chip security encryption submodule 2033, a data processing submodule 2034, a content encryption module 2035, a multimodal system 2036, model channels of different versions 2037, a large model submodule 2038, etc., and the large model module 2038 includes large models of different versions (such as version 1.0, version 1.5, other versions, etc.); the hardware unit module 204 includes a first CPU 2041, a first iGPU 2042, a first dGPU 2043, and a first NPU 2044 of a local device (equivalent to the "hardware unit" in other embodiments); the model security verification module 205 includes a local timestamp 2051, a salt value 2052, and a model security key 2053 that can be generated; the remote device hardware unit module 206 includes a second CPU of the remote device 2061 , second iGPU 2062 , second dGPU 2063 , second NPU 2064 .

[0098] In some embodiments, the local multi-modal multi-model service system 200 can realize multi-APP communication, and the client model service module 203 can be used as a resident background service to communicate with the APP through the client model communication component (dynamic link library) in the APP. Various types of APPs can integrate the same communication component to ensure that the client model service module 203 can communicate with various types of APPs at the same time.

[0099] In some embodiments, various types of multi-task parallel large model APPs can pass tasks to the client model service module 203 through remote Python call (Remote Python Call, RPyC) local distributed communication. The client model service module 203 can allocate the corresponding large model execution tasks according to the order in which the tasks arrive, and load the large model tasks on different hardware units according to the running efficiency of the model on different hardware. The mapping table of the relationship between different large models (and / or sub-functional modules of large models) and hardware priorities is shown in Table 1. The model task load distribution algorithm (such as a neural network distribution algorithm) can be used to distribute the execution hardware unit for the new task according to the mapping table and the execution status of the model task on the hardware.

[0100] In some embodiments, the client model service module 203 can allow the APP to register the available remote devices in the remote call module of the client model service module 203 in advance. If the local computing power has been exhausted, or the APP actively requests to use the remote computing power, the client model service module 203 can detect the remotely callable computing power through the Secure Shell Protocol (SSH) and load the reasoning task on the remote device hardware unit module 206 through the reasoning framework.

[0101] In some embodiments, the client model service module 203 may include an obfuscated encrypted large model module 2037, and the model's unique security decryption key is serialized in the model parameters. Deobfuscation decryption requires a local timestamp, salt value, and the model's unique security decryption key. The client model service module 203 will periodically detect whether the model parameters have been forcibly cracked or tampered with.

[0102] In some embodiments, the client model service module 203 and the AI ​​chip use the local timestamp, salt value, and model-unique security decryption key to calculate the check code through the SHA256 algorithm, and check each other. If they pass, the model is considered safe. If they fail, it is considered to be risky, and the risky model will be temporarily disabled and reported to the APP.

[0103] In some embodiments, the client model service module 203 has a built-in multimodal calling algorithm that converts a unified token into a vector used by different models using a trained text embedding model for direct reasoning.

[0104] The local multimodal multi-model service system provided in the embodiment of the present application can perform load distribution based on GPU, NPU, and GPU, and realize multi-task parallelism by integrating multiple local large model algorithms; it can realize secure encryption of large model content output and hardware encryption of model parameters AI chip; it can realize distributed calling of computing power on other remote devices for distributed computing; it can support unified instructions or prompt words to call multimodal model services.

[0105] In an embodiment of the present application, a user request is obtained; according to the user request, a corresponding target model object is called; according to the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, the data of the sub-function module of the target model object is dispatched to the corresponding target hardware unit, so as to utilize the available resources of the target hardware unit to process the user request. In this way, through the preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, the target hardware unit that can make each sub-function module of the target model object execute the fastest can be determined from the multiple hardware units of the first device, and the data of the sub-function module can be further dispatched to the corresponding target hardware computing unit to process the user request, which can improve the operating efficiency of the target model object and the response speed to the user request.

[0106] The present application also provides a data scheduling system. Figure 3 A schematic diagram of the structure of a data scheduling system provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the data scheduling system 300 includes:

[0107] The application module 301 is used to obtain a user request; and call a corresponding target model object according to the user request;

[0108] The model service module 302 is used to dispatch the data of the sub-function modules of the target model object to the corresponding target hardware unit according to the preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware unit, so as to process the user request by utilizing the available resources of the target hardware unit;

[0109] Hardware unit 303, used for running data of sub-function modules of target model object;

[0110] Among them, each sub-functional module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset corresponding relationship is determined based on the execution speed of each sub-functional module on different hardware units.

[0111] In some embodiments, the model service module 302 is also used to: obtain the current available resources of each hardware unit; determine the target hardware unit corresponding to each specific sub-functional module based on the available resources and the preset correspondence; the specific sub-functional module is at least one sub-functional module used to perform inference calculations on the data requested by the user; and schedule the data of the specific sub-functional module and the data requested by the user to the corresponding target hardware unit.

[0112] In some embodiments, the data scheduling system 300 further includes:

[0113] A first determination module, used to determine a first task type corresponding to a reference sub-function module; the reference sub-function module is any one of the sub-function modules;

[0114] A first scheduling module, configured to schedule the data of the reference sub-function module to different hardware units, and obtain reference execution speeds of different hardware units for user requests of the first task type;

[0115] The first creation module is used to establish the preset corresponding relationship according to each reference execution speed.

[0116] In some embodiments, the model service module 302 is further used to: if the currently available resources of the hardware units of the first device cannot complete the processing of the user request, determine that there is currently a second device with available hardware resources; the second device and the first device have established a communication connection;

[0117] The data of the user request and the data of the corresponding sub-function module are sent to the second device, so that the user request is processed by the hardware unit of the second device.

[0118] In some embodiments, the user request carries task processing indication information; the model service module 302 is also used for: if the task processing indication information indicates that the current user request is processed by the target second device, the data of the current user request is sent to the target second device, so as to use the hardware unit of the target second device to process the current user request.

[0119] In some embodiments, the first device includes a model management module, and the model service module 302 is also used to: determine the target model object corresponding to the user request; if it is determined that the management function of the model management module has passed the verification, obtain the decryption key of the target model object stored in the local device through the model management module; use the decryption key to decrypt the encrypted data of the target model object to obtain the data of each sub-functional module of the target model object, so as to perform inference calculations on the data requested by the user based on the data of each sub-functional module of the target model object.

[0120] In some embodiments, the user request includes multiple requests; the model service module 302 is also used to: determine the target model objects corresponding to each user request, and the sub-functional modules of each target model object; determine the execution order of multiple user requests according to the correspondence between each sub-functional module of each target model object and the execution priority of the hardware unit, and the available resources of the hardware unit, and the execution order of the user requests includes parallel processing and serial processing.

[0121] In some embodiments, the model service module 302 is also used to: convert the text content in the user request into feature information, the feature information including input data for different types of models; if the corresponding target model object is called as the first model object, the feature information is input into the first model object for processing so that the first model object processes the user request; if the corresponding target model object is called as the second model object, the feature information is input into the second model object for processing so that the second model object processes the user request; wherein the first model object and the second model object are different types of model objects.

[0122] It should be noted that the description of the data scheduling system of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0123] It should be noted that in the embodiment of the present application, if the control method of the first electronic device described above is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0124] Accordingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the control method of the first electronic device provided in the above embodiment is implemented.

[0125] The embodiment of the present application also provides a data scheduling device, Figure 4 A schematic diagram of the structure of a data scheduling device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the data scheduling device 500 includes: a memory 401, a processor 402, a communication interface 403 and a communication bus 404. The memory 401 is used to store executable data scheduling instructions; the processor 402 is used to execute the executable data scheduling instructions stored in the memory to implement the data scheduling method provided in the above embodiment, wherein the executable data scheduling instructions are used to implement: obtaining user requests, calling corresponding target model objects according to user requests; according to the preset correspondence between the execution priorities of each sub-function module of the model object and the hardware unit, scheduling the data of the sub-function module of the target model object to the corresponding target hardware unit, so as to use the available resources of the target hardware unit to process the user request; wherein each sub-function module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset correspondence is determined based on the execution speed of each sub-function module on different hardware units.

[0126] The description of the above data scheduling device and storage medium embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the data scheduling device and storage medium embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0127] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises at least one ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0129] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0131] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, disks or optical disks.

[0132] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a product to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0133] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field can be easily conceived of within the technical scope disclosed in the present application. Changes or substitutions should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data scheduling method, applied to a first device, comprising: Get user request; According to the user request, calling the corresponding target model object; According to the preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware units, the data of the sub-function modules of the target model object are dispatched to the corresponding target hardware units, so as to process the user request by utilizing the available resources of the target hardware units; Among them, each sub-functional module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset corresponding relationship is determined based on the execution speed of each sub-functional module on different hardware units.

2. The method according to claim 1, wherein the data of the sub-function modules of the target model object are dispatched to the corresponding target hardware unit according to the preset correspondence between the execution priorities of the sub-function modules of the model object and the hardware unit, comprising: Get the currently available resources of each hardware unit; Based on the available resources and the preset corresponding relationship, determine the target hardware units corresponding to the specific sub-functional modules; The specific sub-functional module is at least one sub-functional module for performing inference calculation on the data requested by the user; The data of the specific sub-function module and the data requested by the user are dispatched to the corresponding target hardware unit.

3. According to the method of claim 1, the process of creating the preset corresponding relationship comprises: Determine a first task type corresponding to the reference sub-function module; The reference sub-functional module is any one of the sub-functional modules; Dispatching the data of the reference sub-function module to different hardware units to obtain reference execution speeds of the different hardware units for user requests of the first task type; The preset corresponding relationship is established according to each reference execution speed.

4. The method according to claim 1, further comprising: If currently available resources of the hardware units of the first device cannot complete the processing of the user request, determining that a second device currently has available hardware resources; the second device and the first device have established a communication connection; The data of the user request and the data of the corresponding sub-function module are sent to the second device, so that the user request is processed by the hardware unit of the second device.

5. The method according to claim 1, wherein the user request carries task processing instruction information; the method further comprises: If the task processing instruction information indicates that the current user request is to be processed by the target second device, the data of the current user request is sent to the target second device so that the current user request is processed by the hardware unit of the target second device.

6. The method according to claim 1, wherein the first device comprises a model management module, and the method further comprises: Determine the target model object corresponding to the user request; If it is determined that the management function of the model management module passes the verification, obtaining the decryption key of the target model object stored in the local device through the model management module; The encrypted data of the target model object is decrypted using the decryption key to obtain data of each sub-functional module of the target model object, so as to perform inference calculation on the data requested by the user based on the data of each sub-functional module of the target model object.

7. The method according to any one of claims 1 to 6, wherein the user request comprises a plurality of requests; the method further comprises: Determine the target model object corresponding to each user request and the sub-function module of each target model object; The execution order of multiple user requests is determined according to the corresponding relationship between each sub-function module of each target model object and the execution priority of the hardware unit, and the available resources of the hardware unit. The execution order of the user requests includes parallel processing and serial processing.

8. The method according to claim 7, further comprising: Converting text content in a user request into feature information, wherein the feature information includes input data for different types of models; If the corresponding target model object is the first model object, the feature information is input into the first model object for processing, so that the first model object processes the user request; If the corresponding target model object is called as the second model object, the feature information is input into the second model object for processing, so that the second model object processes the user request; wherein the first model object and the second model object are model objects of different types.

9. A data scheduling system, comprising: Application module, model service module and multiple hardware units; among them, The application module is used to obtain a user request; and call a corresponding target model object according to the user request; The model service module is used to dispatch the data of the sub-function modules of the target model object to the corresponding target hardware unit according to the preset corresponding relationship between the execution priorities of the sub-function modules of the model object and the hardware unit, so as to process the user request by utilizing the available resources of the target hardware unit; A hardware unit, used to run the data of the sub-function modules of the target model object; Among them, each sub-functional module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset corresponding relationship is determined based on the execution speed of each sub-functional module on different hardware units.

10. A data scheduling device, comprising: A memory for storing executable data scheduling instructions; The processor is used to execute the executable data scheduling instruction stored in the memory, wherein the executable data scheduling instruction is used to implement: obtaining a user request, and calling a corresponding target model object according to the user request; and scheduling the data of the sub-function module of the target model object to the corresponding target hardware unit according to a preset correspondence between each sub-function module of the model object and the execution priority of the hardware unit, so as to process the user request by utilizing the available resources of the target hardware unit; A hardware unit, used to run the data of the sub-function modules of the target model object; Among them, each sub-functional module of the target model object is used to perform inference operations on the data requested by the corresponding user, and the preset corresponding relationship is determined based on the execution speed of each sub-functional module on different hardware units.

Images