AI computing power resource remote calling framework and big data direct connection method

Abstract

The invention relates to the technical field of big data, and discloses an AI computing power resource remote calling framework and a big data direct connection method. The AI computing power resource remote calling framework comprises an interaction layer and a resource layer; the interaction layer is used for interacting with a user application on a user side, and the user application adopts a programming mode and calls the AI computing power service of the resource layer through an API interface provided by the interaction layer to execute model training or model reasoning; and the resource layer is used for bearing and managing AI computing power resources and scheduling the AI computing power resources through an AI computing power service to execute model reasoning or model training. Layered design is adopted, development and application scenes which are more convenient and easier to use are provided for computing resource users and AI development users, remote calling of the AI computing power resources is achieved through dynamic scheduling of the AI computing power resources, the framework can be deployed on multi-heterogeneous computing power platforms such as a cloud end, an edge end and an equipment end in practical application, and the framework is easy to popularize. And the expandability and scalability of the AI computing power resource platform are further improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an AI computing resource remote calling framework and a big data direct access method. Background Art

[0002] Artificial Intelligence (AI) computing power is the core of AI technology research and development. Currently, all IT giants have proposed to build their own AI computing frameworks and computing power resource services that are in line with their own business ecosystems and characteristics. Computing power refers to the ability to execute computing, which is an abstraction of computing resources. Computing power resources are a prerequisite for AI-related services. The execution process of AI application models consumes computing power resources. By managing computing nodes and computing resources, computing resources are provided to application models on demand according to user needs to meet computing needs, which is the AI ​​computing power resource scheduling function.

[0003] There are currently two execution modes for AI application models. One is that after the model is trained and saved, the model is subsequently used for prediction and call, without the need for computing resource management; the other is that after the model is trained, computing resource services are remotely provided through servers (such as cloud services, edge devices, etc.), and the results do not need to be sent back to the user after the algorithm model is executed. However, the current market AI applications are mainly provided to developers by AI public cloud services (such as Google AI, Alibaba Cloud AI, Baidu AI, etc.). Although developers do not need to buy servers to build AI algorithm model execution engine services, computing resource services are all bundled in the public cloud. During the execution of the application model, users cannot remotely call the computing resources of the public cloud according to actual needs. There are also independent computing infrastructure service providers in the current market, such as computing devices (such as NVIDIA Jetson TX2, Google AI accomplAIish, Inspur AI Master, etc.). Users purchase computing resources through leasing or buyout and use them in their own products (such as TVs, cars, smart speakers, etc.), but this type of computing resources cannot provide remote call functions and can only be used in devices close to users. It is impossible to provide computing resource services over long distances. Summary of the invention

[0004] The main purpose of the present invention is to provide an AI computing power resource remote calling framework and a big data direct access method, aiming to solve the technical problem that the existing AI computing power resource services are mainly bundled in the public cloud, and users cannot remotely call the computing power resources of the public cloud according to actual needs. At the same time, the computing power resources provided by independent computing power infrastructure service providers cannot provide remote calling functions and can only be used on devices close to the user, and cannot provide computing power resource services over long distances.

[0005] A first aspect of the present invention provides an AI computing power resource remote calling framework, the AI ​​computing power resource remote calling framework comprising: an interaction layer and a resource layer; The interaction layer is used to interact with the user application on the user side. The user application adopts a programming mode and calls the AI ​​computing service of the resource layer through the API interface provided by the interaction layer to perform model training or model reasoning; the resource layer is used to carry and manage AI computing resources, and schedule AI computing resources to perform model reasoning or model training through the AI ​​computing service; The interaction layer includes a user module, a data module and a model module. The user module is used to receive a remote call request for AI computing resources initiated by a user application on the user side and return a response result; the data module is used to save the business data uploaded by the user application through the user module, and the business data includes training data or inference data; the model module is used to save the AI ​​model and AI algorithm, and the AI ​​model is obtained by training the AI ​​algorithm with the training data in the data module; The resource layer includes a service module and a scheduling module. The service module is used to generate corresponding AI computing service tasks based on the AI ​​computing resource remote call request initiated by the user application. The AI ​​computing service includes AI model training service and AI model reasoning service. The AI ​​model reasoning service includes AI online reasoning service and AI offline reasoning service. The scheduling module is used to respond to the AI ​​computing service tasks of the service module, and schedule AI computing resources to perform model reasoning or model training according to the AI ​​model or AI algorithm provided by the model module.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the resource layer is further provided with a plurality of data exchange areas, and each of the data exchange areas is provided with a plurality of data transmission links connecting the data exchange area and the AI ​​computing power resources; The data exchange area is provided with a plurality of data sub-areas, each of which is provided with an AI model training queue group and an AI model inference queue group; the AI ​​model training queue group includes a plurality of AI model training queues, and the AI ​​model inference queue group includes a plurality of AI model inference queues; the AI ​​model training queue group and the AI ​​model inference queue group in the same data sub-area are connected in series in sequence through the queue exchange area; The AI ​​model training queue establishes a communication connection with the AI ​​model training resources in the AI ​​computing power resources through a corresponding data transmission link, and the AI ​​model inference queue establishes a communication connection with the AI ​​model inference resources in the AI ​​computing power resources through a corresponding data transmission link.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the data module is further used to initiate a business data upload request to the corresponding AI model training queue group or AI model reasoning queue group, so as to store the uploaded business data in the queue of the corresponding AI model training queue group or AI model reasoning queue group; The scheduling module is also used to transmit the business data stored in the queues of the AI ​​model training queue group or the AI ​​model reasoning queue group to the corresponding AI computing power resources through the data transmission link to perform model reasoning or model training.

[0008] Optionally, in a third implementation method of the first aspect of the present invention, when the AI ​​model training resources or AI model reasoning resources in the AI ​​computing power resources are in a busy state, the scheduling module is also used to store the received business data in a temporary storage area; the temporary storage area establishes a communication connection with the AI ​​computing power resources through a corresponding data transmission link.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, all queue groups in the multiple data exchange areas of the resource layer form a data transmission belt in parallel, and each data exchange area is composed of multiple queue groups; when a business data upload request is initiated to the corresponding queue group, the data module will match a data transmission link with each queue in the queue group, and at the same time establish a communication connection with the AI ​​computing power resources and transmit the business data to each AI computing power resource in turn according to the matched data transmission link to perform model reasoning or model training.

[0010] Optionally, in the fifth implementation method of the first aspect of the present invention, the scheduling module is also used to perform data preprocessing on the uploaded business data in parallel in the queue exchange area within each data sub-area and then store the data in the corresponding queue when the data module initiates a business data upload request. The data preprocessing includes one or more of batch data storage, data dimensionality reduction, data compression, and data format conversion.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the data module includes: A splitting submodule is used to split the business data to be calculated into multiple data blocks of the same size according to the data splitting rule set by the user side, wherein the data splitting size is related to the data transmission speed; The upload submodule is used to mark the transmission time and data identifier of each data block, and transmit each data block through the queue. After the data transmission is completed, it is determined whether the transmission time of each data block exceeds the user-specified time. If so, the data is transmitted again.

[0012] Optionally, in a seventh implementation of the first aspect of the present invention, the service module includes: The strategy generation submodule is used to set the AI ​​computing resource selection strategy based on the split size of the business data to be calculated and the data processing speed of each AI computing resource; The task generation submodule is used to generate the corresponding AI computing power service task based on the remote call request of the AI ​​computing power resource initiated by the user application and send it to the scheduling module.

[0013] Optionally, in an eighth implementation of the first aspect of the present invention, the AI ​​computing power resource selection strategy is set based on the following optimization objectives and constraints: ; ; Where T represents the total processing time of the business data to be processed, Z represents the total amount of business data to be processed, represents the amount of business data allocated to the i-th AI computing resource, m represents the total number of AI computing resources, Indicates data transmission parameters, It represents the data processing speed of the i-th AI computing resource, n represents the number of data blocks into which the business data to be processed is split, and p represents the average split size of the business data to be processed.

[0014] A second aspect of the present invention further provides a big data direct access method, which adopts the AI ​​computing power resource remote call framework as described in any one of the above items, and the big data direct access method includes: The user module receives a remote call request for AI computing resources initiated by a user application on the user side and corresponding business data to be processed, and saves the business data through the data module; The data module splits the business data into multiple data blocks, and initiates a request to upload the business data to the scheduling module; The service module generates a corresponding AI computing service task based on the AI ​​computing resource remote call request and sends it to the scheduling module, wherein the AI ​​computing service task includes an AI computing resource selection strategy and an adopted AI computing service; The scheduling module responds to the AI ​​computing power service task of the service module, obtains the AI ​​model or AI algorithm provided by the model module, and schedules the corresponding AI computing power resources to perform model reasoning or model training on the business data uploaded by the data module according to the AI ​​computing power resource selection strategy.

[0015] Optionally, in a first implementation method of the second aspect of the present invention, the data module splits the business data to be calculated into multiple data blocks of the same size according to the data splitting rules set by the user side, wherein the data splitting size is related to the data transmission speed; marks the transmission time and data identifier of each data block, transmits each data block through a queue, and after the data transmission is completed, determines whether the transmission time of each data block exceeds the user-specified time, and if so, retransmits it.

[0016] Optionally, in a second implementation method of the second aspect of the present invention, the service module also sets an AI computing power resource selection strategy based on the split size of the business data to be calculated and the data processing speed of each AI computing power resource; based on the AI ​​computing power resource remote call request initiated by the user application, generates a corresponding AI computing power service task and sends it to the scheduling module.

[0017] Optionally, in a third implementation of the second aspect of the present invention, the AI ​​computing power resource selection strategy is set based on the following optimization objectives and constraints: ; ; Where T represents the total processing time of the business data to be processed, Z represents the total amount of business data to be processed, represents the amount of business data allocated to the i-th AI computing resource, m represents the total number of AI computing resources, Indicates data transmission parameters, It represents the data processing speed of the i-th AI computing resource, n represents the number of data blocks into which the business data to be processed is split, and p represents the average split size of the business data to be processed.

[0018] The present invention provides a remote call framework for AI computing resources, including: an interaction layer and a resource layer; the interaction layer is used to interact with user applications on the user side, and the user application adopts a programming mode and calls the AI ​​computing service of the resource layer through the API interface provided by the interaction layer to perform model training or model reasoning; the resource layer is used to carry and manage AI computing resources, and schedule AI computing resources to perform model reasoning or model training through the AI ​​computing service. The present invention adopts a layered design, which not only provides a more convenient and easy-to-use development and application scenario for computing resource users and AI development users, but also realizes the remote call of AI computing resources through the dynamic scheduling of AI computing resources. The framework can be deployed on multiple heterogeneous computing platforms such as the cloud, edge, and device in actual applications, further improving the scalability and scalability of the AI ​​computing resource platform. The present invention can not only independently provide the computing infrastructure to users, so that users can schedule and remotely call computing resources on demand according to actual application needs, but also enable developers to quickly build artificial intelligence application models. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an embodiment of a remote calling framework for AI computing resources in an embodiment of the present invention; Figure 2 The figure is a schematic diagram of an embodiment of the big data direct access method in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] For ease of understanding, the following describes the AI ​​computing resource remote call framework in the embodiment of the present invention. Figure 1 In an embodiment of the present invention, an embodiment of the AI ​​computing resource remote calling framework includes: 1. Interaction Layer As a bridge between the user side and the resource layer, the interaction layer 10 undertakes important responsibilities such as receiving user requests, processing user data, and managing AI models. The interaction layer 10 is used to interact with user applications on the user side. The user application uses a programming mode and calls the AI ​​computing power service of the resource layer 20 through the API interface provided by the interaction layer 10 to perform model training or model reasoning. The interaction layer 10 is the core component for the user side to interact with user applications. It provides a rich API interface, enabling user applications to call the AI ​​computing power service of the resource layer 20 in a programming mode to perform model training or model reasoning. The design goal of the interaction layer 10 is to ensure that users can use AI computing resources efficiently and conveniently, while ensuring the security of data and the reliability of the model.

[0022] In this embodiment, the interaction layer 10 further includes: 1.1 User Module The user module 101 is used to receive a remote call request for AI computing resources initiated by a user application on the user side and return a response result.

[0023] The user module 101 is a module in the interaction layer 10 that is responsible for receiving remote call requests for AI computing resources initiated by user applications on the user side and returning response results. It implements functions such as user authentication, request parsing, and response generation to ensure that users can call AI computing services safely and accurately.

[0024] (1) User authentication User authentication is the primary task of the user module 101, which ensures that only legitimate users can access the AI ​​computing service. User authentication is implemented as follows: The user needs to enter the username and password in the user application, and the user module 101 sends this information to the authentication server for verification. If the verification is successful, the user module 101 will generate a session token (SessionToken) for the user, which is used to verify the user's identity in subsequent requests. The user module 101 supports the OAuth2.0 authentication protocol, allowing users to log in through third-party authentication services (such as Google, Facebook, etc.). The user module 101 sends the user's OAuth2.0 access token (Access Token) to the authentication server for verification. If the verification is successful, a session token is generated for the user. For users who need to frequently call AI computing services, the user module 101 supports API key authentication. Users can configure API keys in the user application, and the user module 101 carries the API key in each request for verification.

[0025] (2) Request analysis After receiving the request initiated by the user application, the user module 101 needs to parse the request and extract key information in the request, such as AI computing resource type, model name, input data, etc. The implementation method of request parsing is as follows: User module 101 supports the HTTP protocol and can parse the methods (GET, POST, PUT, DELETE, etc.), URL paths, request headers, and request bodies in HTTP requests. Based on the request method and URL path, user module 101 can determine the type of AI computing service that the user wants to call. User module 101 supports data transmission in JSON and XML formats. For POST and PUT requests, user module 101 parses the JSON or XML data in the request body into objects or structures for subsequent processing. User module 101 verifies the parsed request parameters to ensure that the parameters meet the calling requirements of the AI ​​computing service. If the parameters do not meet the requirements, user module 101 will return an error response.

[0026] (3) Response generation After calling the AI ​​computing service of the resource layer 20, the user module 101 needs to generate a response result and return it to the user application. The implementation method of response generation is as follows: User module 101 generates an HTTP response based on the call result, including a status code, a response header, and a response body. The status code is used to indicate the success or failure of the call result. The response header contains some additional information (such as Content-Type, Content-Length, etc.), and the response body contains detailed information of the call result. For the detailed information of the call result, user module 101 serializes it into JSON or XML format so that the user application can parse it correctly. If an error occurs when calling the AI ​​computing power service of resource layer 20, user module 101 will capture the error and generate a corresponding error response. The error response contains an error code, error message, and possible solutions.

[0027] 1.2 Data Module The data module 102 is used to store the business data uploaded by the user application through the user module 101, and the business data includes training data or inference data.

[0028] The data module 102 is a module in the interaction layer 10 responsible for storing the business data uploaded by the user application through the user module 101. Business data includes training data and reasoning data, which are the basis of AI model training and reasoning. The data module 102 implements functions such as data storage, retrieval, encryption and decryption to ensure the security and reliability of data.

[0029] (1) Data storage Data storage is the core task of the data module 102, which implements the function of saving the business data uploaded by the user into the storage system. The implementation of data storage is as follows: The data module 102 supports the storage methods of relational databases (such as MySQL, PostgreSQL, etc.) and non-relational databases (such as MongoDB, Cassandra, etc.). According to the type and characteristics of the business data, the data module 102 can select a suitable database for storage. For large-scale business data (such as images, videos, etc.), the data module 102 supports file storage. It can save business data to distributed file systems (such as HDFS, Ceph, etc.) or object storage (such as AmazonS3, Google Cloud Storage, etc.). Before storing business data, the data module 102 will verify the data to ensure the integrity and correctness of the data. If the data does not meet the requirements, the data module 102 will return an error prompt.

[0030] (2) Data encryption and decryption In order to ensure the security of business data, the data module 102 implements data encryption and decryption functions. The implementation of data encryption and decryption is as follows: The data module 102 supports symmetric encryption algorithms (such as AES, DES, etc.) to encrypt and decrypt business data. Symmetric encryption algorithms use the same key for encryption and decryption, and have high security and efficiency. For business data that requires higher security, the data module 102 supports asymmetric encryption algorithms (such as RSA, ECC, etc.) for encryption and decryption. Asymmetric encryption algorithms use a pair of keys (public key and private key) for encryption and decryption, where the public key is used to encrypt data and the private key is used to decrypt data. The data module 102 implements key management functions, including key generation, storage, distribution, and update. Key management ensures the security and availability of keys and prevents key leakage or tampering.

[0031] In an optional embodiment, the data module 102 further includes: (1) A splitting submodule, which is used to split the business data to be calculated into multiple data blocks of the same size according to the data splitting rules set by the user side, where the data split size is related to the data transmission speed; The main responsibility of the split submodule is to split the business data to be calculated into multiple data blocks of the same size according to the data splitting rules set by the user side. This step not only helps to optimize the efficiency of data transmission, but also improves the parallelism and flexibility of data processing. The data splitting rules are the basis for the work of the split submodule, which determines how to divide the original data into multiple data blocks. These rules usually include the size of the data block, the splitting method (such as sequential splitting, random splitting, etc.), and how to identify the split data blocks.

[0032] The size of the data block is a key factor affecting data transmission speed and processing efficiency. Too large a data block may cause too long transmission time and increase the risk of network congestion; while too small a data block may increase transmission overhead and reduce overall transmission efficiency. Therefore, the split submodule needs to dynamically adjust the size of the data block according to the needs of the user side and network conditions. Generally speaking, the size of the data block can be set to an integer multiple of the network transmission unit (such as a TCP segment) to make full use of the network bandwidth.

[0033] The splitting method determines the order and association between data blocks. Sequential splitting is to divide the data into multiple continuous data blocks according to the natural order of the data; while random splitting is to randomly distribute the data to different data blocks. In practical applications, sequential splitting is more commonly used in scenarios where data order needs to be maintained, such as video stream processing; while random splitting is suitable for scenarios where data order is not required, such as big data analysis. In addition, in order to facilitate subsequent data processing and transmission monitoring, each data block requires a unique identifier. This identifier can be a serial number, hash value or timestamp of the data block. When generating data blocks, the splitting submodule will embed this identification information into the data block or store it separately as metadata. The splitting submodule needs to implement an efficient data splitting algorithm to ensure that the data can be accurately split according to the predetermined rules.

[0034] For example, the sequential splitting algorithm divides the data into multiple continuous data blocks according to the natural order of the data. The basic idea of ​​the algorithm starts from the starting position of the data and reads the data block of a specified size each time until the data is completely read. During the reading process, the algorithm records the starting position and size of each data block and generates the corresponding data block identifier.

[0035] The random split algorithm randomly distributes data into different data blocks. The basic idea of ​​the algorithm is to generate a random number sequence first, and then randomly distribute the data into different data blocks according to this sequence. In order to ensure the uniform distribution of data, the algorithm needs to ensure that the amount of data in each data block is roughly equal. At the same time, the algorithm also needs to record the content of each data block and the correlation information between data blocks.

[0036] In actual applications, the size of data and network conditions may change. Therefore, the split submodule needs to implement a dynamic adjustment algorithm to dynamically adjust the size of the data block according to the current network conditions and data size. This algorithm can achieve dynamic adjustment by monitoring the network bandwidth and data transmission speed. When the network bandwidth is high, the algorithm can appropriately increase the size of the data block to improve transmission efficiency; when the network bandwidth is low, the algorithm needs to reduce the size of the data block to reduce transmission delay.

[0037] In addition, in order to ensure the accuracy and integrity of data splitting, the splitting submodule needs to verify the splitting results. The splitting submodule can perform data integrity verification by calculating the hash value or checksum between the original data and the split data blocks. If the hash value or checksum is not equal, it means that an error or loss occurred during the splitting process and the data needs to be split again. For sequentially split data blocks, the splitting submodule also needs to verify the order of the data blocks. This can be achieved by comparing the serial numbers or timestamps of the data blocks. If the order of the data blocks is not as expected, it means that an error occurred during the splitting process or the data blocks were incorrectly assigned to different queues.

[0038] (2) The upload submodule is used to mark the transmission time and data identifier of each data block, and transmit each data block through the queue. After the data transmission is completed, it is determined whether the transmission time of each data block exceeds the user-specified time. If so, the data is transmitted again.

[0039] The main responsibility of the upload submodule is to transmit the split data blocks through the queue and monitor the transmission time of each data block. If the transmission time exceeds the user-specified time, the transmission is re-performed. This step ensures reliable data transmission and timely processing. The upload submodule needs to manage one or more data block transmission queues to ensure that the data blocks can be transmitted in the predetermined order and priority. According to the needs of the application scenario, the upload submodule can select different types of queues to manage the transmission of data blocks. For example, for scenarios where data order needs to be maintained, a first-in-first-out (FIFO) queue can be selected; for scenarios where certain data blocks need to be processed first, a priority queue can be selected. The capacity of the queue determines how many data blocks can be stored at the same time waiting for transmission. This capacity needs to be set according to network bandwidth, data processing capabilities, and user-side requirements. If the queue capacity is too small, the waiting time for data blocks may be too long; if the queue capacity is too large, it may increase memory usage and transmission delay. The upload submodule needs to implement an efficient queue scheduling strategy to ensure that data blocks can be transmitted in the predetermined order and priority. This strategy can determine the transmission order of data blocks based on factors such as the urgency, size, and transmission speed of the data blocks.

[0040] The upload submodule needs to implement an efficient data block transmission process to ensure that the data blocks can be accurately and quickly transmitted to the target location. According to network conditions and user-side requirements, the upload submodule can select different transmission protocols to transmit data blocks. For example, for scenarios that require high reliability, the TCP protocol can be selected; for scenarios that require low latency, the UDP protocol can be selected. When selecting a transmission protocol, factors such as protocol overhead, compatibility, and security also need to be considered. During the transmission process, the upload submodule needs to encapsulate the data blocks into messages or data packets that meet the requirements of the transmission protocol, and add necessary metadata (such as data block identifiers, sequence numbers, checksums, etc.). Then, these messages or data packets are sent to the target location through the network. In order to ensure the integrity of the data, the upload submodule also needs to encrypt or sign the data before sending it. During the transmission process, the upload submodule needs to monitor the transmission status of each data block in real time, including the transmission speed, the amount of data transmitted, the remaining time, etc. At the same time, it is also necessary to receive feedback information from the target location to confirm whether the data block is successfully received and processed. If the target location fails to respond in time or feedbacks error information, the upload submodule needs to take corresponding retransmission or error handling measures.

[0041] In addition, to ensure reliable data transmission and timely processing, the upload submodule needs to monitor the transmission time of each data block and retransmit it when necessary. Users can set the transmission time threshold according to the needs of the application scenario and network conditions. This threshold determines the maximum allowed time for data block transmission. If the transmission time of a data block exceeds this threshold, the upload submodule needs to retransmit it. The upload submodule can calculate the transmission time by recording the sending time and receiving time of the data block. When sending a data block, the upload submodule records the current sending time; when receiving feedback information from the target location, the upload submodule records the current receiving time. Then, the transmission time is calculated by comparing these two times. When the transmission time of a data block exceeds the user-specified time threshold, the upload submodule needs to trigger the retransmission mechanism. This mechanism can select different retransmission strategies according to the user's configuration, such as immediate retransmission, delayed retransmission, or exponential backoff retransmission. During the retransmission process, the upload submodule needs to repackage the data block and send it to the target location, while updating the corresponding transmission status information. To ensure the reliability of retransmission, the upload submodule can also set parameters such as the number of retransmissions and timeout.

[0042] 1.3 Model Module The model module 103 is used to store the AI ​​model and the AI ​​algorithm. The AI ​​model is obtained by training the AI ​​algorithm using the training data in the data module 102.

[0043] The model module 103 is a module in the interaction layer 10 that is responsible for storing the AI ​​model and AI algorithm. The AI ​​model is obtained by training the AI ​​algorithm using the training data in the data module 102, and it contains the knowledge and rules learned from the training data. The model module 103 implements functions such as storage, loading, updating and version control of the model to ensure the validity and reliability of the model.

[0044] (1) Model storage Model storage is the core task of the model module 103, which implements the function of saving the trained AI model to the storage system. The implementation of model storage is as follows: The model module 103 supports saving the trained AI model as files (such as .h5, .pth, .onnx, etc.), and saving these files to a file storage system (such as HDFS, Ceph, etc.). For AI models that need to be frequently accessed and updated, the model module 103 supports saving the model to a database (such as MySQL, PostgreSQL, etc.). Database storage can speed up the loading of the model and improve the real-time performance of model reasoning. In order to save storage space and increase loading speed, the model module 103 supports compressing the trained AI model. Compression methods include pruning, quantization, distillation, etc.

[0045] (2) Model loading Model loading is another important task of the model module 103, which implements the function of loading the AI ​​model from the storage system according to the user's request. The implementation of model loading is as follows: For AI models stored in the file storage system, the model module 103 supports reading model data from the file and loading it into the memory. The loaded model can be used for subsequent reasoning tasks. For AI models stored in the database, the model module 103 supports retrieving model data from the database through SQL query statements and loading it into the memory. In order to improve the efficiency of model reasoning, the model module 103 supports optimizing the loaded AI model. Optimization methods include graph optimization, memory optimization, computational optimization, etc.

[0046] (3) Model update Model update is another key task of the model module 103, which implements the function of updating and optimizing the AI ​​model according to new training data. The implementation of model update is as follows: For an already trained AI model, the model module 103 supports incremental updates using new training data. Incremental updates can retain the original knowledge and rules of the model while learning new knowledge and rules. When the new training data differs greatly from the original training data, the model module 103 supports retraining the AI ​​model. Retraining can ensure that the model can accurately reflect the new data and rules.

[0047] 2. Resource layer The resource layer 20 is the core part of the AI ​​computing service, responsible for carrying and managing AI computing resources, and scheduling these resources to perform model reasoning or model training according to user application requests. The resource layer 20 is designed to provide efficient, scalable, and manageable AI computing services to meet the needs of different users and applications. The resource layer 20 is used to carry and manage AI computing resources, and schedule AI computing resources to perform model reasoning or model training through AI computing services. The resource layer 20 is the underlying infrastructure of the AI ​​computing service, which provides the necessary hardware and software resources to support AI model training and reasoning tasks. These resources include, but are not limited to, high-performance computing clusters, storage systems, network infrastructure, and AI frameworks and algorithm libraries.

[0048] In this embodiment, the resource layer 20 further includes: 2.1 Service Module The service module 201 is used to generate a corresponding AI computing service task based on the AI ​​computing resource remote call request initiated by the user application. The AI ​​computing service includes an AI model training service and an AI model reasoning service. The AI ​​model reasoning service includes an AI online reasoning service and an AI offline reasoning service. The service module 201 is a module in the resource layer 20 that is responsible for generating corresponding AI computing service tasks based on the AI ​​computing resource remote call request initiated by the user application. It receives user requests from the interaction layer 10, parses the parameters in the request, and generates corresponding AI computing service tasks according to the request type (model training or model reasoning). The service module 201 also provides AI model training services and AI model reasoning services, where the AI ​​model reasoning service includes AI online reasoning services and AI offline reasoning services.

[0049] (1) AI computing service task generation The process of service module 201 generating AI computing service tasks is as follows: Step 1, request reception and analysis: The service module 201 receives the user request from the interaction layer 10 through the API interface. The request is analyzed to extract key information in the request, such as task type (model training / model inference), model name (if specified), input data (training data / inference data), parameter configuration (such as learning rate, batch size, number of iterations, etc.).

[0050] Step 2: Task type determination: Based on the task type in the request, the service module 201 determines whether to generate an AI model training task or an AI model reasoning task.

[0051] Step 3, task generation: For AI model training tasks, service module 201 generates a training task including training data and algorithm configuration according to the parameter configuration and model algorithm in the request (obtained from model module 103). For AI model inference tasks, service module 201 further determines whether it is AI online inference or AI offline inference. Online inference usually requires low latency, while offline inference focuses more on batch processing and throughput. Based on the input data and model name (if specified) in the request, service module 201 generates an inference task including inference data and model configuration.

[0052] Step 4, task storage and distribution: The generated tasks are stored in the task queue, waiting for scheduling by the scheduling module 202. The service module 201 may also store the task information (such as task ID, status, priority, etc.) in the database for user query and management.

[0053] (2) AI model training service The AI ​​model training service is a key function provided by the service module 201, which allows users to use AI computing resources to train their own AI models. The training process usually includes the following steps: Step 1: Data preparation: The user uploads the training data to the data module 102. The service module 201 obtains the training data from the data module 102 and performs preprocessing (such as data cleaning, normalization, enhancement, etc.) as needed.

[0054] Step 2: Model selection: The user specifies the AI ​​algorithm to be used (such as convolutional neural network CNN, recurrent neural network RNN, deep learning framework TensorFlow, PyTorch, etc.), or selects a pre-trained model from the model library as a starting point. The service module 201 obtains the specified AI algorithm or pre-trained model from the model module 103.

[0055] Step 3: Training configuration: The user configures training parameters, such as learning rate, batch size, number of iterations, optimization algorithm, etc. The service module 201 packages these parameters together with the training data and model algorithm into a training task.

[0056] Step 4, resource scheduling and training execution: The service module 201 submits the training task to the scheduling module 202 for resource scheduling. The scheduling module 202 selects a suitable computing node to execute the training task based on the current availability of AI computing resources, task priority, and constraints (such as resource quotas, cost budgets, etc.). The computing node loads the training data and model algorithm, executes the training process, and records the training logs and performance indicators (such as accuracy, loss function value, etc.) in real time.

[0057] Step 5: Model evaluation and saving: After training is completed, the service module 201 can automatically evaluate the performance of the model (such as using a validation data set for testing). The evaluation results (including performance indicators and model files) are saved in the model module 103 for subsequent reasoning tasks.

[0058] (3) AI model reasoning service AI model inference service allows users to use trained AI models to predict or classify new input data. Depending on the application scenario, the inference service can be divided into AI online inference service and AI offline inference service.

[0059] AI online reasoning service: Online reasoning requires low latency and high throughput and is usually used in real-time application scenarios (such as video surveillance, autonomous driving, etc.). Service module 201 receives the reasoning request submitted by the user, including input data and model name (or ID). Load the specified AI model from model module 103 into the memory. After preprocessing the input data, send it to the model for reasoning calculation. The reasoning results (such as predicted category, probability distribution, etc.) are returned to the user application in real time.

[0060] AI offline reasoning service: Offline reasoning focuses more on batch processing and cost-effectiveness, and is usually used in non-real-time application scenarios (such as batch data analysis, image recognition, etc.). The user uploads the data set to be reasoned to the data module 102. The service module 201 generates offline reasoning tasks based on user requests and model configuration. The scheduling module 202 schedules AI computing resources to perform offline reasoning tasks, and after processing all input data, saves the reasoning results to the specified output location (such as a directory in the data module 102).

[0061] In one embodiment, the service module 201 further includes: (1) A strategy generation submodule, which is used to set the AI ​​computing resource selection strategy based on the split size of the business data to be calculated and the data processing speed of each AI computing resource; The split size of the business data to be calculated directly affects the selection of AI computing resources. If the data is split too large, some AI computing resources may not be able to complete the calculation within the specified time; if the data is split too small, it may increase the overhead of data transmission and scheduling. Therefore, the strategy generation submodule needs to conduct a detailed analysis of the split size of the business data. The data split size is usually determined according to the business scenario and data characteristics. For example, for image recognition tasks, the data split size can be set according to the resolution and complexity of the image; for natural language processing tasks, the data split size can be set according to the length and semantic structure of the text.

[0062] In practical applications, the data split size may need to be dynamically adjusted according to the actual situation of AI computing resources. For example, when AI computing resources are tight, the data split size can be appropriately increased to reduce the overhead of task scheduling and data transmission; when AI computing resources are sufficient, the data split size can be appropriately reduced to improve computing efficiency and accuracy. The strategy generation submodule needs to implement a dynamic adjustment mechanism to dynamically adjust the data split size according to the real-time status and data characteristics of AI computing resources. The data processing speed of AI computing resources is a key factor in selecting strategies. Different AI computing resources have different data processing capabilities and speeds, so it is necessary to formulate selection strategies based on business needs and the actual situation of AI computing resources. The strategy generation submodule needs to evaluate the data processing capabilities of various AI computing resources, including computing speed, memory size, storage capacity, etc. These evaluation results can be used as the basis for selecting strategies.

[0063] In addition, in order to understand the data processing speed of AI computing resources in real time, the strategy generation submodule needs to implement a monitoring mechanism. This mechanism can collect data processing speed information of AI computing resources regularly or in real time, and analyze and process it. Through monitoring, the strategy generation submodule can promptly discover performance bottlenecks and abnormal situations of AI computing resources, and take corresponding measures to adjust and optimize them. Based on the split size of the business data to be calculated and the data processing speed of each AI computing resource, the strategy generation submodule needs to formulate a reasonable AI computing resource selection strategy.

[0064] The formulation of the selection strategy should follow the following principles: First, efficiency, that is, select AI computing resources with fast data processing speed and strong computing power as much as possible; second, reliability, that is, ensure that the selected AI computing resources are stable and reliable and can complete the computing tasks on time; third, economy, that is, select AI computing resources with lower costs as much as possible under the premise of meeting performance and reliability requirements. The strategy generation submodule can formulate a variety of selection strategies based on business needs and the actual situation of AI computing resources. For example, a threshold can be set based on the relationship between data split size and data processing speed. When the data split size exceeds this threshold, AI computing resources with faster data processing speed are selected; when the data split size is less than this threshold, AI computing resources with slower data processing speed but lower cost can be selected. In addition, more complex selection strategies can be formulated based on factors such as the real-time load of AI computing resources and geographical location. With the development of business and the continuous changes in AI computing resources, the selection strategy also needs to be continuously updated and optimized. The strategy generation submodule can collect business data and performance information of AI computing resources regularly or in real time to evaluate and adjust the selection strategy. By continuously optimizing the selection strategy, the utilization efficiency and computing performance of AI computing resources can be further improved.

[0065] In an optional embodiment, the AI ​​computing resource selection strategy is set based on the following optimization objectives and constraints: ; ; Among them, T represents the total processing time of the business data to be processed, which is specifically the longest processing time of all AI computing resources (including data transmission time and processing time). The constraint is to minimize T. Z represents the total amount of business data to be processed. represents the amount of business data allocated to the i-th AI computing resource, m represents the total number of AI computing resources, Indicates data transmission parameters, It represents the data processing speed of the i-th AI computing resource, n represents the number of data blocks into which the business data to be processed is split, and p represents the average split size of the business data to be processed.

[0066] (2) Task generation submodule: This submodule is used to generate corresponding AI computing service tasks based on the remote call request for AI computing resources initiated by the user application and send them to the scheduling module.

[0067] User applications request AI computing services by initiating AI computing resource remote call requests. The task generation submodule needs to receive and process these requests to generate corresponding AI computing service tasks. The task generation submodule needs to receive AI computing resource remote call requests initiated by user applications and parse the requests. The parsing process includes extracting key information in the request, such as task type, data split size, computing requirements, etc. In order to ensure the validity and security of the request, the task generation submodule needs to verify and filter the received requests. The verification process includes checking whether the format of the request is correct and whether the parameters in the request are legal. The filtering process includes excluding invalid requests and malicious attacks.

[0068] Based on the request information of the user application, the task generation submodule needs to generate the corresponding AI computing service task. According to the request information of the user application, the task generation submodule can determine the type of task, such as image recognition, natural language processing, speech recognition, etc. Different types of tasks require different AI models and computing resources. The task generation submodule needs to configure the parameters of the task according to the request information of the user application, including data splitting size, calculation accuracy, model selection, etc. These parameters will directly affect the calculation efficiency and result accuracy of the task. In actual applications, there may be dependencies between certain tasks. For example, a complex AI computing task may require multiple subtasks to be completed before it can proceed. The task generation submodule needs to handle these dependencies to ensure that the tasks are executed in the correct order and manner.

[0069] After generating AI computing service tasks, the task generation submodule needs to send these tasks to the scheduling module for further scheduling and execution. The task generation submodule can choose different ways to send tasks to the scheduling module. For example, task information can be sent to the scheduling module through message queues, HTTP requests, etc. The choice of which way to send depends on factors such as the system architecture and communication protocol. In order to ensure that the scheduling module can correctly parse and execute tasks, the task generation submodule needs to encapsulate the task information. The encapsulation process includes packaging and transmitting information such as task type, parameter configuration, and dependencies in a predetermined format. In order to ensure that tasks can be smoothly sent to the scheduling module and executed correctly, the task generation submodule needs to implement a monitoring mechanism. This mechanism can track the status of task delivery and execution in real time, and take corresponding measures to handle exceptions.

[0070] 2.2 Scheduling Module The scheduling module 202 is a module in the resource layer 20 that is responsible for responding to the AI ​​computing service tasks of the service module 201, and scheduling AI computing resources to perform model reasoning or model training according to the AI ​​model or AI algorithm provided by the model module 103. It dynamically allocates and manages AI computing resources based on factors such as task priority, resource demand and availability to ensure efficient execution of tasks. The scheduling module 202 is used to respond to the AI ​​computing service tasks of the service module 201, and schedule AI computing resources to perform model reasoning or model training according to the AI ​​model or AI algorithm provided by the model module 103.

[0071] (1) Resource management and monitoring The scheduling module 202 needs to effectively manage and monitor AI computing resources to ensure full utilization of resources and smooth execution of tasks. This includes the following aspects: Resource discovery and registration: The scheduling module 202 automatically discovers and registers available AI computing resources (such as GPUs, TPUs, etc.) through resource discovery mechanisms (such as DNS, Kubernetes services, etc.). Each resource node contains a resource agent that is responsible for reporting its status, performance, availability, and other information to the scheduling module 202.

[0072] Resource monitoring and alarm: The scheduling module 202 continuously monitors the status (such as CPU usage, memory usage, GPU utilization, etc.) and performance indicators (such as throughput, latency, etc.) of each resource node. When the status or performance indicators of a resource node exceed a preset threshold, the scheduling module 202 triggers an alarm mechanism to notify the administrator or automatically perform corresponding recovery operations.

[0073] Resource quotas and limits: The scheduling module 202 sets resource quotas and limits for each user or user group to ensure fair allocation of resources and prevent resource abuse. Users can apply for additional resource quotas as needed, and the scheduling module 202 approves them based on resource usage and approval process.

[0074] (2) Task Scheduling and Resource Management The task scheduling and resource management process of the scheduling module 202 is as follows: Task reception and analysis: The scheduling module 202 receives AI computing service tasks from the service module 201, including task type (model training / model inference), task priority, resource requirements (such as the number of GPUs, memory size, etc.), and task dependencies, etc. The tasks are analyzed and preprocessed to generate executable scheduling tasks.

[0075] Resource evaluation and selection: The scheduling module 202 evaluates whether each resource node meets the task requirements based on the status, performance, availability and other information of the current resource node. According to the priority of the task, the resource requirements and constraints (such as resource quota, cost budget, etc.), the appropriate resource node is selected for task scheduling.

[0076] Task distribution and execution: The scheduling module 202 distributes the task to the selected resource node for execution. After receiving the task, the task executor on the resource node loads the necessary AI model or algorithm, input data, etc., and executes the training or reasoning process. During the task execution process, the scheduling module 202 continuously monitors the execution status and performance indicators of the task to ensure the smooth execution of the task.

[0077] Task management and optimization: The scheduling module 202 provides task management functions, allowing users to query task status, view task logs, terminate tasks, etc. According to the execution of tasks and the use of resources, the scheduling module 202 can also optimize tasks, such as dynamically adjusting resource allocation and optimizing the order of task execution.

[0078] Resource recovery and release: When the task is completed, the scheduling module 202 is responsible for recovering and releasing the occupied resources. For resource nodes that have not been used for a long time, the scheduling module 202 can put them into low power consumption mode or remove them from the resource pool to save energy and reduce costs.

[0079] In one embodiment of the present invention, in the construction of the AI ​​computing power service system, the resource layer 20 plays a vital role. It not only carries out the effective management and scheduling of AI computing power resources, but also is responsible for the efficient transmission and processing of data. In this embodiment, the design of the resource layer 20 is further refined, and multiple data exchange areas are provided, each of which is provided with multiple data transmission links connecting the data exchange area and the AI ​​computing power resources; multiple data sub-areas are provided in the data exchange area, and each data sub-area is provided with an AI model training queue group and an AI model reasoning queue group; the AI ​​model training queue group contains multiple AI model training queues, and the AI ​​model reasoning queue group contains multiple AI model reasoning queues; the AI ​​model training queue group and the AI ​​model reasoning queue group in the same data sub-area are connected in series in sequence through the queue exchange area; The AI ​​model training queue establishes a communication connection with the AI ​​model training resources in the AI ​​computing power resources through the corresponding data transmission link, and the AI ​​model inference queue establishes a communication connection with the AI ​​model inference resources in the AI ​​computing power resources through the corresponding data transmission link.

[0080] The resource layer is the core infrastructure of AI computing services. It integrates high-performance computing resources, storage resources, network resources, and AI algorithms and frameworks to provide users with efficient AI model training and reasoning services. The design goal of the resource layer is to maximize the utilization of resources and ensure the efficient execution of tasks, while providing a flexible and scalable architecture to adapt to changing business needs. Among them, the data exchange area plays a vital role as a hub for data flow. The data exchange area is a key area in the resource layer for data temporary storage and transmission. By setting up multiple data transmission links, it realizes efficient communication between data sub-areas and AI computing resources.

[0081] The data exchange area is an independent area in the resource layer for storing and managing business data. It supports multiple data types and data formats, including images, videos, text, audio, etc. The design goal of the data exchange area is to achieve fast data transmission and efficient processing to meet the real-time requirements of AI model training and reasoning. The data transmission link is the communication channel between the data exchange area and the AI ​​computing resources. Each data exchange area is equipped with multiple data transmission links to ensure that data can be efficiently transmitted between different data sub-areas and AI computing resources. The design of the data transmission link takes into account a variety of factors, including bandwidth, latency, reliability, etc. In order to optimize data transmission performance, high-performance network technologies and equipment such as Ethernet switches and fiber optic transmission can be used. At the same time, the data transmission link also supports a variety of communication protocols and data formats to meet different business needs.

[0082] The data sub-zone is a logical unit in the data exchange zone, which is used to store and manage specific types of business data. Each data sub-zone is set up with an AI model training queue group and an AI model inference queue group to achieve classified storage and efficient processing of data. AI model training queue group: contains multiple AI model training queues for storing and managing model data to be trained. Each training queue corresponds to a specific AI model training task to ensure that the data can be processed in an orderly manner according to task requirements. AI model inference queue group: contains multiple AI model inference queues for storing and managing business data to be inferred. Each inference queue corresponds to a specific AI model inference task to ensure that the data can be processed quickly according to task requirements.

[0083] The queue exchange area is a special area in the data sub-area, which is used to realize data exchange between the AI ​​model training queue group and the AI ​​model inference queue group. The design of the queue exchange area can ensure that data flows in an orderly manner between training tasks and inference tasks, thereby realizing continuous processing and efficient use of data. The design of the queue exchange area takes into account a variety of factors, including data throughput, processing speed, data integrity, etc. In order to achieve efficient data exchange, high-performance data processing technologies and equipment can be used, such as distributed storage systems, data caching technologies, etc.

[0084] The data module 102 is a module in the interaction layer 10 for managing and processing business data. In addition to the basic data upload, download and storage functions, the data module 102 also extends the following functions: Business data upload request processing: The data module 102 can initiate a business data upload request to the resource layer 20, thereby storing the business data in the AI ​​model training queue group or AI model reasoning queue group in the corresponding data sub-area.

[0085] Data synchronization and backup: In order to ensure the reliability and security of data, the data module 102 also implements data synchronization and backup functions. It can synchronize data to multiple storage nodes to prevent data loss or damage. At the same time, the data module 102 can also regularly back up data to remote storage devices to ensure long-term storage and recoverability of data.

[0086] In this embodiment, the data module 102 is further used to initiate a business data upload request to the corresponding AI model training queue group or AI model reasoning queue group to store the uploaded business data in the queue of the corresponding AI model training queue group or AI model reasoning queue group; The scheduling module 202 is also used to transmit the business data stored in the queue of the AI ​​model training queue group or the AI ​​model reasoning queue group to the corresponding AI computing power resource through the data transmission link to perform model reasoning or model training. The scheduling module 202 is a module in the resource layer for managing and scheduling AI computing power resources. In addition to the basic resource allocation and task scheduling functions, the scheduling module 202 also extends the following functions: Task priority management: The scheduling module 202 can dynamically adjust the execution order of tasks and resource allocation according to the priority of tasks and resource requirements. For high-priority tasks, the scheduling module 202 can prioritize resource allocation to ensure that tasks can be completed as soon as possible. At the same time, for low-priority tasks, the scheduling module 202 can put them in a waiting queue and wait for resources to be free before scheduling.

[0087] Resource status monitoring and early warning: The scheduling module 202 can monitor the status of AI computing resources in real time, including CPU usage, memory occupancy, GPU utilization, etc. When the resource status reaches the preset threshold, the scheduling module 202 can trigger the early warning mechanism to notify the administrator or automatically perform the corresponding recovery operation. This helps to timely discover and solve resource bottleneck problems and ensure the stability and reliability of the system.

[0088] Business data storage and transmission management: The scheduling module 202 can manage the business data stored in the AI ​​model training queue group or the AI ​​model reasoning queue group. When the AI ​​model training resources or AI model reasoning resources in the AI ​​computing power resources are busy, the scheduling module 202 can also store the received business data in a temporary storage area. At the same time, the scheduling module 202 can also transmit the data stored in the queue to the corresponding AI computing power resources through the corresponding data transmission link to perform model reasoning or model training according to task requirements and resource status, which helps to achieve continuous processing and efficient use of data.

[0089] The temporary storage area is an important component in the resource layer. It is used to store business data that cannot be processed immediately, and establishes a communication connection with the AI ​​computing power resources through the corresponding data transmission link. When the AI ​​computing power resources are busy, the scheduling module 202 can store the received business data in the temporary storage area and wait for the resources to be idle before processing. The temporary storage area is a high-performance storage system that supports fast reading and writing and efficient data management. It adopts a distributed storage architecture to store data in multiple storage nodes to improve data reliability and scalability. At the same time, the temporary storage area also supports a variety of data formats and communication protocols to meet different business needs.

[0090] The data storage and transmission mechanism of the temporary storage area is as follows: When the AI ​​computing resources are busy, the scheduling module 202 stores the received business data in a temporary storage area. During the storage process, the scheduling module 202 selects a suitable storage node and storage strategy according to the type and size of the data. At the same time, the scheduling module 202 also generates a unique identifier (such as a data ID) for the stored data for subsequent data retrieval and processing.

[0091] When AI computing resources are idle, the scheduling module 202 will retrieve the data to be processed from the temporary storage area according to the task requirements and resource status. During the retrieval process, the scheduling module 202 will read the data from the corresponding storage node according to the data identifier and storage strategy. At the same time, the scheduling module 202 will also perform format conversion and preprocessing on the read data to meet the needs of AI model training and reasoning.

[0092] The scheduling module 202 transmits the data stored in the temporary storage area to the corresponding AI computing resources through the corresponding data transmission link to perform model reasoning or model training. During the transmission process, the scheduling module 202 will select the appropriate transmission protocol and bandwidth resources according to the type and size of the data. At the same time, the scheduling module 202 will also encrypt and verify the transmitted data to ensure the integrity and security of the data.

[0093] In this embodiment, the workflow of the resource layer 20 includes multiple stages such as data uploading, data processing, and task scheduling.

[0094] (1) In the data upload phase, the user or application uploads the data to the data module 102 of the resource layer through the interaction layer. After receiving the uploaded data, the data module 102 stores it in the AI ​​model training queue group or AI model reasoning queue group in the corresponding data sub-area according to the type and size of the data. At the same time, the data module 102 also pre-processes and converts the uploaded data to meet the needs of AI model training and reasoning.

[0095] (2) In the data processing stage, the scheduling module 202 retrieves the data to be processed from the AI ​​model training queue group or the AI ​​model reasoning queue group according to the task requirements and resource status. The scheduling module 202 reads the data from the corresponding storage node according to the data identifier and storage strategy, and performs format conversion and preprocessing. Then, the scheduling module 202 transmits the processed data to the corresponding AI computing power resource through the corresponding data transmission link to perform model reasoning or model training.

[0096] When the AI ​​computing resources are busy, the scheduling module 202 will store the received business data in the temporary storage area. When the AI ​​computing resources are idle, the scheduling module 202 will retrieve the data to be processed from the temporary storage area and process it according to the above process.

[0097] (3) During the task scheduling phase, the scheduling module 202 dynamically adjusts the execution order of tasks and resource allocation according to the task priority and resource demand. The scheduling module 202 monitors the status of AI computing resources, including CPU usage, memory occupancy, GPU utilization, etc. When the resource status reaches the preset threshold, the scheduling module 202 triggers the early warning mechanism and takes corresponding recovery actions.

[0098] In an optional embodiment, all queue groups in multiple data exchange areas of the resource layer 20 form a data transmission belt in parallel, and each data exchange area is composed of multiple queue groups; when a business data upload request is initiated to the corresponding queue group, the data module will match a data transmission link with each queue in the queue group, and at the same time establish a communication connection with the AI ​​computing power resources and transmit the business data to each AI computing power resource in turn according to the matched data transmission link to execute model reasoning or model training.

[0099] The resource layer 20 is composed of multiple data exchange areas, which form an efficient data transmission belt in parallel. Each data exchange area contains multiple queue groups, which are responsible for temporarily storing and forwarding business data from different data modules. The data exchange area is responsible for storing and managing business data, and providing an interface for data upload and download for the data module 102. Each data exchange area has its own storage space and access rights to ensure the security and isolation of data. The queue group is a logical unit in the data exchange area, which contains multiple queues for storing business data and waiting for processing by AI computing resources. Each queue has its own data storage space and processing logic, and can independently handle the upload and transmission of business data. These queues can be queues based on memory, disk or other storage media, used to temporarily store data waiting to be processed or transmitted.

[0100] The multiple data exchange areas and queue groups in the resource layer 20 form an efficient data transmission belt through parallel processing. This transmission belt can handle a large number of business data upload requests and quickly transmit the data to the AI ​​computing power resources. The data exchange area and the queue group adopt parallel processing to process multiple business data upload requests at the same time. This can make full use of the processing power of the system and improve the efficiency of data upload and transmission. When the data module 102 initiates a business data upload request to the corresponding queue group, the resource layer 20 will match a data transmission link for each queue. This link is responsible for transmitting data from the data module 102 to the queue group and waiting for processing by the AI ​​computing power resources. A communication connection is established between the resource layer 20 and the AI ​​computing power resources to ensure rapid data transmission and synchronous processing. This connection can be a network connection based on TCP / IP or a local connection based on shared memory or message queues.

[0101] In order to achieve efficient data transmission, the resource layer 20 adopts a parallel processing mechanism. When the data module initiates a business data upload request to the resource layer 20, the resource layer 20 will dynamically allocate multiple queue groups for parallel processing according to the needs of the data module and the amount of data. Each queue group is responsible for processing a part of the data, thereby speeding up the overall data transmission speed. A data transmission link needs to be established between the data module and the queue group. These links can be communication links based on network protocols (such as TCP / IP, UDP, etc.), or they can be links based on local communication mechanisms such as shared memory and message queues. The data module transmits the business data to each queue group in turn through these links.

[0102] In this embodiment, when the data module 102 initiates a service data upload request to the resource layer 20, the resource layer 20 processes the request according to the following steps: Step 1, receiving a request: the resource layer 20 first receives a service data upload request sent by the data module 102, and parses the information in the request, such as data size, type, target queue group, etc.

[0103] Step 2: Matching a data transmission link: Based on the information in the request, the resource layer 20 matches a data transmission link for each queue. This link can be a link based on network transmission or a link based on memory replication.

[0104] Step 3: Establish a communication connection: Establish a communication connection between the resource layer 20 and the AI ​​computing resources to ensure synchronous data transmission and processing. This connection can be a network connection based on TCP / IP or a local connection based on shared memory or message queues.

[0105] Step 4, data transmission: After the communication connection is established, the resource layer 20 will transmit the service data from the data module 102 to the corresponding queue group according to the matching data transmission link. During the transmission process, the resource layer 20 will also check and error handle the data to ensure the integrity and reliability of the data.

[0106] Step 5: Waiting for processing: After the business data is transferred to the queue group, it will wait for AI computing resources to process it. AI computing resources will take data out of the queue group for processing based on the model's reasoning or training requirements.

[0107] Step 6: Feedback of processing results: After the AI ​​computing resources process the data, they will feed back the processing results to the resource layer 20. The resource layer 20 will store the processing results in a specified location and notify the data module 102 or the user module to download or query the processing results.

[0108] To improve the efficiency and reliability of data transmission, the resource layer 20 may also adopt the following optimization strategies: Data compression strategy: Before data transmission, business data can be compressed to reduce the amount of data transmission and increase the transmission speed. At the same time, compression can also reduce storage space occupation and reduce storage costs.

[0109] Data caching strategy: In order to improve the response speed of data transmission, data caching can be set in the resource layer 20. When the data module 102 initiates an upload request, the data can be stored in the cache first, and then transmitted according to the processing capacity and requirements of the AI ​​computing resources.

[0110] Load balancing strategy: When multiple data modules 102 initiate upload requests at the same time, the resource layer 20 can use a load balancing strategy to allocate data transmission tasks, which can avoid overloading of a certain data exchange area or queue group and improve the overall performance and stability of the system.

[0111] Data verification and error handling strategy: During data transmission, the resource layer 20 needs to verify and handle errors on the data. If data errors or loss are found, a retransmission mechanism or data recovery strategy can be used to ensure data integrity and reliability.

[0112] In an optional embodiment, the scheduling module 202 is also used to perform data preprocessing on the uploaded business data in parallel in the queue exchange area in each data sub-area and store it in the corresponding queue when the data module initiates a business data upload request. The data preprocessing function of the scheduling module 202 includes one or more of batch data storage, data dimension reduction, data compression, data format conversion, etc. These preprocessing operations can improve the quality and processing efficiency of the data, and facilitate the subsequent data processing and AI computing resource processing.

[0113] (1) Batch data storage: Combining multiple business data into a batch for storage and transmission can improve data processing efficiency and reduce transmission overhead.

[0114] (2) Data dimensionality reduction: Reducing the complexity of data by reducing the number of features or dimensions can speed up subsequent data processing and model reasoning / training.

[0115] (3) Data compression: Compressing business data can reduce the amount of data transmitted and the storage space occupied, thereby improving transmission speed and storage efficiency.

[0116] (4) Data format conversion: Convert business data from one format to another to adapt to different processing requirements and AI algorithms. For example, convert image data from JPEG format to a format supported by TensorFlow or PyTorch.

[0117] When the data module 102 initiates a business data upload request, the scheduling module 202 performs data preprocessing according to the following steps: Step 1, receiving request and parsing: the scheduling module 202 first receives the business data upload request sent by the data module 102, and parses the information in the request, such as data size, type, target queue group, etc.

[0118] Step 2, select preprocessing operation: According to the type and requirements of the business data, the scheduling module 202 will select appropriate preprocessing operations. For example, for image data, data compression and data format conversion can be selected; for text data, data dimension reduction and batch data storage can be selected.

[0119] Step 3: Execute preprocessing operations: After selecting the preprocessing operations, the scheduling module 202 will call the corresponding preprocessing functions or modules to execute the operations. These operations can be executed on the CPU, or on the GPU or other acceleration hardware to increase the processing speed.

[0120] Step 4: Storing processed data: The pre-processed data will be stored in the corresponding queue. The scheduling module 202 will write the data to the specified location according to the address and storage space of the target queue group.

[0121] Step 5: Update metadata and status: When storing processed data, the scheduling module 202 will also update related metadata and status information, such as recording the processing time, processing results, storage location, etc. of the data. This information can be used for subsequent data processing, query, error handling, and other operations.

[0122] In order to improve the efficiency of data preprocessing, the scheduling module 202 may adopt a parallel processing method, which may be achieved through multi-threading, multi-process or distributed computing technologies.

[0123] Multithreaded processing: The scheduling module 202 can create multiple threads in each data sub-area to process business data in parallel. Each thread can be responsible for one or more pre-processing operations and store the processed data in the corresponding queue.

[0124] Multi-process processing: In addition to multi-thread processing, the scheduling module 202 can also adopt a multi-process processing method. Each process can independently process a set of business data and communicate and synchronize with other processes. This can avoid competition and deadlock problems between threads and improve the stability and scalability of the system.

[0125] Distributed computing: For large-scale data processing tasks, the scheduling module 202 can also adopt a distributed computing approach. It splits the task into multiple subtasks and assigns these subtasks to different computing nodes for processing. The computing nodes can share data and results through network communication to achieve efficient distributed computing and data processing.

[0126] Reference Figure 2 The embodiment of the present invention further provides a big data direct access method, which adopts the AI ​​computing resource remote calling framework described in any one of the above items, and the big data direct access method includes: S1. The user module receives an AI computing resource remote call request and corresponding business data to be processed initiated by a user application on the user side, and saves the business data through the data module; The user module is the front-end interface of the system, responsible for receiving requests initiated by user applications on the user side. These requests usually include remote call requests for AI computing resources and business data to be processed. User applications are client software on the user side, used to initiate remote call requests for AI computing resources. AI computing resource remote call requests are tasks initiated by user applications to the system to request AI computing resources for data processing. The request usually contains information such as the type of task (such as model inference, model training, etc.), the AI ​​model or algorithm required for the task, and the format and size of the input data.

[0127] Business data is the data that the user application needs to process. It can be any type of data such as text, image, video, audio, etc. This data will be sent to the data module for storage and processing. The data module is the data storage and management center of the system, responsible for receiving and saving business data sent by user applications. The data module can use storage technologies such as distributed file systems (such as HDFS), relational databases (such as MySQL), and non-relational databases (such as MongoDB) to store and manage data.

[0128] After the user module receives the request from the user application, it first performs request verification and permission check to ensure that the request is legal and valid. After the verification is passed, the user module sends the business data in the request to the data module for storage. After the data module receives the data, it selects an appropriate storage method to save it according to the type and size of the data. At the same time, the user module saves the detailed information of the request (such as task type, AI model or algorithm, format and size of input data, etc.) to the system's task queue, waiting for subsequent processing.

[0129] S2, the data module splits the business data to obtain multiple data blocks, and initiates a request to upload the business data to the scheduling module; After receiving the business data, the data module splits it according to the type and size of the data. For large data sets, it can be split into multiple smaller data blocks for more efficient processing and transmission. Data splitting can be based on the logical structure of the data (such as paragraphs of text files, regions of image files, etc.) or physical structure (such as file size, number of data blocks, etc.). The data block is the data unit after splitting, and each data block contains a part of the original data. The size of the data block can be dynamically adjusted according to the system's processing power and network bandwidth.

[0130] The scheduling module is the task scheduling center of the system, responsible for receiving and processing task requests from other modules, and scheduling and allocating tasks according to task requirements and resource status. After splitting the data, the data module initiates an upload request for business data to the scheduling module. The upload request usually contains the identification information of the data block (such as the serial number, size, location, etc. of the data block), task type, target AI computing resources, and other information.

[0131] After splitting the data, the data module generates unique identification information for each data block and stores it together with the data block. Then, the data module selects the appropriate upload method and path according to the system configuration and strategy, and initiates an upload request to the scheduling module. After receiving the upload request, the scheduling module performs task scheduling and resource allocation based on information such as the task type and target AI computing resources in the request. If there are multiple AI computing resources available in the system, the scheduling module can select the appropriate AI computing resources for task allocation based on the AI ​​computing resource selection strategy (such as load balancing, priority sorting, etc.). The scheduling module can also dynamically adjust the concurrency and priority of the upload task according to the size and number of data blocks to ensure efficient execution of tasks and full utilization of resources.

[0132] S3. The service module generates a corresponding AI computing service task based on the AI ​​computing resource remote call request and sends it to the scheduling module. The AI ​​computing service task includes an AI computing resource selection strategy and an adopted AI computing service. The service module is the service management center of the system, responsible for receiving and processing AI computing resource remote call requests from the user module, and generating corresponding AI computing service tasks based on the requests. AI computing resource remote call requests are tasks sent by the user module to the service module to request AI computing resources for data processing. The request usually contains information such as the type of task (such as model inference, model training, etc.), the AI ​​model or algorithm required for the task, and the format and size of the input data.

[0133] AI computing service tasks are generated by the service module based on the remote call request of AI computing resources, and are used to describe and execute specific information of data processing tasks. Tasks usually include the type of task, identification information of input data, identification information of AI models or algorithms, selection strategy of AI computing resources, etc. AI computing resource selection strategy is the strategy formulated by the service module based on system resources and task requirements when generating AI computing service tasks. Strategies can be formulated based on a variety of factors, such as the availability, performance, and cost of AI computing resources. AI computing service is a service provided by the system for performing data processing tasks. Services can be implemented based on different AI models or algorithms, such as deep learning models, machine learning algorithms, etc.

[0134] After the service module receives the remote call request for AI computing resources from the user module, it first parses and verifies the request to ensure that the request is legal and valid. Then, the service module generates the corresponding AI computing service task based on the task type, AI model or algorithm and other information in the request. When generating tasks, the service module will formulate appropriate AI computing resource selection strategies based on the system's resources and task requirements. Strategies can be formulated based on factors such as load balancing, priority sorting, and cost optimization. At the same time, the service module will select appropriate AI computing services for task execution based on the task type and the requirements of the AI ​​model or algorithm. The service can be a built-in AI model or algorithm in the system, or a user-defined model or algorithm. Finally, the service module sends the generated AI computing service task to the scheduling module for task scheduling and allocation.

[0135] S4. The scheduling module responds to the AI ​​computing power service task of the service module, obtains the AI ​​model or AI algorithm provided by the model module, and schedules the corresponding AI computing power resources to perform model reasoning or model training on the business data uploaded by the data module according to the AI ​​computing power resource selection strategy.

[0136] The scheduling module is the task scheduling center of the system, responsible for receiving and processing AI computing service tasks from the service module, and scheduling and allocating tasks according to task requirements and resource status. AI computing service tasks are generated by the service module and are used to describe and execute specific information of data processing tasks. Tasks usually include the type of task, identification information of input data, identification information of AI models or algorithms, and selection strategies for AI computing resources.

[0137] The model module is the AI ​​model or algorithm management center of the system, responsible for storing and managing the AI ​​models or algorithms built into the system, as well as user-defined models or algorithms. AI models or algorithms are tools provided by the system to perform data processing tasks. Models can be deep learning models, machine learning algorithms, etc., and algorithms can be image processing algorithms, natural language processing algorithms, etc. AI computing resources are computing resources provided by the system to perform data processing tasks. Resources can be different types of computing units such as CPU, GPU, FPGA, or different types of computing devices such as cloud servers and edge computing devices. Model inference is the process of processing and analyzing input data using a trained AI model, and model training is the process of training and optimizing an AI model using training data.

[0138] After the scheduling module receives the AI ​​computing service task from the service module, it first performs task analysis and verification to ensure that the task is legal and valid. Then, the scheduling module obtains the corresponding AI model or algorithm from the model module according to the identification information of the AI ​​model or algorithm in the task. When obtaining the model or algorithm, the scheduling module will perform necessary model loading and initialization operations to ensure that the model or algorithm can be executed normally. The scheduling module selects appropriate AI computing resources for task scheduling and allocation based on the AI ​​computing resource selection strategy in the task. The scheduling process can be formulated based on factors such as load balancing, priority sorting, and cost optimization. When scheduling resources, the scheduling module will consider factors such as the availability, performance, and cost of AI computing resources, and dynamically adjust according to the requirements of the task and the status of the resources. Finally, the scheduling module assigns the task to the selected AI computing resources and starts the task execution. During the execution process, the scheduling module monitors the execution status and progress of the task, and makes necessary adjustments and optimizations as needed.

[0139] A remote call framework for AI computing resources proposed in this embodiment includes: an interaction layer and a resource layer; the interaction layer is used to interact with user applications on the user side, and the user application adopts a programming mode and calls the AI ​​computing service of the resource layer through the API interface provided by the interaction layer to perform model training or model reasoning; the resource layer is used to carry and manage AI computing resources, and schedule AI computing resources through AI computing services to perform model reasoning or model training. This embodiment adopts a layered design, which not only provides a more convenient and easy-to-use development and application scenario for computing resource users and AI development users, but also realizes the remote call of AI computing resources through the dynamic scheduling of AI computing resources. In actual applications, the framework can be deployed on multiple heterogeneous computing platforms such as the cloud, edge, and device, further improving the scalability and scalability of the AI ​​computing resource platform. The present invention can not only independently provide the computing infrastructure to users, so that users can schedule and remotely call computing resources on demand according to actual application needs, but also enable developers to quickly build artificial intelligence application models.

[0140] This embodiment realizes efficient processing of big data in the process of remote calling calculation of AI computing power resources, and has good applicability to big data scenarios of AI computing power resource calls. At the same time, an AI computing power resource selection strategy can be formulated according to the data processing speed of the AI ​​computing power resources and the size of the split data blocks, thereby improving the utilization rate of AI computing power resources, reducing unnecessary resource consumption, and saving hardware costs. The big data direct-access method based on the remote calling scenario of AI computing power resources provided in this embodiment realizes the joint computing of multiple AI computing powers, reduces data transmission, circulation and storage, improves the direct-access efficiency of big data, optimizes the remote calling of AI computing power resources, realizes multi-level sharing of AI computing power resources, and improves data processing efficiency on the premise of saving equipment costs.

[0141] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote call framework for AI computing resources, characterized in that: The AI ​​computing power resource remote call framework includes: an interaction layer and a resource layer; The interaction layer is used to interact with the user application on the user side. The user application adopts a programming mode and calls the AI ​​computing service of the resource layer through the API interface provided by the interaction layer to perform model training or model reasoning; the resource layer is used to carry and manage AI computing resources, and schedule AI computing resources to perform model reasoning or model training through the AI ​​computing service; The interaction layer includes a user module, a data module and a model module. The user module is used to receive a remote call request for AI computing resources initiated by a user application on the user side and return a response result; the data module is used to save the business data uploaded by the user application through the user module, and the business data includes training data or inference data; the model module is used to save the AI ​​model and AI algorithm, and the AI ​​model is obtained by training the AI ​​algorithm with the training data in the data module; The resource layer includes a service module and a scheduling module. The service module is used to generate corresponding AI computing service tasks based on the AI ​​computing resource remote call request initiated by the user application. The AI ​​computing service includes AI model training service and AI model reasoning service. The AI ​​model reasoning service includes AI online reasoning service and AI offline reasoning service. The scheduling module is used to respond to the AI ​​computing service tasks of the service module, and schedule AI computing resources to perform model reasoning or model training according to the AI ​​model or AI algorithm provided by the model module.

2. The AI ​​computing resource remote calling framework according to claim 1, characterized in that: The resource layer is further provided with a plurality of data exchange areas, each of which is provided with a plurality of data transmission links connecting the data exchange area and the AI ​​computing power resources; The data exchange area is provided with a plurality of data sub-areas, each of which is provided with an AI model training queue group and an AI model inference queue group; the AI ​​model training queue group includes a plurality of AI model training queues, and the AI ​​model inference queue group includes a plurality of AI model inference queues; the AI ​​model training queue group and the AI ​​model inference queue group in the same data sub-area are connected in series in sequence through the queue exchange area; The AI ​​model training queue establishes a communication connection with the AI ​​model training resources in the AI ​​computing power resources through a corresponding data transmission link, and the AI ​​model inference queue establishes a communication connection with the AI ​​model inference resources in the AI ​​computing power resources through a corresponding data transmission link.

3. The AI ​​computing resource remote calling framework according to claim 2, characterized in that: The data module is also used to initiate a business data upload request to the corresponding AI model training queue group or AI model reasoning queue group to store the uploaded business data in the queue of the corresponding AI model training queue group or AI model reasoning queue group; The scheduling module is also used to transmit the business data stored in the queues of the AI ​​model training queue group or the AI ​​model reasoning queue group to the corresponding AI computing power resources through the data transmission link to perform model reasoning or model training.

4. The AI ​​computing resource remote calling framework according to claim 3 is characterized in that: When the AI ​​model training resources or AI model reasoning resources in the AI ​​computing power resources are in a busy state, the scheduling module is also used to store the received business data in a temporary storage area; the temporary storage area establishes a communication connection with the AI ​​computing power resources through a corresponding data transmission link.

5. The AI ​​computing resource remote calling framework according to claim 3 is characterized in that: All queue groups in the multiple data exchange areas of the resource layer form a data transmission belt in parallel, and each data exchange area is composed of multiple queue groups; when a business data upload request is initiated to the corresponding queue group, the data module will match a data transmission link with each queue in the queue group, and at the same time establish a communication connection with the AI ​​computing power resources and transmit the business data to each AI computing power resource in turn according to the matched data transmission link to execute model reasoning or model training.

6. The AI ​​computing resource remote calling framework according to claim 3, characterized in that: The scheduling module is also used to perform data preprocessing on the uploaded business data in parallel in the queue exchange area within each data sub-area and store the data in the corresponding queue when the data module initiates a business data upload request. The data preprocessing includes one or more of batch data storage, data dimensionality reduction, data compression, and data format conversion.

7. The AI ​​computing resource remote calling framework according to claim 1, characterized in that: The data module includes: A splitting submodule is used to split the business data to be calculated into multiple data blocks of the same size according to the data splitting rule set by the user side, wherein the data splitting size is related to the data transmission speed; The upload submodule is used to mark the transmission time and data identifier of each data block, and transmit each data block through the queue. After the data transmission is completed, it is determined whether the transmission time of each data block exceeds the user-specified time. If so, the data is transmitted again.

8. The AI ​​computing resource remote calling framework according to claim 7, characterized in that: The service module includes: The strategy generation submodule is used to set the AI ​​computing resource selection strategy based on the split size of the business data to be calculated and the data processing speed of each AI computing resource; The task generation submodule is used to generate the corresponding AI computing power service task based on the remote call request of the AI ​​computing power resource initiated by the user application and send it to the scheduling module.

9. The AI ​​computing resource remote calling framework according to claim 8, characterized in that: The AI ​​computing resource selection strategy is set based on the following optimization objectives and constraints: ； ； Where T represents the total processing time of the business data to be processed, Z represents the total amount of business data to be processed, represents the amount of business data allocated to the i-th AI computing resource, m represents the total number of AI computing resources, Indicates data transmission parameters, It represents the data processing speed of the i-th AI computing resource, n represents the number of data blocks into which the business data to be processed is split, and p represents the average split size of the business data to be processed.

10. A big data direct access method, characterized in that: Using the AI ​​computing resource remote calling framework as described in any one of claims 1 to 9, the big data direct access method includes: The user module receives a remote call request for AI computing resources initiated by a user application on the user side and corresponding business data to be processed, and saves the business data through the data module; The data module splits the business data into multiple data blocks, and initiates a request to upload the business data to the scheduling module; The service module generates a corresponding AI computing service task based on the AI ​​computing resource remote call request and sends it to the scheduling module, wherein the AI ​​computing service task includes an AI computing resource selection strategy and an adopted AI computing service; The scheduling module responds to the AI ​​computing power service task of the service module, obtains the AI ​​model or AI algorithm provided by the model module, and schedules the corresponding AI computing power resources to perform model reasoning or model training on the business data uploaded by the data module according to the AI ​​computing power resource selection strategy.

Images