Model storage management method and system and server

Abstract

The invention provides a model storage management method and system and a server, and relates to the technical field of computers.The method comprises the steps that a to-be-stored model file is packaged through an algorithm framework adapter module, and packaged target storage content is determined; a storage management module is used for conducting storage management on the target storage content and determining the actual storage position of the target storage content, the storage management module comprises an index storage unit, a version information storage unit and a file information storage unit, and the index storage unit comprises a global index storage unit and a local index storage unit; according to the actual storage position and the target underlying storage type received by the storage interface module, the target storage content is stored, and index change during model storage is synchronized to a database through a mixed storage framework in the index management module, so that model query and retrieval are completed. According to the method, the adaptation difficulty of algorithm model storage can be remarkably reduced, and the flexibility of model management is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model storage management method, system and server. Background Art

[0002] In terms of model training, deployment, and maintenance, how to efficiently store algorithm models to facilitate subsequent inference services and continuous optimization is a key issue. First, to meet rapidly developing business needs, the storage system must be able to support low-latency data reading to ensure the responsiveness of the model in practical applications. In addition, version management is also an indispensable part. Since the model will go through multiple iterations, each update may bring performance improvements or the addition of new features. Therefore, effectively managing and tracking different versions of the model is crucial to ensuring system stability and traceability.

[0003] Currently, relevant technologies suggest that algorithm models can be stored through Object Storage Service (OSS) or Git Large File Storage (LFS). However, when storing large model files, object storage services are costly, difficult to ensure data consistency, and have weak data controllability. LFS not only requires the configuration of corresponding remote storage, but also requires additional client configuration every time a new team member is added, resulting in a complex and time-consuming storage process. In addition, the above two storage solutions are general solutions. When storing and managing specific types of files, they require additional adaptation work to meet the specific needs of the model in training, deployment, and maintenance, which brings additional burdens to developers. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a model storage management method, system and server, which can significantly reduce the adaptation difficulty of algorithm model storage and improve the flexibility of model management.

[0005] In the first aspect, an embodiment of the present invention provides a model storage management method, which is applied to a model storage management system, which includes: an algorithm framework adapter module, a storage management module, an index management module and a storage interface module. The method includes: using the algorithm framework adapter module to encapsulate the model file to be stored, and determine the target storage content after encapsulation, wherein the target storage content includes: configuration information and file information, and the configuration information includes: metadata of the model file to be stored; using the storage management module to store and manage the target storage content, and determine the actual storage location of the target storage content, wherein the storage management module includes: an index storage unit, a version information storage unit and a file information storage unit, and the index storage unit includes: a global index storage unit and a local index storage unit; according to the actual storage location and the target underlying storage type received by the storage interface module, the target storage content is stored, and the index changes during model storage are synchronized to the database through the hybrid storage architecture in the index management module to complete model query retrieval.

[0006] In one embodiment, the steps of encapsulating the model file to be stored through the algorithm framework adapter module and determining the target storage content after encapsulation include: identifying the file type of the model file to be stored through the preset algorithm framework adapter module to determine the target file type; matching the target deployment plan corresponding to the model file to be stored from the preset deployment plan set according to the target file type, and encapsulating the model file to be stored according to the target deployment plan to determine the target storage content after encapsulation.

[0007] In one embodiment, a storage management module is used to perform storage management on target storage content, and the steps of determining the actual storage location of the target storage content include: performing hash calculation on the target storage content through a preset digest algorithm in the storage management module to generate a digest value of a fixed length; superimposing the digest value and the preset storage prefix to determine the actual storage location of the target storage content in the storage system, and generating a text file corresponding to the target storage content, wherein the digest value is an identifier corresponding to the model file, and the text file is used to record digest algorithm information and configuration information.

[0008] In one embodiment, after the steps of superimposing the summary value and the preset storage prefix, determining the actual storage location of the target storage content in the storage system, and generating a text file corresponding to the target storage content, it includes: matching the summary value of the target storage content with the information in the database, determining the query result, and updating the local index and global index corresponding to the target storage content, wherein, when the storage management module stores any model file to be stored, a local index corresponding to the target storage content is generated, the local index is used to collect version information of the model file to be stored, and the global index is used to record all model information of the model file to be stored in the storage space.

[0009] In one embodiment, the summary value of the target storage content is matched with the information in the database, the query result is determined, and the local index and global index corresponding to the target storage content are updated, including: if the query result is that the summary value has been stored in the database, the summary value is stored in a text file, and a local index corresponding to the target storage content is generated, and the global index is synchronously refreshed; if the query result is that the summary value has not been stored in the database, the summary value is stored in the database, and a local index corresponding to the target storage content is generated, and the global index is synchronously refreshed.

[0010] In one embodiment, before the step of storing the target storage content according to the actual storage location and the target underlying storage type received by the storage interface module, it includes: setting an abstract storage behavior interface set through the storage interface module to receive the target underlying storage type sent by the user end according to the abstract storage behavior interface set.

[0011] In one embodiment, the step of synchronizing index changes during model storage to the database through the hybrid storage architecture in the index management module includes: when the first retrieval of any model file to be stored is completed, a cache key corresponding to the summary value is generated based on the summary value of the model file to be stored and the retrieval result, so that in subsequent retrievals, the database query can be skipped to directly obtain the retrieval result.

[0012] In the second aspect, an embodiment of the present invention also provides a model storage management system, which includes: an algorithm framework adapter module, a storage management module, an index management module and a storage interface module, and the storage management module is connected to the algorithm framework adapter module, the index management module and the storage interface module respectively; wherein the algorithm framework adapter module is used to encapsulate the model file to be stored and determine the target storage content after encapsulation, wherein the target storage content includes: configuration information and file information, and the configuration information includes: metadata of the model file to be stored; the storage interface module is used to set an abstract storage behavior interface set to receive the target underlying storage type sent by the user end according to the abstract storage behavior interface set; the storage management module is used to store and manage the target storage content, determine the actual storage location of the target storage content, and store the target storage content according to the actual storage location and the target underlying storage type received by the storage interface module; the index management module is used to synchronize the index changes during model storage to the database to complete model query retrieval.

[0013] In a third aspect, an embodiment of the present invention further provides a server, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0015] The embodiments of the present invention bring the following beneficial effects: An embodiment of the present invention provides a model storage management method, system and server. The method uses an algorithm framework adapter module to encapsulate the stored model file, determines the target storage content after encapsulation, and then uses the storage management module to store the target storage content and determine the actual storage location of the target storage content. Finally, the target storage content is stored according to the actual storage location and the target underlying storage type received by the storage interface module, and the index changes during model storage are synchronized to the database through the hybrid storage architecture in the index management module to complete model query retrieval. The embodiment of the present invention supports model file definitions of multiple algorithm frameworks by adding an algorithm framework adapter, and uses the model file management module to achieve unified management of storage structure and version. In addition, the model index module is used to achieve low-latency reading of the model storage location, and the interface design of the model storage module supports the free expansion of the underlying storage technology. Therefore, the present invention can significantly improve the flexibility of model management and support incremental updates, version management and fine-grained access control of the model.

[0016] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of the structure of a model storage management system provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a storage management module provided in an embodiment of the present invention; Figure 3 A schematic diagram of a flow chart of a model storage management method provided by an embodiment of the present invention; Figure 4 A schematic diagram of a model storage process provided by an embodiment of the present invention; Figure 5 A schematic diagram of a model search process provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] Currently, with the rapid development of deep learning, the parameter scale of AI models has expanded from the initial hundreds of millions to hundreds of billions. This change not only marks a significant increase in computing power and data processing capabilities, but also brings new challenges, especially in model training, deployment, and maintenance. How to efficiently store these models to facilitate subsequent inference services and continuous optimization has become a critical issue. First, to meet rapidly evolving business needs, storage systems must be able to support low-latency data reading to ensure the responsiveness of models in practical applications. In addition, version management is also an indispensable part. Because models undergo multiple iterations, and each update may bring performance improvements or the addition of new features, effectively managing and tracking different versions of models is crucial to ensuring system stability and traceability.

[0022] Several common existing storage solutions, such as Object Storage Service (OSS) and Git Large File Storage (LFS), have solved some of these problems to a certain extent, but they also have obvious limitations: Object Storage Service (OSS): Object Storage is a cloud-based data storage method widely popular for its large storage capacity, strong scalability, and high data reliability. Object Storage is suitable for storing unstructured data that does not require versioning, such as text, images, and videos. Its advantages lie in its distributed architecture, which allows for easy horizontal scalability of storage space and the flexibility to adjust storage scale without data migration to meet business needs at different stages. Furthermore, as a commercial storage service, Object Storage employs multi-layered data security mechanisms, including but not limited to encrypted transmission, access control, and data backup and recovery, ensuring data security and reliability throughout its lifecycle. It also utilizes CDN acceleration technology to optimize data access, intelligently dispatching the nearest service node based on the user's geographic location, effectively improving data read speeds. However, these technologies also come with challenges such as high costs, difficulty ensuring data consistency, and relatively limited data controllability. For large model files, Object Storage is not the preferred choice due to cost and version management considerations.

[0023] Git Large File Storage (LFS) is an open-source extension to Git designed to optimize the management of large binary files in Git repositories. While Git excels at versioning source code and small files, it struggles with managing large files. Specifically, Git needs to store full copies of these files for each commit, causing repositories to grow in size over time and slowing down common Git operations like cloning, fetching, and pulling. With Git LFS, the actual data for large files is stored on a dedicated server, while the Git repository only stores text pointers to these files. This significantly reduces the burden on the repository and speeds up these operations. Given Git's distributed architecture, consisting of both server and client components, implementing Git LFS requires not only configuring the appropriate remote storage but also additional client configuration for each new team member, a complex and time-consuming process.

[0024] In addition, both of the above storage technologies are general solutions. When storing and managing specific types of files, especially algorithm models, they often require additional adaptation work to meet the specific needs of the model in training, deployment and maintenance, which undoubtedly brings additional burdens to developers.

[0025] Based on this, the model storage management method, system and server provided by the present invention support the model file definition of multiple algorithm frameworks by adding an algorithm framework adapter, and utilize the model file management module to achieve unified management of storage structure and version. In addition, the model index module is used to achieve low-latency reading of the model storage location, and through the interface design of the model storage module, it supports the free expansion of the underlying storage technology. Therefore, the present invention can significantly improve the flexibility of model management and support incremental updates, version management and fine-grained access control of the model.

[0026] To facilitate understanding of this embodiment, a model storage management method disclosed in an embodiment of the present invention is first described in detail. The method is applied to a model storage management system. To facilitate understanding of the model storage management method, an embodiment of the present invention provides a structural diagram of a model storage management system, as shown in FIG. Figure 1 As shown, the model storage management system includes: an algorithm framework adapter module, a storage management module, an index management module and a storage interface module, and the storage management module is connected to the algorithm framework adapter module, the index management module and the storage interface module respectively.

[0027] The algorithm framework adapter module is responsible for encapsulating model files to be stored and determining the target storage content after encapsulation. This target storage content includes configuration information and file information, with the configuration information including metadata for the model files to be stored. In one embodiment, the algorithm framework adapter module can adapt to model files and their corresponding deployment solutions for different algorithm frameworks, providing customized processing for specific frameworks. The algorithm framework adapter module adopts a factory-like design and consists of three main components: an adapter factory (AdapterFactor), a data construction service (ManifestBuilder), and specific build services (PytorchManifestBuilder, TensorflowManifestBuilder, etc.). The AdapterFactory identifies the model file type and creates an appropriate adapter instance. It uses the ManifestBuilder interface to define creation rules, while the specific PytorchManifestBuilder or TensorflowManifestBuilder implements these rules to construct the data file for deployment in the specific framework. The final content of this processing is the encapsulated model file, that is, the actual content of the final storage, which can be represented by a Manifest. The Manifest primarily includes configuration information (config) and file information (models), both represented by ManifestFile. config stores metadata of some model files, such as model name, model framework, instructions, version number, and creation date; models stores the model file itself and the data required for the corresponding deployment.

[0028] The storage management module is used to store and manage the target storage content, determine the actual storage location of the target storage content, and store the target storage content according to the actual storage location and the target underlying storage type received by the storage interface module. In one embodiment, the storage management module is used to effectively organize and manage the encapsulated Manifest file. It is responsible for defining a unified storage structure and index structure, aiming to solve the confusion and retrieval problems that may arise when multiple versions of models coexist. This module can realize fine management of model versions, including but not limited to version creation, retrieval, deletion, incremental update and other functions.

[0029] See also Figure 2 The schematic diagram of the structure of a storage management module shown in FIG. 1 shows a storage structure defined by the storage management module. The storage structure mainly consists of three parts, as shown in (1) to (3) below: (1) Index: It is divided into global index and local index. Global Index is responsible for pointing to all local indexes. Each time a new model file is submitted, it corresponds to a local index. Local Index is responsible for collecting all tag information of a specific model. Local Index records tag information; Global Index records local index location information.

[0030] (2) Tag: Tag is also called version information. A model has multiple tags, and each tag contains N blob information. The version information is used to record the file location of any ManifestFile in the Manifest corresponding to its actual storage blob.

[0031] (3) Blob: Blob is also called file information. Blob is the actual binary storage file. It is created based on the information defined in the Manifest file and is the binary data of this information. Among them, H(M) determines the actual storage location of the Blob by obtaining the file name of the Blob data when it is stored: Path + H(M).

[0032] Specifically, after the model file is encapsulated by the algorithm framework adapter module, a digest algorithm (such as SHA-256) is used to generate the actual storage location of the Blob binary file. Simultaneously, the algorithm information generated by the digest algorithm is saved in the tag file. After the storage file and tag file generation actions are completed, it is determined whether the model is new, and the local index and global index are updated or added simultaneously.

[0033] The index management module is used to synchronize index changes during model storage to the database to complete model query retrieval. Specifically, the index management module is responsible for synchronously recording all index change operations in the database, and combined with an efficient caching mechanism, it ensures the consistency and accuracy of the index data. In one embodiment, the index management module includes: a hybrid storage architecture that integrates two storage methods, MySQL and Redis. The advantages of this structure are as follows: First, as a scalable structured data, the index has relevance and is an important means of locating data. It has high requirements for long-term storage and data persistence. MySQL, due to its powerful transaction processing capabilities and stability, is particularly suitable as a persistent storage layer for index data to ensure data security and integrity. At the same time, by introducing Redis, the data reading speed and system response performance can be effectively improved to meet the needs of high-concurrency queries. This combination not only ensures the persistence and security of data, but also improves the overall performance of the system.

[0034] The storage interface module is used to set up an abstract storage behavior interface set to receive the target underlying storage type sent by the user terminal based on the abstract storage behavior interface set. In one embodiment, the storage interface module is used to enhance the flexibility and maintainability of the system. By defining a set of abstract storage behavior interfaces, it allows users to select different underlying storage technologies based on specific needs. In other words, whether it is a traditional file system, a distributed file system, or a cloud storage service, as long as it complies with the storage interface specification, it can be seamlessly integrated into the model warehouse system of the present invention. Such a design not only reduces dependence on specific storage technologies, but also facilitates future technology upgrades and migrations.

[0035] based on Figure 1 The structural diagram of the model storage management system shown in Figure 2 The embodiment of the present invention introduces the model storage management method in detail. Figure 3 The flowchart of a model storage management method shown is applied to a model storage management system. The model storage management system includes: an algorithm framework adapter module, a storage management module, an index management module, and a storage interface module. The method mainly includes the following steps S302 to S306: Step S302: The model file to be stored is encapsulated by the algorithm framework adapter module to determine the target storage content after encapsulation, wherein the target storage content includes configuration information and file information, and the configuration information includes metadata of the model file to be stored.

[0036] In step S304, the target storage content is managed using the storage management module to determine the actual storage location of the target storage content. The storage management module includes an index storage unit, a version information storage unit, and a file information storage unit. The index storage unit includes a global index storage unit and a local index storage unit. In one embodiment, the storage management module is also used to update or add a new index. Specifically, updating the index: if the user previously uploaded model A and specified version v1, and then uploaded model A (regardless of whether it is a new model) and specified the same version v1, it means updating the index because v1 will be stored as the version file name V, which will replace the tag file actually stored in the underlying layer, causing hm to point to the new content; adding a new index and specifying a new version v2 (which has not appeared before) means adding a new index.

[0037] Step S306, stores the target storage content according to the actual storage location and the target underlying storage type received by the storage interface module, and synchronizes the index changes during model storage to the database through the hybrid storage architecture in the index management module to complete the model query retrieval. In one embodiment, before storage, an abstract storage behavior interface set can be set through the storage interface module to receive the target underlying storage type sent by the user terminal according to the abstract storage behavior interface set.

[0038] The above-mentioned model storage management method provided by the embodiment of the present invention can realize the algorithm framework adapter module, support the automatic conversion and configuration of model file definitions and deployment schemes of multiple algorithm frameworks, thereby reducing the difficulty of model file interoperability between different frameworks, and providing a set of standardized model preparation processes to facilitate the subsequent deployment of the model. In addition, the present invention designs an efficient and convenient storage structure, which can realize the refined management and version control of model versions, and combines an index management module of a hybrid storage architecture to speed up the retrieval speed of the model and achieve low-latency reading, and proposes a layered and scalable system architecture to improve the flexibility and maintainability of the system, so as to better adapt to the development and changes of future technologies.

[0039] See also Figure 4 The schematic diagram of a model storage process shown in FIG. 1 is a diagram of a model storage process shown in FIG. 1 . The embodiment of the present invention also provides an implementation method for model storage management. For details, see (1) to (6) below: (1) Model packaged files: The preset algorithm framework adapter module is used to identify the file type of the model file to be stored and determine the target file type. Then, according to the target file type, the target deployment scheme corresponding to the model file to be stored is matched from the preset deployment scheme set, and the model file to be stored is packaged according to the target deployment scheme to determine the target storage content after packaging. Thus, the system model file uploaded or selected by the user is packaged by the algorithm framework adapter module to obtain Mainfest, that is, the target storage content. One of the files in Manifest is recorded as ManifestFile. .

[0040] (2) Digest algorithm: The target storage content is hashed using the preset digest algorithm in the storage management module to generate a digest value of fixed length. In practical applications, a digest algorithm (e.g., SHA-256) is used to hash M to generate a digest value of fixed length, denoted as H(M).

[0041] H(M)=SHA-256(M) (3) Determine the storage location: The summary value and the preset storage prefix are superimposed to determine the actual storage location of the target storage content in the storage system. The summary value is the identifier corresponding to the model file. In actual applications, the generated summary value H (M) is used as the unique identifier of the Blob file to determine its actual storage location in the storage system. Path is a unified storage prefix.

[0042] Blob Storage Location=Path +H(M) In one embodiment, if the Path is models / , the actual storage location is models / H(M).

[0043] (4) Generate a Tag file, that is, generate a text file corresponding to the target storage content. The text file is used to record the summary algorithm information and configuration information. To store a Manifest, a corresponding Tag file needs to be created. The file name takes the user input parameter version as V, and records the summary algorithm information and the metadata of the model file. In one embodiment, the Tag file can also use the Json format. The content example is: { config:{ desc: description information time: creation date type: model type } blobs:[{ name: file name digest: digest algorithm hm: summary value H (M) size: file size }] } (5) Determine whether the file exists: match the summary value of the target storage content with the information in the database, determine the query result, and update the local index and global index corresponding to the target storage content. When the storage management module stores any model file to be stored, it generates a local index corresponding to the target storage content. The local index is used to collect the version information of the model file to be stored, and the global index is used to record all model information of the model file to be stored in the storage space.

[0044] Specifically, if the query result shows that the summary value has been stored in the database, the summary value will be stored in a text file, and a local index corresponding to the target storage content will be generated, and the global index will be refreshed synchronously; if the query result shows that the summary value has not been stored in the database, the summary value will be stored in the database, and a local index corresponding to the target storage content will be generated, and the global index will be refreshed synchronously. In other words, the database is queried to see whether H (M) has been stored. If the file already exists, the same file will no longer be saved. Only H (M) will be stored in the Tag file, and a new Local Index will be added; if the file does not exist, H (M) will be stored, and a new Local Index will be added. Finally, the Global Index will be refreshed and synchronized to ensure data consistency.

[0045] (6) Update or add new indexes: Local Index: records all tag information of a model. It can use Json format. The content example is: {tags:[{name: version name V; time: creation time}]}.

[0046] Global Index: records all model information stored in the storage space. It can use the Json format. The content example is: {models:[{path: the location where the model is stored, which depends on the specific situation, mainly used to locate the Local Index; time: creation time}]}.

[0047] See also Figure 5 The schematic diagram of a model search process shown in FIG. 1 is a diagram of a model search process shown in FIG. 1 . The embodiment of the present invention also provides an implementation method for database index storage and retrieval, as shown in (1) to (2) below: (1) Define the storage table structure, as shown in Table 1, Table 2, and Table 3 below: Table 1. Global_Index index table structure example table:

[0048] Table 2. Local_Index index table structure example table:

[0049] Table 3. H_m table structure design table:

[0050] (2) Query retrieval. In actual applications, the target object of the retrieval is the version of the model name "Pytorch". For all H(M), the first query operation can usually be understood as: SELECT model_id FROM Global_Index WHERE model_name=”Pytorch” SELECT version_id as v_id FROM Local_Index WHERE id=model_id SELECT hm FROM H_m WHERE version_id=v_id That is, the conditions for querying model files are model name and , the target value is an array H(M).

[0051] SELECT model_id FROM Global_Index WHERE model_name="Pytorch" is used to retrieve the corresponding model ID from the "Global Index" table. The model ID is used to filter out the version ID (v_id) of the model with the "Pytorch" ID in the "Local Index" table. Finally, all the summary values ​​hm corresponding to v_id are retrieved from the H_m table. In other words, all the stored information of the "Pytorch" model in version v1 is viewed.

[0052] In one embodiment, when the first search of any model file to be stored is completed, a cache key corresponding to the digest value is generated based on the digest value of the model file to be stored and the search results, so that in subsequent searches, the database query can be skipped and the search results can be directly obtained. In actual applications, a suitable cache key can be designed after the first search is completed, for example: Key=model+”Pytorch”+version+V "Pytorch" is the name of the model actually retrieved, and its cached content Value is: Value=List(hm) The purpose of caching keys is to speed up retrieval. That is, through a Map(key, value) method, the actual stored information H(M) can be directly obtained through the key. For example, if a user queries H(M) corresponding to the "Pytorch" version V of the model twice, the database query can be skipped the second time and the result can be obtained directly.

[0053] In summary, the present invention can implement an algorithm framework adapter module, supporting the automatic conversion and configuration of model file definitions and deployment schemes of multiple algorithm frameworks (such as Pytorch, Tensorflow, etc.), thereby reducing the difficulty of model file interoperability between different frameworks and providing a set of standardized model preparation processes to facilitate the subsequent deployment of the model.

[0054] In addition, the present invention designs an efficient and convenient storage structure that can realize refined management and version control of model versions. It combines an index management module with a hybrid storage architecture to speed up the retrieval speed of the model and achieve low-latency reading. It also proposes a layered and scalable system architecture to improve the flexibility and maintainability of the system, so that it can better adapt to future technological developments and changes.

[0055] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0056] An embodiment of the present invention provides a server. Specifically, the server includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0057] Figure 6 A structural diagram of a server provided in an embodiment of the present invention, wherein the server 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0058] Memory 61 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. Communication between the system network element and at least one other network element is achieved through at least one communication interface 63 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0059] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0060] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0061] The processor 60 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method may be performed by hardware integrated logic circuits or software instructions within the processor 60. The processor 60 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules may be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 61 , and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0062] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0063] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0064] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A model storage management method, characterized in that: The method is applied to a model storage management system, which includes: an algorithm framework adapter module, a storage management module, an index management module and a storage interface module. The method includes: Through the algorithm framework adapter module, the model file to be stored is encapsulated and processed to determine the target storage content after encapsulation, wherein the target storage content includes: configuration information and file information, and the configuration information includes: metadata of the model file to be stored; Utilizing the storage management module to perform storage management on the target storage content and determine the actual storage location of the target storage content, wherein the storage management module includes: an index storage unit, a version information storage unit and a file information storage unit, and the index storage unit includes: a global index storage unit and a local index storage unit; The target storage content is stored according to the actual storage location and the target underlying storage type received by the storage interface module, and the index changes during model storage are synchronized to the database through the hybrid storage architecture in the index management module to complete model query retrieval.

2. The model storage management method according to claim 1, characterized in that: The step of encapsulating the model file to be stored and determining the target storage content after encapsulation by the algorithm framework adapter module comprises: Through the preset algorithm framework adapter module, the file type of the model file to be stored is identified and processed to determine the target file type; According to the target file type, the target deployment scheme corresponding to the model file to be stored is matched from a preset deployment scheme set, and the model file to be stored is packaged according to the target deployment scheme to determine the target storage content after packaging.

3. The model storage management method according to claim 1, characterized in that: The step of utilizing the storage management module to perform storage management on the target storage content and determining the actual storage location of the target storage content comprises: Performing hash calculation on the target storage content through a preset digest algorithm in the storage management module to generate a digest value of a fixed length; The summary value and the preset storage prefix are superimposed to determine the actual storage location of the target storage content in the storage system, and a text file corresponding to the target storage content is generated, wherein the summary value is an identifier corresponding to the model file, and the text file is used to record summary algorithm information and the configuration information.

4. The model storage management method according to claim 3, characterized in that: After the steps of superimposing the summary value and the preset storage prefix, determining the actual storage location of the target storage content in the storage system, and generating a text file corresponding to the target storage content, the method further includes: The summary value of the target storage content is matched with the information in the database to determine the query result, and the local index and global index corresponding to the target storage content are updated, wherein when the storage management module stores any of the model files to be stored, the local index corresponding to the target storage content is generated, the local index is used to collect version information of the model file to be stored, and the global index is used to record all model information of the model file to be stored in the storage space.

5. The model storage management method according to claim 4, characterized in that: The step of matching the summary value of the target storage content with the information in the database, determining the query result, and updating the local index and the global index corresponding to the target storage content includes: If the query result is that the summary value has been stored in the database, the summary value is stored in the text file, and a local index corresponding to the target storage content is generated, and the global index is synchronously refreshed; If the query result is that the summary value has not been stored in the database, the summary value is stored in the database, and the local index corresponding to the target storage content is generated, and the global index is synchronously refreshed.

6. The model storage management method according to claim 1, characterized in that: Before the step of storing the target storage content according to the actual storage location and the target underlying storage type received by the storage interface module, the method includes: An abstract storage behavior interface set is set through the storage interface module to receive the target underlying storage type sent by the user end according to the abstract storage behavior interface set.

7. The model storage management method according to claim 1, characterized in that: The step of synchronizing index changes during model storage to the database through the hybrid storage architecture in the index management module includes: When the first search of any of the model files to be stored is completed, a cache key corresponding to the summary value is generated based on the summary value of the model file to be stored and the search result, so as to skip the database query and directly obtain the search result during subsequent searches.

8. A model storage management system, characterized in that: The system comprises: an algorithm framework adapter module, a storage management module, an index management module and a storage interface module, wherein the storage management module is connected to the algorithm framework adapter module, the index management module and the storage interface module respectively; wherein: The algorithm framework adapter module is used to encapsulate the model file to be stored and determine the target storage content after encapsulation, wherein the target storage content includes: configuration information and file information, and the configuration information includes: metadata of the model file to be stored; The storage interface module is used to set an abstract storage behavior interface set to receive a target underlying storage type sent by a user terminal according to the abstract storage behavior interface set; The storage management module is used to perform storage management on the target storage content, determine the actual storage location of the target storage content, and store the target storage content according to the actual storage location and the target underlying storage type received by the storage interface module; The index management module is used to synchronize index changes during model storage to the database to complete model query retrieval.

9. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images