File version control management method and device

Abstract

The invention discloses a file version control management method. A client in a version control system and a server of the version control system are respectively provided with a data warehouse; a data warehouse deployed on the server side adopts a distributed architecture and comprises a first distributed storage service for storing metadata of the version file and a second distributed storage service for storing file content of the version file; comprising the following steps: acquiring a modified version file stored in a locally deployed data warehouse; generating a version record based on the modified version file; the version record comprises metadata of the modified version file and file content of the modified version file; and uploading the version record to a server, so that the server stores metadata contained in the version record to a first distributed storage service and stores file contents contained in the version record to a second distributed storage service.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of databases, and more particularly, to a method and device for controlling and managing file versions. Background Art

[0002] VCS (Version Control System) is a management system that records every change to a version file, clearly saves the time, author, and specific content of the version file change, and facilitates review and audit. Using the VCS system, developers can easily switch between different file versions, or restore a version file to a previous version. In addition, the VCS system also supports multiple developers to modify the same version file (usually corresponding to a development project) stored in the VCS system's data warehouse at the same time, and can manage conflicts between different versions to ensure smoother development collaboration between multiple developers.

[0003] However, in actual applications, as the number of version records submitted by developers in the VCS system's data warehouse continues to increase and the size of the data warehouse becomes larger and larger, the performance of the VCS system may gradually degrade or even become unavailable. Moreover, most VCS systems usually cannot support the submission of huge version files, which also makes the VCS system unable to meet the needs of some development scenarios. Summary of the invention

[0004] This specification proposes a file version control management method, which is applied to a client of a version control system; wherein a data warehouse is deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; including:

[0005] Obtain the modified version files stored in the locally deployed data warehouse;

[0006] Generate a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file;

[0007] The version record is uploaded to the server, so that the server stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

[0008] Optionally, before generating the version record based on the modified version file, the method further includes:

[0009] Determine whether there is a target version file whose file size reaches a preset first threshold value among the modified version files;

[0010] If the target version file exists in the modified version file, the target version file is divided into multiple version file fragments, and the unique identifiers of the multiple version file fragments are respectively added to the metadata of the target version file to establish a link relationship between the metadata and the multiple version file fragments.

[0011] Optionally, before uploading the version record to the server, the method further includes:

[0012] Determine whether there is a version file whose file size is lower than a preset second threshold value among the modified version files included in the version record; wherein the second threshold value is lower than the first threshold value;

[0013] If there is at least one version file whose file size is lower than a preset second threshold value among the modified version files included in the version record, the at least one version file is packaged and compressed.

[0014] Optionally, the method further includes:

[0015] In response to an operation initiated by a user to retrieve a version file based on the metadata of the version file, a search request is generated; wherein the search request includes a query index constructed based on the metadata of the version file;

[0016] Sending the retrieval request to the server, so that the server retrieves the metadata in the first distributed storage service based on the query index;

[0017] The metadata of the version file returned by the server is obtained, and the obtained file content of the version file is stored in a data warehouse deployed on the client.

[0018] Optionally, the metadata includes a unique identifier of the version file;

[0019] The method further comprises:

[0020] In response to an operation initiated by a user to retrieve a version file based on a unique identifier of the version file, a retrieval request is generated; wherein the retrieval request includes the unique identifier of the version file;

[0021] Sending the retrieval request to the server, so that the server retrieves the file content of the version file in the second distributed storage service based on the unique identifier of the version file;

[0022] The file content of the version file returned by the server is obtained, and the obtained file content of the version file is stored in a data warehouse deployed on the client.

[0023] Optionally, the first distributed storage service is a distributed database; the second distributed storage service is a distributed file system or a distributed object storage system.

[0024] This specification also proposes a file version control management method, which is applied to the server of the version control system; wherein a data warehouse is deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; including:

[0025] Receiving the version record uploaded by the client; wherein the version record includes metadata of the modified version file stored in the data warehouse deployed on the client and file content of the modified version file;

[0026] The metadata included in the version record is stored in the first distributed storage service, and the file content included in the version record is stored in the second distributed storage service.

[0027] Optionally, if there is a target version file in the modified version file whose file size reaches a preset first threshold, the version record includes multiple version file fragments obtained by the client splitting the target version file, and the metadata of the target version file included in the version record includes unique identifiers of the multiple version file fragments.

[0028] Optionally, among the modified version files included in the version record, there is at least one version file whose file size is lower than a preset second threshold; wherein the second threshold is lower than the first threshold; and the at least one version file is packaged and compressed by the client;

[0029] Before storing the metadata included in the version record in the first distributed storage service and storing the file content included in the version record in the second distributed storage service, the method further includes:

[0030] Decompressing the at least one version file contained in the version record to obtain a packaged file corresponding to the at least one version file;

[0031] The package file is parsed to split the package file into the at least one version file.

[0032] Optionally, the method further includes:

[0033] Receiving a search request sent by the client; wherein the search request includes a query index constructed based on metadata of the version file to be searched;

[0034] Retrieving the metadata in the first distributed storage service based on the query index;

[0035] If the metadata is retrieved in the first distributed storage service, the metadata is returned to the client.

[0036] Optionally, the metadata includes a unique identifier of the version file;

[0037] The method further comprises:

[0038] Receiving a search request sent by the client; wherein the search request includes a unique identifier of the version file to be searched;

[0039] Based on the unique identifier of the version file, the file content of the version file is retrieved in the second distributed storage service, and the retrieved file content of the version file is returned to the client.

[0040] Optionally, the data warehouse deployed on the server further includes a cache constructed based on the cache space on the server; wherein the cache caches the metadata stored in the data warehouse and hot data in the file content;

[0041] Retrieving the metadata in the first distributed storage service based on the query index includes:

[0042] Retrieving the metadata in the cache based on the query index; if the metadata is not retrieved in the cache, further retrieving the metadata in the first distributed storage service based on the query index constructed from the metadata;

[0043] Retrieving the file content of the version file in the second distributed storage service based on the unique identifier of the version file includes:

[0044] Based on the unique identifier of the version file, the file content of the version file is retrieved in the cache; if the file content of the version file is not retrieved in the cache, the file content of the version file is further retrieved in the second distributed storage service based on the unique identifier of the version file.

[0045] Optionally, the cache includes a first-level cache built based on memory space, and a second-level cache built based on disk space; wherein the first-level cache is used to cache hot data in the metadata stored in the data warehouse; and the second-level cache space is used to cache hot data in the file content stored in the data warehouse;

[0046] Retrieving the metadata in the cache based on the query index; if the metadata is not retrieved in the cache, further retrieving the metadata in the first distributed storage service based on the query index constructed from the metadata, including:

[0047] Retrieving the metadata in the first level cache based on the query index; if the metadata is not retrieved in the first cache, further retrieving the metadata in the first distributed storage service based on the query index;

[0048] Based on the unique identifier of the version file, searching the cache for the file content of the target version file; if the file content of the version file is not found in the cache, further searching the second distributed storage service for the file content of the version file based on the unique identifier of the version file, including:

[0049] Based on the unique identifier of the version file, the file content of the version file is retrieved in the second cache; if the file content of the version file is not retrieved in the second cache, the file content of the version file is further retrieved in the second distributed storage service based on the unique identifier of the version file.

[0050] Optionally, the method further includes:

[0051] Cache the metadata stored in the data warehouse into the first-level cache, and run a cache elimination algorithm to delete non-hotspot data in the metadata cached in the first-level cache, so as to retain the hotspot data in the metadata in the first cache;

[0052] The file content of at least one version file whose size is lower than the second threshold and stored in the data warehouse is obtained, and the obtained file content is cached in the second-level cache space.

[0053] Optionally, the second-level cache is a backup cache of the first-level cache; the second-level cache is also used to cache hot data in the metadata that has been cached in the first-level cache;

[0054] Retrieving the metadata in the first-level cache based on the query index; if the metadata is not retrieved in the first cache, further retrieving the metadata in the first distributed storage service based on the query index constructed from the metadata, including:

[0055] Based on the query index, the metadata is retrieved from the first-level cache; if the metadata is not retrieved in the first-level cache, based on the query index, the metadata is retrieved from the second-level cache; if the metadata is still not retrieved in the second-level cache, further based on the query index, the metadata is retrieved from the first distributed storage service.

[0056] Optionally, the first distributed storage service is a distributed database; the second distributed storage service is a distributed file system or a distributed object storage system.

[0057] This specification also proposes a file version control management system, including a client and a server; data warehouses are deployed on the client and the server respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; wherein:

[0058] The client is used to obtain a modified version file stored in a locally deployed data warehouse, and generate a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file; and upload the version record to the server;

[0059] The server is used to receive the version record uploaded by the client; store the metadata contained in the version record in the first distributed storage service, and store the file content contained in the version record in the second distributed storage service.

[0060] The specification also proposes a file version control management device, which is applied to a client of a version control system; wherein a data warehouse is deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; the device includes:

[0061] An acquisition module is used to obtain modified version files stored in a locally deployed data warehouse;

[0062] A generation module generates a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file;

[0063] An upload module uploads the version record to the server, so that the server stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

[0064] The specification also proposes a file version control management device, which is applied to the server of the version control system; wherein a data warehouse is deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; the device includes:

[0065] A receiving module receives the version record uploaded by the client; wherein the version record includes metadata of the modified version file stored in the data warehouse deployed on the client and the file content of the modified version file;

[0066] The storage module stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

[0067] In the above embodiment, on the one hand, since the central data warehouse deployed on the server side of the version control system is a data warehouse built based on the distributed storage service, it has the ability to expand capacity infinitely; therefore, the developer can synchronize the modified oversized file versions stored in the local data warehouse deployed on the client to the central data warehouse deployed on the server side for storage;

[0068] On the other hand, when the client synchronizes a modified, very large file version stored in the local data warehouse to the central data warehouse deployed on the server for storage, the metadata of the version file and the file content of the version file are stored separately. This allows developers to quickly retrieve and synchronize to the required file version in the distributed storage service that stores the metadata when they need to retrieve file versions, without having to scan the file content of the complete version file stored in the distributed storage service that stores the file content of the version file. This can significantly improve the retrieval performance of the version control system when retrieving file versions. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0070] Figure 1 is a system architecture diagram of a VCS system shown in an embodiment of this specification;

[0071] Figure 2 is a flow chart of a file version control management method shown in an embodiment of this specification;

[0072] Figure 3 This is a flowchart of synchronizing file versions of a data warehouse locally deployed on a client and a data warehouse deployed on a server, as shown in an embodiment of this specification;

[0073] Figure 4 is a flow chart of another file version control management method shown in an embodiment of this specification;

[0074] Figure 5 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0075] Figure 6 It is a block diagram of a file version control and management device shown in an embodiment of this specification;

[0076] Figure 7 It is a block diagram of a file version control and management device shown in an embodiment of this specification. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0078] As the number of version records submitted by developers in the VCS system's data warehouse continues to increase and the size of the data warehouse becomes larger and larger, the performance of the VCS system may gradually decline or even become unavailable. Moreover, most VCS systems usually cannot support the submission of huge version files, which also makes the VCS system unable to meet the needs of some development scenarios.

[0079] In a traditional VCS system, in order to solve the problem of the performance of the VCS system gradually declining due to the increasing number of version records in the data warehouse of the VCS system and the increasing size of the data warehouse, on the one hand, the method of streamlining historical version records can be adopted to delete useless historical records in the data warehouse of the VCS system, or to compress multiple file versions into one file version to reduce version records, thereby releasing the storage space occupied by these file versions; on the other hand, the data warehouse of the VCS system can be split into multiple sub-warehouses, and the multiple sub-warehouses can include a main warehouse and multiple sub-warehouses. In the main warehouse, only some public data related to the file version and the link information of the sub-warehouse can be stored, and in the sub-warehouse, the specific content of the file version can be stored, thereby achieving the purpose of streamlining the size of the data warehouse.

[0080] However, by simplifying the historical version records, some file version records may be lost, making the file versions stored in the entire data warehouse incomplete. And by splitting the VCS system's data warehouse into multiple sub-warehouses, the entire data warehouse becomes fragmented, requiring users to manage different sub-warehouses separately, which increases the management cost of the data warehouse.

[0081] In addition, in traditional VCS systems, in order to solve the problem that VCS systems usually cannot support the submission of huge version files, an independent LFS (Large File Storage) service is usually deployed for the data warehouse of the VCS system, and the LFS is used to independently store and manage the huge version files stored in the data warehouse.

[0082] In this way, on the one hand, the huge version files stored in the data warehouse can be separated from the data warehouse, and only the pointers to these version files are retained in the data warehouse. This can make the data warehouse lighter; on the other hand, when developers need to access the huge version files stored in the data warehouse, they can still access the file contents of the huge version files stored in the LFS service based on the pointers stored in the data warehouse.

[0083] However, although this method can make the VCS system's data warehouse lightweight, since the LFS service usually has restrictions on the size of stored files and does not support infinitely large files, in actual applications, there may still be some problems with giant version files that cannot be submitted.

[0084] Moreover, deploying an independent LFS service for the VCS system's data warehouse not only increases the user's learning cost, but also increases the dependence of the entire VCS system's services on the LFS service. Once an abnormality occurs in the LFS service, the VCS system will face stability risks.

[0085] Based on this, this specification proposes a high-performance VCS system architecture that can support the storage of huge version files.

[0086] See also Figure 1 , Figure 1 This is a system architecture diagram of a VCS system provided in this manual.

[0087] Traditional VCS systems are usually divided into two categories: centralized VCS systems and distributed VCS systems.

[0088] Centralized VCS systems usually adopt the CS architecture, that is, the client-server service model. In this architecture, a central data warehouse will be deployed on the server side of the VCS system. Every version file developed by developers on the client side can be submitted to the server side, and the central data warehouse on the server side will be uniformly stored and managed according to the version.

[0089] Distributed VCS systems, unlike centralized VCS systems, use a distributed networking architecture, where each developer's local device and the server together form a P2P peer-to-peer network. In addition to deploying a data warehouse on the server, each developer can also independently deploy a local data warehouse on their local device. This local data warehouse is functionally equivalent to a complete copy of the data warehouse deployed on the server. Only when necessary will the local data warehouse deployed on the developer's local device synchronize file versions with the data warehouse deployed on the server.

[0090] and Figure 1 The VCS system shown is a system architecture that integrates the features of a centralized VCS system and a distributed VCS system.

[0091] like Figure 1 As shown, in the system architecture of the VCS system, a C / S architecture similar to a centralized VCS system can still be adopted, which can specifically include a client on the developer's local device and a backend server.

[0092] Unlike the traditional centralized VCS system, Figure 1 The architecture of the VCS system shown can integrate the characteristics of a distributed VCS system. In addition to deploying a data warehouse on the server side, a data warehouse can also be independently deployed on the client side. The data warehouse on the client side can synchronize file versions with the data warehouse on the server side.

[0093] The data warehouse deployed on the server may specifically adopt a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files.

[0094] The client can obtain the modified version file stored in the locally deployed data warehouse, generate a version record based on the modified version file, and upload the version record to the server; wherein the version record can specifically include the metadata of the modified version file and the file content of the modified version file;

[0095] The above-mentioned server can receive the version record uploaded by the client, and adopt a method of separately storing the file content and metadata of the version file, storing the metadata contained in the version record to the first distributed storage service, and storing the file content contained in the version record to the second distributed storage service.

[0096] See also Figure 2 , Figure 2 This is a flowchart of a file version control management method shown in this specification, which is applied to Figure 1 The client in the VCS system architecture shown; the method includes the following execution process:

[0097] Step 202: Obtain a modified version file stored in a locally deployed data warehouse;

[0098] In actual applications, the client can provide a working interface for developers, so that the developers can control and manage file versions through the working interface.

[0099] On the one hand, developers can carry out development work in the work interface and then submit the developed version files to the data warehouse deployed locally on the client for storage; on the other hand, developers can also operate in the work interface to modify the developed version files stored in the locally deployed data warehouse.

[0100] It should be noted that the above-mentioned version files generally refer to files that can be stored and managed according to versions.

[0101] For example, in one embodiment shown, the working interface may be a working interface for code development and management, the version file may be a code file stored and managed according to the version, and accordingly, the data warehouse may be a code warehouse. Of course, in practical applications, the version file may also be a file stored and managed according to the version in other forms other than the code file, which is not particularly limited in this specification.

[0102] Among them, whether it is a data warehouse deployed locally on the client or a data warehouse deployed on the server, version files can usually be stored and managed according to versions. For example, when a version file stored in the data warehouse is modified, the modified version file can be treated as a new version file and saved separately in the data warehouse.

[0103] Since the data warehouse deployed locally on the client is functionally equivalent to a complete copy of the data warehouse deployed on the server, in actual applications, the client can also synchronize the version files stored in the locally deployed data warehouse with the version files stored in the data warehouse deployed on the server.

[0104] For example, in actual applications, when the version file stored in the data warehouse deployed locally on the client is modified by the developer and a new version file is generated, the client can synchronize the new version file to the server in a timely manner and store it in the data warehouse deployed on the server.

[0105] It should be noted that when synchronizing file versions between the data warehouse deployed locally on the client and the data warehouse deployed on the server, the synchronization can be automatically triggered by the client after certain synchronization conditions are met, or the developer can manually trigger the synchronization on the client based on specific needs.

[0106] For example, in one example, the client can periodically scan the locally deployed data warehouse to determine whether a new version of the version file has been updated in the data warehouse. Once a new version of the version file is found to be updated in the data warehouse, it can immediately trigger the synchronization of the file version with the data warehouse deployed on the server, upload the new version of the version file to the server in real time, and store it in the data warehouse deployed on the server.

[0107] In another example, the client may also provide developers with the function of manually submitting a new version of the version file based on commands through the above-mentioned development interface. In this case, when the developer modifies a version file stored in the data warehouse deployed locally on the client and generates a new version of the version file, the developer can manually trigger the synchronization of the file version with the data warehouse deployed on the server by entering a command, and manually submit the new version of the version file to the server for storage in the data warehouse deployed on the server.

[0108] See also Figure 3 , Figure 3 This is a flowchart provided in this specification for synchronizing file versions of a data warehouse deployed locally on a client and a data warehouse deployed on a server.

[0109] like Figure 3 As shown, when the synchronization of file versions of the locally deployed data warehouse and the data warehouse deployed on the server is triggered, the client can further obtain the modified version files stored in the locally deployed data warehouse.

[0110] For example, in one example, if the client periodically scans the locally deployed data warehouse to determine whether a new version of the version file has been updated in the data warehouse, and automatically triggers the synchronization of file versions between the locally deployed data warehouse and the data warehouse deployed on the server, in this case, the client can further read the scanned new version of the version file after scanning that a new version of the version file has been updated in the locally deployed data warehouse.

[0111] In another example, if the client uses a developer-based command approach to manually trigger the synchronization of file versions between the locally deployed data warehouse and the data warehouse deployed on the server, in this case, the client can further read the new version file that needs to be manually submitted from the locally deployed data warehouse specified by the developer based on the command.

[0112] Step 204: Generate a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file;

[0113] Please continue to see Figure 3 After obtaining the modified version file, the client can generate a version record based on the obtained modified version file. The version record is also called a file version snapshot, which can specifically include the metadata of the modified version file and the file content of the modified version file. In order to establish a link relationship between the metadata of the version file and the file content of the version file, the metadata of the version file can specifically include a unique identifier of the version file.

[0114] The above-mentioned metadata may specifically include any form of data used to describe the version file, which is not particularly limited in this specification; for example, the metadata may specifically include a unique identifier of the version file, directory data corresponding to the version file, version data related to the version of the version file, segmentation records of the version file, and so on.

[0115] Please continue to see Figure 3 In one embodiment shown, before generating a version record based on a modified version file, the client may also determine whether there is a target version file in the modified version file whose file size reaches a preset first threshold; wherein, in actual applications, the first threshold may be flexibly set based on specific synchronization requirements; for example, in one example, the first threshold may be set to a threshold that is much larger than the file size of an ordinary version file.

[0116] If there is a target version file whose file size reaches a preset first threshold value among the modified version files, and the target version file is a version file with a larger file size stored in the local data warehouse, the client can further perform sharding on the target version file, and divide the target version file into multiple version file shards. For example, the target version file can be divided into multiple version file shards of the same size.

[0117] In addition, the client can also calculate the unique identifiers of the multiple version file fragments respectively, and add the calculated unique identifiers of the multiple version file fragments to the metadata of the target version file, so as to establish a link relationship between the metadata of the multiple version file fragments and the file content of the multiple version file fragments.

[0118] For example, the calculated unique identifiers of the multiple version file fragments can be used to generate an identifier list according to the order of each version file fragment in the above target version file, and then the unique identifiers of the multiple version file fragments can be added to the metadata of the target version file in the form of an identifier list.

[0119] In this case, the version record generated by the client based on the modified version file obtained may include multiple version file fragments obtained by the client from segmenting the above-mentioned target version file, and the metadata of the target version file contained in the version record may also include unique identifiers of the multiple version file fragments.

[0120] The unique identifier may be any unique identifier that can uniquely represent a file, and is not particularly limited in this specification.

[0121] For instance, in one example, the unique identifier may be a hash value, and the client may calculate the hash values ​​of the multiple version file shards respectively as the unique identifiers of the multiple version file shards, and then add the hash values ​​of the multiple version file shards to the target version file in the form of a hash list, so as to establish a link relationship between the metadata of the multiple version file shards and the file contents of the multiple version file shards.

[0122] Step 206: Upload the version record to the server, so that the server stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

[0123] Please continue to see Figure 3 After the client generates a version record based on the modified version file, it can further upload the version record to Figure 1 The server side in the VCS system architecture is shown.

[0124] In one embodiment shown, since in actual applications the file size of the above-mentioned target version file may usually be much larger than the unsegmented version file existing in the modified version file, the file size of the version file fragments obtained after segmenting the target version file is likely to be larger than the unsegmented version file existing in the modified version file. In this case, in order to further reduce the amount of data transmission when the client uploads version records to the server, the client may also compress the version file fragments obtained by segmenting the above-mentioned target version file contained in the version record before uploading the generated version record to the server.

[0125] For example, in one example, when compressing the version file fragments contained in the version record, it can be determined whether to compress the version file fragments according to the file type of the version file fragments; the client can determine the file type of the version file fragments contained in the version record, and if it is determined that the file type of the version file fragments is a specified file type, the version file fragments can be compressed. Among them, the above-mentioned specified file type can be a pre-specified file type that needs to be compressed. For example, in actual applications, version files can usually include binary character type version files and text type version files, and text type version files usually occupy more storage space than binary character type version files. Therefore, in this case, the specified file type can be a text type, that is, when the client determines that the file type of the version file fragments contained in the version record is a text type, it will compress the version file fragments.

[0126] It should be noted that the compression algorithm used to compress the version file segments is not particularly limited in this specification; for example, in one example, a ZSTD compression algorithm with a higher supported compression ratio may be selected.

[0127] In an embodiment shown, since the modified version files included in the version record generated by the client may contain some relatively small version files that have not been fragmented in addition to fragmented version file fragments, in order to further avoid frequent transmission of these relatively small version files, before uploading the generated version record to the server, the client can also determine whether there are any version files whose file size is lower than a preset second threshold among the modified version files included in the version record; wherein the second threshold can be a threshold much smaller than the first threshold; if there is at least one version file whose file size is lower than the preset second threshold among the modified version files included in the version record, then the at least one version file can be packaged and compressed. It should be noted that, unlike ordinary compression processing, package compression processing usually refers to packaging the above at least one version file into one file and then compressing it as a whole.

[0128] In one embodiment shown, before uploading a version record to a server, in addition to packaging and compressing at least one version file whose size is lower than a preset second threshold among the modified version files contained in the version record, the client may further package and compress the metadata of the modified version files contained in the version record.

[0129] In order to allow the data warehouse deployed on the server to have unlimited capacity expansion capabilities to store some giant version files, the data warehouse deployed on the server can specifically adopt a distributed architecture. In addition, in order to improve the retrieval performance of the data warehouse deployed on the server, the data warehouse deployed on the server can specifically adopt a method of storing the file content and metadata of the version file separately. In this case, the data warehouse deployed on the server using a distributed architecture can specifically include a first distributed storage service for storing the metadata of the version file, and a second distributed storage service for storing the file content of the version file.

[0130] Among them, in actual applications, the above-mentioned first distributed storage service can be specifically a storage service that is suitable for storing metadata and has good retrieval performance; and the above-mentioned second distributed storage service can be specifically a storage service that is suitable for storing files and has the ability to expand capacity infinitely.

[0131] For example, in one embodiment shown, the first distributed storage service may specifically be a distributed database; and the second distributed storage service may specifically be a distributed file system or a distributed object storage system.

[0132] Please continue to see Figure 3 After receiving the version record uploaded by the client, the server can parse the version record, obtain the metadata of the modified version file and the file content of the modified version file contained in the version record, and then store the obtained metadata in the above-mentioned first distributed storage service, and store the obtained file content in the second distributed storage service to complete the synchronization of file versions between the data warehouse deployed locally on the client and the data warehouse deployed on the server. Of course, for the multiple version file fragments corresponding to the above-mentioned target version file contained in the version record, the server can still store these version file fragments in the second distributed storage service in the form of fragments.

[0133] After the synchronization of file versions between the data warehouse deployed locally on the client and the data warehouse deployed on the server is completed, developers can efficiently retrieve version files from the data warehouse deployed on the server based on the metadata of the version files they want to retrieve when they have retrieval needs.

[0134] In one embodiment shown, when a developer has a retrieval requirement for a version file stored in a data warehouse deployed on a server (for example, when the developer wants to query a version file of a certain historical version stored in a data warehouse deployed on a server), an operation based on the metadata of the version file can be initiated on the work interface provided by the client. The client can generate a retrieval request in response to the operation; wherein the generated retrieval request can specifically include a query index constructed based on the metadata of the version file to be retrieved.

[0135] The specific form of the query index is not particularly limited in this specification.

[0136] For example, in actual applications, in the process of building a query index based on the metadata of the version file to be retrieved, the query index can specifically be partial metadata contained in the metadata that can uniquely identify the metadata, or it can also be calculated for the metadata of the version file as a whole to obtain a metadata identifier that can uniquely identify the metadata as the query index of the metadata.

[0137] After generating a retrieval request, the client may further send the retrieval request to the server, and after receiving the retrieval request, the server may retrieve the metadata from the first distributed storage service in the data warehouse deployed on the server based on the query index contained in the retrieval request.

[0138] For example, taking the above-mentioned first distributed storage service as a distributed database, the server can build a query index corresponding to the metadata based on the index mechanism supported by the distributed database, and the client can initiate a retrieval based on the SQL statement supported by the distributed database to achieve efficient data retrieval and update.

[0139] Furthermore, if metadata is retrieved in the first distributed storage service, it indicates that the version file of the version has been pre-stored in the data warehouse deployed on the server, and the server can return the metadata of the version file of the version to the client. After obtaining the metadata of the version file of the version, the client can store the metadata of the version file of the version in the data warehouse deployed on the client.

[0140] In this way, when developers perform file version retrieval for the version file of that version, they can quickly retrieve and synchronize the metadata of the required file version in the first distributed storage service based on the metadata query index, without having to scan the complete version file in the second distributed storage service, thereby significantly improving the retrieval performance of the version control system when retrieving file versions.

[0141] In actual applications, developers can not only efficiently retrieve the metadata of the version file they want to retrieve from the first distributed storage service in the data warehouse deployed on the server based on the metadata of the version file they want to retrieve, but also efficiently retrieve the version file from the data warehouse deployed on the server based on the unique identifier of the version file contained in the metadata of the retrieved version file.

[0142] In one embodiment shown, when a developer has a need to retrieve the file content of a version file stored in a data warehouse deployed on a server (for example, when a developer wants to roll back a version file in a data warehouse deployed locally on a client to a historical version), an operation based on the unique identifier of the version file can be initiated on the work interface provided by the client. The client can generate a retrieval request in response to the operation; wherein the generated retrieval request can specifically include the unique identifier of the version file.

[0143] After generating a search request, the client can further send the search request to the server, and after receiving the search request, the server can retrieve the file content of the version file in the second distributed storage service in the data warehouse deployed on the server based on the unique identifier of the version file contained in the search request, and return the retrieved file content of the version file to the client. After obtaining the file content of the version file of the version, the client can store the file content of the version file of the version in the data warehouse deployed on the client.

[0144] Of course, if the version file of the version to be retrieved has been sharded in advance, the metadata of the version file of this version will usually contain the unique identifier of each version file shard in the multiple version file shards obtained by sharding the version file. In this case, the retrieval request generated by the client can specifically include the unique identifiers of the multiple version file shards, and the server can retrieve the file content of each version file shard from the second distributed storage service in turn based on the unique identifiers of the multiple version file shards, and return the retrieved file content of the multiple version file shards to the client.

[0145] For instance, in one example, taking the above-mentioned unique identifier as a hash value as an example, if the version file of the version to be retrieved has been pre-sharded, the metadata of the version file of this version will usually include a hash list consisting of the hash values ​​of the multiple version file shards. In this case, the retrieval request generated by the client can specifically include the hash list, and the server can retrieve the file content of each version file shard from the second distributed storage service in turn based on the hash value of each version file shard contained in the hash list, and return the retrieved file content of the multiple version file shards to the client.

[0146] In one embodiment shown, when the server returns the file content of the retrieved multiple version file fragments to the client, it can specifically return each version file fragment to the client in sequence. After the client obtains the file content of the multiple version file fragments, it can splice the file content of the multiple version file fragments to restore the file content of the original version file, and then store the restored file content of the version file in a data warehouse deployed locally on the client.

[0147] Through this sharding mechanism, after a single large version file is uploaded in pieces, it not only solves the problem of uneven distribution of data on various storage nodes in the distributed storage service when a single large version file is stored in the distributed storage service, but also enables the server to transmit each shard to the client in sequence in a manner close to breakpoint resumption when returning a single large version file to the client, without having to remember the specific transmission status, thereby enhancing the robustness of data transmission.

[0148] Of course, in actual applications, when the server returns the file contents of the retrieved multiple version file fragments to the client, the server can also splice the file contents of the multiple version file fragments to restore the file contents of the original version file, and then return the restored file contents of the version file to the client.

[0149] See also Figure 4 , Figure 4 This is a flowchart of a file version control management method shown in this specification, which is applied to Figure 1 The server in the VCS system architecture shown; the method includes the following execution process:

[0150] Step 402: receiving the version record uploaded by the client; wherein the version record includes metadata of the modified version file stored in the data warehouse deployed on the client and the file content of the modified version file;

[0151] For the specific implementation details of the client generating the version record based on the modified version file stored in its locally deployed data warehouse, please refer to the records of the previous embodiment, which will not be repeated in this embodiment.

[0152] After receiving the version record uploaded by the client, the server can parse the version record to obtain the metadata of the modified version file and the file content of the modified version file contained in the version record.

[0153] For example, when parsing the version record, the server may first obtain the metadata of the modified version file contained in the version record, and then further read the file content of the version file from the version record based on the unique identifier of the version file contained in the metadata.

[0154] In an embodiment shown, if there is a target version file whose file size reaches a preset first threshold value in the modified version file, then the version record received by the server may specifically include multiple version file fragments obtained by segmenting the target version file by the client, and the metadata of the target version file included in the version record may also include unique identifiers of the multiple version file fragments. For example, the unique identifier may be a hash value of the version file fragment.

[0155] In this case, when parsing the version record, the server can first obtain the metadata of the modified version file contained in the version record, and then read the multiple version file fragments from the version record in sequence based on the unique identifiers of the multiple version file fragments contained in the metadata.

[0156] Step 404: store the metadata included in the version record in the first distributed storage service, and store the file content included in the version record in the second distributed storage service.

[0157] After the server parses the version record uploaded by the client and obtains the metadata of the modified version file and the file content of the modified version file contained in the version record, the server can store the obtained metadata in the above-mentioned first distributed storage service, and store the obtained file content in the second distributed storage service to complete the synchronization of file versions between the data warehouse deployed locally on the client and the data warehouse deployed on the server.

[0158] Of course, for the multiple version file fragments corresponding to the above-mentioned target version file contained in the version record, the server can still store these version file fragments in the second distributed storage service in the form of fragments.

[0159] In one embodiment shown, as described above, if among the modified version files contained in the version record, there is a target version file whose file size reaches a preset first threshold, and the client segments the target version file to obtain version file fragments, which are compressed by the client, the server may also decompress the version file fragments contained in the version record before storing them in the second distributed storage service.

[0160] For example, the version file segments may be decompressed using a decompression method corresponding to a compression algorithm used by the client during compression processing.

[0161] In one embodiment shown, as described above, if there is at least one version file whose file size is lower than a preset second threshold value among the modified version files included in the version record, and the at least one version file is packaged and compressed by the client, then before the server stores the file contents of these version files included in the version record in the second distributed storage service, it can also first decompress the at least one version file included in the version record to obtain a package file corresponding to the at least one version file; and then parse the package file to split the package file into the at least one version file. For example, the decompression method corresponding to the compression algorithm used by the client can be used to decompress these version file fragments.

[0162] In one embodiment shown, as described above, if the metadata of the modified version file contained in the version record is packaged and compressed by the client, the server can first decompress the metadata of the modified version file contained in the version record before storing the metadata in the first distributed storage service to obtain a package file corresponding to the metadata; and then parse the package file to split the package file into metadata. For example, the package rule used by the client in the process of package compression can be used to reversely parse the package file to split each packaged metadata from the package file.

[0163] After the synchronization of file versions between the data warehouse deployed locally on the client and the data warehouse deployed on the server is completed, developers can efficiently retrieve version files from the data warehouse deployed on the server based on the metadata of the version files they want to retrieve when they have retrieval needs.

[0164] The client can generate a related search request in response to the operation initiated by the developer, and then send the generated search request to the server. After receiving the search request sent by the client, the server can complete the corresponding search action based on the query index contained in the search request.

[0165] In one embodiment shown, when a developer initiates an operation to retrieve a version of a version file based on the metadata of the version file, the retrieval request generated by the client at this time may generally include a query index constructed based on the metadata of the version file to be retrieved. After receiving the retrieval request, the server can retrieve the metadata in the first distributed storage service in the data warehouse deployed on the server based on the query index contained in the retrieval request. If the metadata is retrieved in the first distributed storage service, it indicates that the version file of this version has been pre-stored in the data warehouse deployed on the server. At this time, the server can return the metadata of the version file of this version to the client. After obtaining the metadata of the version file of this version, the client can store the metadata of the version file of this version in the data warehouse deployed on the client.

[0166] In one embodiment shown, when a developer initiates an operation to further retrieve the file content of a version of a version file based on the unique identifier of the version of the version file contained in the metadata of the retrieved version of the version file, the retrieval request generated by the client can usually include the unique identifier of the version file. After receiving the retrieval request, the server can retrieve the file content of the version file in the second distributed storage service in the data warehouse deployed on the server based on the unique identifier of the version file contained in the retrieval request, and return the retrieved file content of the version file to the client. After obtaining the file content of the version file of the version, the client can store the file content of the version file of the version in the data warehouse deployed on the client.

[0167] Of course, if the version file of the version to be retrieved has been sharded in advance, the metadata of the version file of this version will usually contain the unique identifier of each version file shard in the multiple version file shards obtained by sharding the version file. In this case, the retrieval request generated by the client can specifically include the unique identifiers of the multiple version file shards, and the server can retrieve the file content of each version file shard from the second distributed storage service in turn based on the unique identifiers of the multiple version file shards, and return the retrieved file content of the multiple version file shards to the client.

[0168] For example, in one embodiment shown, when the server returns the file content of the retrieved multiple version file fragments to the client, it can specifically return each version file fragment to the client in sequence. After the client obtains the file content of the multiple version file fragments, it can splice the file content of the multiple version file fragments to restore the file content of the original version file, and then store the restored file content of the version file in a data warehouse deployed locally on the client.

[0169] In one embodiment shown, in addition to the above-mentioned first distributed storage service and second distributed storage service, the data warehouse deployed on the server may also include a cache built based on the cache space on the server; wherein the cache is specifically used to cache hot data in the metadata and file content of the modified version files contained in the version records stored in the data warehouse deployed on the server. It should be explained that the so-called hot data generally refers to metadata and file content that need to be frequently accessed in the metadata and file content stored in the data warehouse deployed on the server. In actual applications, hot data can be flexibly determined from the metadata and file content stored in the data warehouse based on specific needs.

[0170] In this case, when the server retrieves metadata in the first distributed storage service based on the query index contained in the retrieval request, it can specifically first retrieve the metadata in the above-mentioned cache based on the query index; if the metadata is not retrieved in the cache, it can further retrieve the metadata in the first distributed storage service based on the query index.

[0171] Correspondingly, when the server retrieves the file content of the version file in the second distributed storage service based on the unique identifier of the version file contained in the retrieval request, it can also first retrieve the file content of the version file in the above-mentioned cache based on the unique identifier of the version file; if the file content of the version file is not retrieved in the cache, the server can further retrieve the file content of the version file in the second distributed storage service based on the unique identifier of the version file.

[0172] In this way, the retrieval efficiency of hot data in some metadata and file content can be improved.

[0173] In one embodiment shown, in order to further utilize the cache space on the server, the above-mentioned cache can specifically include a first-level cache built based on the memory space on the server, and a second-level cache built based on the disk space on the server; wherein the first-level cache can be specifically used to cache hot data in the metadata stored in the data warehouse; and the second-level cache space can be specifically used to cache hot data in the file content of the version file stored in the data warehouse.

[0174] In this case, when the server retrieves metadata in the first distributed storage service based on the query index contained in the retrieval request, it can first retrieve the metadata in the first-level cache, that is, the cache built based on the memory space, based on the query index; if the metadata is not retrieved in the first-level cache, the server can further retrieve the metadata in the first distributed storage service based on the query index.

[0175] Correspondingly, when the server retrieves the file content of the version file in the second distributed storage service based on the unique identifier of the version file contained in the retrieval request, it can also first retrieve the file content of the version file in the second-level cache, that is, the cache built based on the disk space; if the file content of the version file is not retrieved in the second-level cache, the server can further retrieve the file content of the version file in the second distributed storage service based on the unique identifier of the version file.

[0176] In this way, on the one hand, the memory space on the server can be used to cache metadata with a smaller capacity, and the memory space has the characteristic of higher query efficiency, so as to enhance the retrieval efficiency when retrieving metadata; on the other hand, the disk space on the server can be used to cache the file content of version files with a larger capacity but no high requirements for retrieval efficiency, so as to make full use of the disk space on the server.

[0177] In one embodiment shown, the hot data in the metadata stored in the data warehouse cached in the first-level cache may specifically be the hot data determined by the server by running a cache elimination algorithm to calculate the metadata stored in the data warehouse.

[0178] In this case, the server can cache all metadata stored in the data warehouse into the first-level cache by default, and then run the cache elimination algorithm to continuously delete non-hot data in the metadata cached in the first-level cache, so as to retain only the hot data in these metadata in the first cache.

[0179] The specific algorithm type of the cache elimination algorithm adopted by the server is not particularly limited in this embodiment; for example, in one example, the cache elimination algorithm may specifically include the LFULFU (Least Frequently Used) algorithm. For example, in practical applications, the TinyLFU algorithm or the SampleLFU algorithm in the LFU class of algorithms may be adopted.

[0180] In this way, the server's memory space can be continuously optimized by continuously running the cache elimination algorithm, thereby making the most reasonable use of the memory space.

[0181] In one embodiment shown, for hot data in the file content of version files stored in the data warehouse cached in the first-level cache, it is not necessary to determine it by running a cache elimination algorithm. Instead, the file content of version files of smaller sizes and not fragmented stored in the data warehouse can be used as hot data.

[0182] In this case, the server can obtain the file content of at least one version file whose size is lower than the second threshold stored in the data warehouse, and then cache the obtained file content of at least one version file as hot data in the second-level cache space.

[0183] In this way, the efficiency of retrieving the file contents of the smaller version files stored in the data warehouse can be improved.

[0184] In one embodiment shown, the second-level cache can also be used as a backup cache for the first-level cache. In this case, the second-level cache can also be used to cache hot data in the metadata that has been cached in the first-level cache. That is, the hot data in the metadata that has been cached in the first-level cache can also be stored in the second-level cache as a backup.

[0185] For example, in actual applications, when the server runs a cache elimination algorithm and continuously deletes non-hot data in the metadata cached in the first-level cache, so as to retain only the hot data in these metadata in the first cache, the hot data determined from the metadata cached in the first-level cache by running the cache elimination algorithm can be synchronized in real time to the second-level cache for storage.

[0186] In this case, when the server retrieves metadata in the first distributed storage service based on the query index contained in the retrieval request, the server can first retrieve the metadata in the first-level cache based on the query index; if the metadata is not retrieved in the first cache, it may be due to power failure or other reasons in the first-level cache that has caused data loss. At this time, the server can further retrieve the metadata in the second-level cache based on the query index; if the metadata is still not retrieved in the second-level cache, the server can further retrieve the metadata in the first distributed storage service based on the query index.

[0187] Of course, in actual applications, since the above-mentioned second-level cache can be used as a backup cache for the above-mentioned first-level cache, when the above-mentioned first-level cache encounters situations such as power failure that cause data loss, the server can also read the hot data in the above-mentioned metadata cached in the second-level cache, and reload the hot data in the above-mentioned metadata read into the above-mentioned first-level cache to perform abnormal recovery of the first-level cache.

[0188] In the above technical solution, on the one hand, since the central data warehouse deployed on the above-mentioned server is a data warehouse built based on the distributed storage service, it has the ability to expand capacity infinitely; therefore, developers do not need to deploy additional LFS services for the central data warehouse, and can synchronize the modified large file versions stored in the local data warehouse deployed on the client to the central data warehouse deployed on the server for storage, thereby not increasing the user's learning cost and the dependence of the entire VCS system's services on the LFS service.

[0189] On the other hand, when the client synchronizes a modified, very large file version stored in the local data warehouse to the central data warehouse deployed on the server for storage, the metadata of the version file and the file content of the version file are stored separately. This allows developers to quickly retrieve and synchronize to the required file version in the distributed storage service that stores the metadata when they need to retrieve file versions, without having to scan the file content of the complete version file stored in the distributed storage service that stores the file content of the version file. This can significantly improve the retrieval performance of the version control system when retrieving file versions.

[0190] For example, assuming that a developer wants to query a historical version of a version file stored in the data warehouse deployed on the server, the developer may not need to synchronize the file content of the version file of that version from the data warehouse deployed on the server to the local data warehouse on the client for the time being. At this time, if the developer performs file version retrieval for the version file of that version, based on the metadata query index, the metadata of the required file version can be quickly retrieved and synchronized in the first distributed storage service without scanning the complete version file in the second distributed storage service. This can obviously significantly improve the retrieval performance when retrieving file versions.

[0191] Moreover, if the developer wants to roll back a version file of a certain version in the data warehouse deployed locally on the client to the version file of that version after viewing the metadata of the version file of that version, he can also continue to retrieve the file content of the version file in the above-mentioned second distributed storage service based on the unique identifier of the version file contained in the metadata.

[0192] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0193] Figure 5 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include other required hardware. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0194] like Figure 6 As shown, Figure 6 This is a block diagram of a file version control management according to an exemplary embodiment of the present specification. The device can be applied to Figure 5 In the electronic device shown, the technical solution of this specification is implemented. Wherein, a data warehouse is deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; the device 600 includes:

[0195] An acquisition module 601 is used to acquire a modified version file stored in a locally deployed data warehouse;

[0196] A generation module 602 generates a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file;

[0197] The upload module 603 uploads the version record to the server, so that the server stores the metadata contained in the version record to the first distributed storage service, and stores the file content contained in the version record to the second distributed storage service.

[0198] like Figure 7 As shown, Figure 7is a block diagram of another file version control management shown in this specification according to an exemplary embodiment. The device can also be applied to Figure 5 In the electronic device shown, the technical solution of this specification is implemented. Wherein, a data warehouse is deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; the device 700 includes:

[0199] The receiving module 701 receives the version record uploaded by the client; wherein the version record includes metadata of the modified version file stored in the data warehouse deployed on the client and the file content of the modified version file;

[0200] The storage module 702 stores the metadata included in the version record in the first distributed storage service, and stores the file content included in the version record in the second distributed storage service.

[0201] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.

[0202] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0203] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0204] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0205] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0206] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0207] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0208] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0210] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0211] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0212] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0213] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0214] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0215] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0216] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0217] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0218] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims.

Claims

1. A file version control management method, applied to a client of a version control system; wherein: Data warehouses are deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; including: Obtain the modified version files stored in the locally deployed data warehouse; Generate a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file; The version record is uploaded to the server, so that the server stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

2. The method according to claim 1, before generating a version record based on the modified version file, further comprises: Determine whether there is a target version file whose file size reaches a preset first threshold value among the modified version files; If the target version file exists in the modified version file, the target version file is divided into multiple version file fragments, and the unique identifiers of the multiple version file fragments are respectively added to the metadata of the target version file to establish a link relationship between the metadata and the multiple version file fragments.

3. The method according to claim 2, before uploading the version record to the server, further comprises: Determine whether there is a version file whose file size is lower than a preset second threshold value among the modified version files included in the version record; wherein the second threshold value is lower than the first threshold value; If there is at least one version file whose file size is lower than a preset second threshold value among the modified version files included in the version record, the at least one version file is packaged and compressed.

4. The method of claim 2, further comprising: In response to an operation initiated by a user to retrieve a version file based on the metadata of the version file, a search request is generated; wherein the search request includes a query index constructed based on the metadata of the version file; Sending the retrieval request to the server, so that the server retrieves the metadata in the first distributed storage service based on the query index; The metadata of the version file returned by the server is obtained, and the obtained file content of the version file is stored in a data warehouse deployed on the client.

5. The method according to claim 4, wherein the metadata includes a unique identifier of the version file; The method further comprises: In response to an operation initiated by a user to retrieve a version file based on a unique identifier of the version file, a retrieval request is generated; wherein the retrieval request includes the unique identifier of the version file; Sending the retrieval request to the server, so that the server retrieves the file content of the version file in the second distributed storage service based on the unique identifier of the version file; The file content of the version file returned by the server is obtained, and the obtained file content of the version file is stored in a data warehouse deployed on the client.

6. In the method as claimed in claim 1, the first distributed storage service is a distributed database; the second distributed storage service is a distributed file system or a distributed object storage system.

7. A file version control management method, applied to the server of a version control system; wherein: Data warehouses are deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; including: Receiving the version record uploaded by the client; wherein the version record includes metadata of the modified version file stored in the data warehouse deployed on the client and file content of the modified version file; The metadata included in the version record is stored in the first distributed storage service, and the file content included in the version record is stored in the second distributed storage service.

8. According to the method as claimed in claim 7, if there is a target version file in the modified version file whose file size reaches a preset first threshold, then the version record contains multiple version file fragments obtained by the client splitting the target version file, and the metadata of the target version file contained in the version record contains unique identifiers of the multiple version file fragments.

9. The method according to claim 7, wherein among the modified version files contained in the version record, there is at least one version file whose file size is lower than a preset second threshold; wherein The second threshold is lower than the first threshold; the at least one version file is packaged and compressed by the client; Before storing the metadata included in the version record in the first distributed storage service and storing the file content included in the version record in the second distributed storage service, the method further includes: Decompressing the at least one version file contained in the version record to obtain a packaged file corresponding to the at least one version file; The package file is parsed to split the package file into the at least one version file.

10. The method of claim 7, further comprising: Receiving a search request sent by the client; wherein the search request includes a query index constructed based on metadata of the version file to be searched; Retrieving the metadata in the first distributed storage service based on the query index; If the metadata is retrieved in the first distributed storage service, the metadata is returned to the client.

11. The method according to claim 10, wherein the metadata includes a unique identifier of the version file; The method further comprises: Receiving a search request sent by the client; wherein the search request includes a unique identifier of the version file to be searched; Based on the unique identifier of the version file, the file content of the version file is retrieved in the second distributed storage service, and the retrieved file content of the version file is returned to the client.

12. The method according to claim 11, wherein the data warehouse deployed on the server further comprises a cache constructed based on the cache space on the server; wherein, The cache caches the metadata stored in the data warehouse and hot data in the file content; Retrieving the metadata in the first distributed storage service based on the query index includes: Retrieving the metadata in the cache based on the query index; if the metadata is not retrieved in the cache, further retrieving the metadata in the first distributed storage service based on the query index constructed from the metadata; Retrieving the file content of the version file in the second distributed storage service based on the unique identifier of the version file includes: Based on the unique identifier of the version file, the file content of the version file is retrieved in the cache; if the file content of the version file is not retrieved in the cache, the file content of the version file is further retrieved in the second distributed storage service based on the unique identifier of the version file.

13. The method according to claim 12, wherein the cache comprises a first-level cache constructed based on memory space and a second-level cache constructed based on disk space; wherein, The first level cache is used to cache hot data in the metadata stored in the data warehouse; the second level cache space is used to cache hot data in the file content stored in the data warehouse; Retrieving the metadata in the cache based on the query index; if the metadata is not retrieved in the cache, further retrieving the metadata in the first distributed storage service based on the query index constructed from the metadata, including: Retrieving the metadata in the first level cache based on the query index; if the metadata is not retrieved in the first cache, further retrieving the metadata in the first distributed storage service based on the query index; Based on the unique identifier of the version file, searching the cache for the file content of the target version file; if the file content of the version file is not found in the cache, further searching the second distributed storage service for the file content of the version file based on the unique identifier of the version file, including: Based on the unique identifier of the version file, the file content of the version file is retrieved in the second cache; if the file content of the version file is not retrieved in the second cache, the file content of the version file is further retrieved in the second distributed storage service based on the unique identifier of the version file.

14. The method of claim 13, further comprising: Cache the metadata stored in the data warehouse into the first-level cache, and run a cache elimination algorithm to delete non-hotspot data in the metadata cached in the first-level cache, so as to retain the hotspot data in the metadata in the first cache; The file content of at least one version file whose size is lower than the second threshold and stored in the data warehouse is obtained, and the obtained file content is cached in the second-level cache space.

15. The method according to claim 13, wherein the second-level cache is a backup cache of the first-level cache; the second-level cache is also used to cache hot data in the metadata that has been cached in the first-level cache; Retrieving the metadata in the first-level cache based on the query index; if the metadata is not retrieved in the first cache, further retrieving the metadata in the first distributed storage service based on the query index constructed from the metadata, including: Retrieving the metadata in the first level cache based on the query index; If the metadata is not retrieved in the first cache, the metadata is retrieved in the second-level cache based on the query index; if the metadata is still not retrieved in the second-level cache, the metadata is further retrieved in the first distributed storage service based on the query index.

16. The method according to claim 7, wherein the first distributed storage service is a distributed database; and the second distributed storage service is a distributed file system or a distributed object storage system.

17. A file version control management system, comprising a client and a server; data warehouses are deployed on the client and the server respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; wherein: The client is used to obtain a modified version file stored in a locally deployed data warehouse, and generate a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file; and upload the version record to the server; The server is used to receive the version record uploaded by the client; store the metadata contained in the version record in the first distributed storage service, and store the file content contained in the version record in the second distributed storage service.

18. A file version control management device, applied to a client of a version control system; wherein: Data warehouses are deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; the device includes: An acquisition module is used to obtain modified version files stored in a locally deployed data warehouse; A generation module generates a version record based on the modified version file; wherein the version record includes metadata of the modified version file and file content of the modified version file; An upload module uploads the version record to the server, so that the server stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

19. A file version control management device, applied to the server of a version control system; wherein: Data warehouses are deployed on the client of the version control system and the server of the version control system respectively; the data warehouse deployed on the server adopts a distributed architecture, including a first distributed storage service for storing metadata of version files and a second distributed storage service for storing file contents of version files; the device includes: A receiving module receives the version record uploaded by the client; wherein the version record includes metadata of the modified version file stored in the data warehouse deployed on the client and the file content of the modified version file; The storage module stores the metadata contained in the version record in the first distributed storage service, and stores the file content contained in the version record in the second distributed storage service.

20. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

22. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images