Method and system for storing and updating model data

Abstract

The invention relates to a method and system for storing and updating model data, and the method comprises the steps: adding N keywords to obtained model data according to a preset data range, carrying out the fragmentation of the model data after the N keywords are added according to the range of M keywords, and obtaining all fragmented data, each piece of fragmented data is uploaded to different fragmented servers for storage, the keyword range of the corresponding fragmented data on each fragmented server is recorded, and when the model data is changed, the corresponding fragmented server is updated according to the keyword range to which the changed data range belongs. Therefore, when the model data is changed, the changed fragment server can be determined only by determining the keyword range to which the model data belongs according to the changed data range, the whole model data does not need to be completely updated, and only the corresponding fragment server needs to be updated, so that the updating efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for updating model data storage. Background Art

[0002] Currently, artificial intelligence is the hottest field and will develop in all walks of life. The basis for realizing artificial intelligence is to train various data models and conduct intelligent analysis through data models. Usually, the data for training data models is in the billions, which can generate hundreds of GB of model data. However, the storage of these model data is usually directly stored at one time, which takes a long time. When the model data needs to be updated, it is necessary to re-upload new hundreds of GB of model data, which increases the overall time consumption and affects the update efficiency.

[0003] Further optimization was carried out later, and the model data was split into multiple smaller file data for storage. Although this improved the upload efficiency and shortened the storage time, when the model data needed to be updated, it was still necessary to completely re-upload the updated model data, which increased the overall time consumption and affected the update efficiency. Summary of the invention

[0004] The technical problem to be solved by the present invention is: the present invention provides a method and system for storing and updating model data to improve the updating efficiency.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a method for storing and updating model data, comprising:

[0007] Acquire model data, add N keywords to the model data according to a preset data range, and obtain the model data after the keywords are added;

[0008] The model data after adding keywords is sharded according to M keyword ranges, all shard data is obtained, all shard servers are obtained, each piece of shard data is uploaded to a different shard server for storage, and the keyword range of the corresponding shard data on each shard server is recorded;

[0009] When the model data is changed, the changed data range is obtained, and the corresponding shard server is updated according to the keyword range to which the changed data range belongs.

[0010] The beneficial effect of the present invention is that by adding keywords to model data and sharding the model data according to keyword ranges, the obtained shard data is uploaded to different shard servers. Since each shard server records the keyword range, when the model data changes, it is only necessary to determine the keyword range to which the changed data range belongs based on the changed data range to determine the shard server that has changed. There is no need to completely update the entire model data, and only the corresponding shard server needs to be updated, thereby improving the update efficiency.

[0011] Optionally, obtaining all shard servers and uploading each shard data to a different shard server for storage includes:

[0012] The server number of the shard server and the shard number of the shard data are obtained, and when the shard number exceeds the server number, the shard data is uploaded to different shard servers in batches for storage.

[0013] According to the above description, when the number of shards of sharded data exceeds the number of shard servers, the sharded data will be uploaded in batches to improve the efficiency of data upload.

[0014] Optionally, a concurrent uploading method is adopted when uploading each piece of shard data to different shard servers for storage.

[0015] According to the above description, the concurrent uploading method is used to improve the data uploading efficiency.

[0016] Optionally, when the model data is changed, obtaining a changed data range, and updating a corresponding shard server according to a keyword range to which the changed data range belongs includes:

[0017] Determine whether there is a keyword range corresponding to the changed data range. If so, update the corresponding shard server according to the keyword range to which the changed data range belongs. If not, add a new keyword to the changed data range to obtain the changed data range after adding the new keyword, and shard the changed data range after adding the new keyword to obtain new shard data, and upload the new shard data to the shard server for storage.

[0018] According to the above description, when the changed data range does not have a corresponding keyword range, it means that the changed data range is newly added data based on the original model data. Therefore, new keywords are added to the changed data range, and the changed data range after adding the new keywords is sharded, and then uploaded to the sharding server to realize the upload and storage of the newly added data and lay the foundation for the subsequent update of this part of the content.

[0019] Optionally, M is 2.

[0020] In a second aspect, the present invention provides a system for storing and updating model data, comprising:

[0021] A keyword adding module, used to obtain model data, add N keywords to the model data according to a preset data range, and obtain the model data after the keywords are added;

[0022] The sharding module is used to shard the model data after adding keywords according to M keyword ranges, obtain all shard data, obtain all shard servers, upload each piece of shard data to different shard servers for storage, and record the keyword range of the corresponding shard data on each shard server;

[0023] The update module is used to obtain the changed data range when the model data is changed, and update the corresponding shard server according to the keyword range to which the changed data range belongs.

[0024] The beneficial effect of the present invention is that by adding keywords to model data and sharding the model data according to keyword ranges, the obtained shard data is uploaded to different shard servers. Since each shard server records the keyword range, when the model data changes, it is only necessary to determine the keyword range to which the changed data range belongs based on the changed data range to determine the shard server that has changed. There is no need to completely update the entire model data, and only the corresponding shard server needs to be updated, thereby improving the update efficiency.

[0025] Optionally, the sharding module includes:

[0026] The uploading module is used to obtain the server number of the shard server and the number of shards of the shard data. When the number of shards exceeds the number of servers, the shard data is uploaded in batches to different shard servers for storage.

[0027] According to the above description, when the number of shards of sharded data exceeds the number of shard servers, the sharded data will be uploaded in batches to improve the efficiency of data upload.

[0028] Optionally,

[0029] The fragmentation module is specifically:

[0030] The method of uploading each piece of shard data to different shard servers for storage is adopted in a concurrent uploading manner.

[0031] According to the above description, the concurrent uploading method is used to improve the data uploading efficiency.

[0032] Optionally, the update module is specifically:

[0033] Determine whether there is a keyword range corresponding to the changed data range. If so, update the corresponding shard server according to the keyword range to which the changed data range belongs. If not, add a new keyword to the changed data range to obtain the changed data range after adding the new keyword, and shard the changed data range after adding the new keyword to obtain new shard data, and upload the new shard data to the shard server for storage.

[0034] According to the above description, when the changed data range does not have a corresponding keyword range, it means that the changed data range is newly added data based on the original model data. Therefore, new keywords are added to the changed data range, and the changed data range after adding the new keywords is sharded, and then uploaded to the sharding server to realize the upload and storage of the newly added data and lay the foundation for the subsequent update of this part of the content.

[0035] Optionally, M is 2. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a method for storing and updating model data provided in this embodiment;

[0037] Figure 2 A schematic diagram of the overall process of a method for storing and updating model data provided in this embodiment;

[0038] Figure 3 A schematic diagram of the structure of a system for storing and updating model data provided in this embodiment.

[0039] [Description of Reference Numerals]

[0040] 1. A system for storing and updating model data;

[0041] 2. Keyword adding module;

[0042] 3. Sharding module; 31. Upload module;

[0043] 4. Update module. DETAILED DESCRIPTION

[0044] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0045] Embodiment 1

[0046] Please refer to Figure 1 to Figure 2 The present invention provides a method for storing and updating model data, comprising the steps of:

[0047] S1. Obtain model data, add N keywords to the model data according to a preset data range, and obtain the model data after the keywords are added;

[0048] In this embodiment, if Figure 2 As shown, N keywords are added to the acquired model data according to the preset data range. For example, the preset data range is 10 rows of data, that is, keywords are added to the model data according to the data range of every 10 rows of data. The keywords can be in the form of keyi, where i represents the ith one, such as the first keyword is key1, and the second keyword is key2. The number of keywords N here depends on how many 10 rows of data the model data can be divided into. The specific preset data range can be adjusted according to the actual situation, so as to obtain the model data after adding keywords.

[0049] S2. Slice the model data after adding keywords according to M keyword ranges, obtain all slicing data, obtain all slicing servers, upload each slicing data to different slicing servers for storage, and record the keyword range of the corresponding slicing data on each slicing server;

[0050] In this embodiment, if Figure 2As shown, the model data after adding keywords is sharded according to M keyword ranges, such as M is 2, that is, sharded according to 2 keyword ranges, such as: there are 6 keywords at this time, namely: key1, key2, key3, key4, key5 and key6, then the first shard data is: keyword key1-keyword key2 data, the second shard data is: keyword key3-keyword key4 data, the third shard data is: keyword key5-keyword key6 data, so as to obtain all shard data, obtain the shard server, upload each shard data to a different shard server for storage, at this time, adopt the concurrent upload method, and record the keyword range of the corresponding shard data on each shard server, such as: the first shard data is uploaded to shard server 1, then the keyword range of shard server 1 is: key1-key2, the second shard data is uploaded to shard server 2, then the keyword range of shard server 2 is: key3-key4, and the specific value of M can be adjusted according to actual conditions.

[0051] At this time, the acquisition of all shard servers in step S2 and uploading each shard data to different shard servers for storage includes:

[0052] S21. Obtain the server number of the shard server and the number of shards of the shard data. When the number of shards exceeds the number of servers, upload the shard data in batches to different shard servers for storage.

[0053] In this embodiment, if Figure 2 As shown, when the number of shards of shard data exceeds the number of shard servers, the shard data is uploaded to different shard servers in batches for storage.

[0054] S3. When the model data is changed, the changed data range is obtained, and the corresponding shard server is updated according to the keyword range to which the changed data range belongs.

[0055] In this embodiment, if Figure 2 As shown, when the model data changes, the keyword range to which the changed data belongs is determined by obtaining the changed data range, and the corresponding shard server is determined according to the keyword range, and the shard server is updated.

[0056] At this time, when the model data is changed in step S3, obtaining the changed data range, and updating the corresponding shard server according to the keyword range to which the changed data range belongs includes:

[0057] S31. Determine whether there is a keyword range corresponding to the changed data range. If so, update the corresponding shard server according to the keyword range to which the changed data range belongs. If not, add a new keyword to the changed data range to obtain the changed data range after adding the new keyword, and shard the changed data range after adding the new keyword to obtain new shard data, and upload the new shard data to the shard server for storage.

[0058] In this embodiment, if Figure 2 As shown, when the model data changes, the changed data range is obtained, and it is determined whether there is a corresponding keyword range for the changed data range, that is, if the changed data range is rows 10-15, then its corresponding keyword range is key1-key2. Therefore, according to the keyword range, its corresponding shard server is determined to be shard server 1. Therefore, shard server 1 is updated. If the changed data range is rows 100-110, there is no corresponding keyword range. Therefore, a new keyword is added to the changed data range and the changed data range after adding the new keyword is sharded, so that the obtained new sharded data is uploaded to the shard server for storage.

[0059] Embodiment 2

[0060] Please refer to Figure 3 The present invention provides a system 1 for storing and updating model data, including: a keyword adding module 2, a sharding module 3, an uploading module 31 and an updating module 4.

[0061] The keyword adding module 2 is used to obtain model data, add N keywords to the model data according to a preset data range, and obtain the model data after adding keywords;

[0062] Sharding module 3, used to shard the model data after adding keywords according to M keyword ranges, obtain all shard data, obtain all shard servers, upload each piece of shard data to different shard servers for storage, and record the keyword range of the corresponding shard data on each shard server;

[0063] The updating module 4 is used to obtain the changed data range when the model data is changed, and update the corresponding shard server according to the keyword range to which the changed data range belongs.

[0064] Specifically, the slicing module 3 includes:

[0065] The uploading module 31 is used to obtain the server number of the shard server and the number of shards of the shard data. When the number of shards exceeds the number of servers, the shard data is uploaded to different shard servers in batches for storage.

[0066] Specifically, the slicing module 31 is specifically:

[0067] The method of uploading each piece of shard data to different shard servers for storage is adopted in a concurrent uploading manner.

[0068] Specifically, the updating module 4 is:

[0069] Determine whether there is a keyword range corresponding to the changed data range. If so, update the corresponding shard server according to the keyword range to which the changed data range belongs. If not, add a new keyword to the changed data range to obtain the changed data range after adding the new keyword, and shard the changed data range after adding the new keyword to obtain new shard data, and upload the new shard data to the shard server for storage.

[0070] Specifically, M is 2.

[0071] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0072] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0074] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0075] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0076] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0077] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A method for storing and updating model data, characterized in that: include: Acquire model data, add N keywords to the model data according to a preset data range, and obtain the model data after the keywords are added; The model data after adding keywords is sharded according to M keyword ranges, all shard data is obtained, all shard servers are obtained, each piece of shard data is uploaded to a different shard server for storage, and the keyword range of the corresponding shard data on each shard server is recorded; When the model data is changed, the changed data range is obtained, and the corresponding shard server is updated according to the keyword range to which the changed data range belongs.

2. A method for storing and updating model data according to claim 1, characterized in that: The obtaining of all shard servers and uploading each shard data to a different shard server for storage includes: The server number of the shard server and the shard number of the shard data are obtained, and when the shard number exceeds the server number, the shard data is uploaded to different shard servers in batches for storage.

3. A method for storing and updating model data according to claim 1, characterized in that: The method of uploading each piece of shard data to different shard servers for storage is adopted in a concurrent uploading manner.

4. A method for storing and updating model data according to claim 1, characterized in that: When the model data is changed, obtaining the changed data range, and updating the corresponding shard server according to the keyword range to which the changed data range belongs includes: Determine whether there is a keyword range corresponding to the changed data range. If so, update the corresponding shard server according to the keyword range to which the changed data range belongs. If not, add a new keyword to the changed data range to obtain the changed data range after adding the new keyword, and shard the changed data range after adding the new keyword to obtain new shard data, and upload the new shard data to the shard server for storage.

5. A method for storing and updating model data according to claim 1, characterized in that: The M is 2.

6. A system for storing and updating model data, characterized in that: include: A keyword adding module, used to obtain model data, add N keywords to the model data according to a preset data range, and obtain the model data after the keywords are added; The sharding module is used to shard the model data after adding keywords according to M keyword ranges, obtain all shard data, obtain all shard servers, upload each piece of shard data to different shard servers for storage, and record the keyword range of the corresponding shard data on each shard server; The update module is used to obtain the changed data range when the model data is changed, and update the corresponding shard server according to the keyword range to which the changed data range belongs.

7. A system for storing and updating model data as claimed in claim 6, characterized in that: The sharding module includes: The uploading module is used to obtain the server number of the shard server and the number of shards of the shard data. When the number of shards exceeds the number of servers, the shard data is uploaded in batches to different shard servers for storage.

8. A system for storing and updating model data as claimed in claim 6, characterized in that: The fragmentation module is specifically: The method of uploading each piece of shard data to different shard servers for storage is adopted in a concurrent uploading manner.

9. A system for storing and updating model data as claimed in claim 6, characterized in that: The update module is specifically: Determine whether there is a keyword range corresponding to the changed data range. If so, update the corresponding shard server according to the keyword range to which the changed data range belongs. If not, add a new keyword to the changed data range to obtain the changed data range after adding the new keyword, and shard the changed data range after adding the new keyword to obtain new shard data, and upload the new shard data to the shard server for storage.

10. A system for storing and updating model data as claimed in claim 6, characterized in that: The M is 2.

Images