Intelligent SQL database processing method and system

Abstract

The invention relates to the field of big data processing, and discloses an intelligent SQL (Structured Query Language) database processing method, which comprises the following steps of: when first data is stored, judging whether content which is overlapped with the first data and exceeds a second threshold value of the content of the first data exists in second data or not; and when the first data is called, synthesizing the third data corresponding to the first data stored in the data list and the second data into the first data and outputting the first data. According to the method, the overlapped part of the first data and the second data with relatively large storage capacity is not stored, so that the storage capacity pressure of the SQL database is reduced, and the first data and the second data are distributed in a non-overlapped manner as the third data to be independently stored in the SQL database. By means of the method, storage with smaller capacity can be achieved, and it is guaranteed that the relative calling speed is high.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a processing method for optimizing an SQL database. Background Art

[0002] SQL (Structured Query Language) is a database language with multiple functions such as data manipulation and data definition. This language is interactive and can provide great convenience for users. Database management systems should make full use of SQL language to improve the work quality and efficiency of computer application systems. SQL language can not only be used independently in terminals, but also as a sub-language to provide effective assistance for other program designs. In this program application, SQL can optimize program functions together with other program languages, thereby providing users with more comprehensive information.

[0003] The current SQL database data storage and retrieval is simply storage and retrieval, and cannot identify duplicate data or partial data, which increases its storage pressure and occupies a lot of valuable storage space.

[0004] Therefore, there is a need for an intelligent SQL database processing method that can optimize storage and retrieval methods to reduce data storage space and ensure a relatively fast retrieval speed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an intelligent SQL database processing method that can optimize storage and retrieval methods to reduce data storage space and ensure a relatively fast retrieval speed.

[0006] The present invention provides an intelligent SQL database processing method, comprising:

[0007] When storing the first data:

[0008] Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database by using a hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data; if so, it is not necessary to store the first data in the SQL database and the first hash value of the first data is updated to the data list, that is, the first data is deemed to have been stored in the SQL database;

[0009] If not, determine whether the capacity of the first data exceeds the first threshold; if not, store the first data in the SQL database and update the first hash value of the first data to the data list; if exceeded, determine whether the second data has content that overlaps with the first data and exceeds the second threshold of the content of the first data; if not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0010] If so, convert the non-overlapping part of the first data and the second data into the third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list;

[0011] When retrieving the first data:

[0012] It is determined according to the data list whether there is a second hash value of the second data which is the same as the first hash value of the first data in the SQL database; if so, the second data is converted into the first data and output; if not, the third data and the second data corresponding to the first data stored in the data list are synthesized into the first data and output.

[0013] The present invention provides an intelligent SQL database processing system, wherein the non-overlapping part of the first data and the second data is converted into the third data, and a third hash value of the third data is generated, the third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the overlapping part of the first data and the second data is updated to the data list, including:

[0014] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0015] S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304;

[0016] S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301;

[0017] S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

[0018] The present invention provides an intelligent SQL database processing system, wherein the non-overlapping part of the first data and the second data is converted into the third data, and a third hash value of the third data is generated, the third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the overlapping part of the first data and the second data is updated to the data list, including:

[0019] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0020] S302, converting the fourth data into a format processable by the convolutional neural network;

[0021] S303, constructing and training a convolutional neural network model, wherein the convolutional neural network model is capable of receiving fourth data in a format that can be processed by the convolutional neural network model, and outputting a predicted combination scheme of at least one fourth data, wherein each fourth data in the combination scheme has no intersection with a position that overlaps with the first data;

[0022] S304, using the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data;

[0023] S305, calculating the first data of each combination scheme after deleting the content overlapping with the at least one fourth data, and the remaining part is the third data, and selecting the combination scheme corresponding to the smallest content capacity of the third data as the optimal solution;

[0024] S306. Output the optimal solution and its corresponding third data, wherein the third data includes the data sequence number of the overlapping second data and the position of the second data, and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0025] The present invention provides an intelligent SQL database processing system, wherein the memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data are updated to the data list, including:

[0026] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0027] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, retaining only the second data of the same file type as the first data and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0028] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0029] The present invention provides an intelligent SQL database processing system, wherein the memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data are updated to the data list, including:

[0030] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0031] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, deleting the fourth data with the fifth threshold percentage from the fourth data and the fourth data with the longest storage time away from the fourth data with the largest file capacity from all the fourth data, and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0032] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0033] The present invention provides an intelligent SQL database processing system, comprising:

[0034] The data storage module, when storing the first data:

[0035] Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database by using a hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data; if so, it is not necessary to store the first data in the SQL database and the first hash value of the first data is updated to the data list, that is, the first data is deemed to have been stored in the SQL database;

[0036] If not, determine whether the capacity of the first data exceeds the first threshold; if not, store the first data in the SQL database and update the first hash value of the first data to the data list; if exceeded, determine whether the second data has content that overlaps with the first data and exceeds the second threshold of the content of the first data; if not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0037] If so, convert the non-overlapping part of the first data and the second data into the third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list;

[0038] The data retrieval module, when it retrieves the first data:

[0039] It is determined according to the data list whether there is a second hash value of the second data which is the same as the first hash value of the first data in the SQL database; if so, the second data is converted into the first data and output; if not, the third data and the second data corresponding to the first data stored in the data list are synthesized into the first data and output.

[0040] The present invention provides an intelligent SQL database processing system, wherein the non-overlapping part of the first data and the second data is converted into the third data, and a third hash value of the third data is generated, the third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the overlapping part of the first data and the second data is updated to the data list, including:

[0041] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0042] S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304;

[0043] S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301;

[0044] S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

[0045] The present invention provides an intelligent SQL database processing system, wherein the non-overlapping part of the first data and the second data is converted into the third data, and a third hash value of the third data is generated, the third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the overlapping part of the first data and the second data is updated to the data list, including:

[0046] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0047] S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304;

[0048] S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301;

[0049] S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

[0050] The present invention provides an intelligent SQL database processing system, wherein the memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data are updated to the data list, including:

[0051] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0052] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, retaining only the second data of the same file type as the first data and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0053] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0054] The present invention provides an intelligent SQL database processing system, wherein the memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data are updated to the data list, including:

[0055] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0056] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, deleting the fourth data with the fifth threshold percentage from the fourth data and the fourth data with the longest storage time away from the fourth data with the largest file capacity from all the fourth data, and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0057] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0058] The difference between the intelligent SQL database processing method of the present invention and the prior art is that the intelligent SQL database processing method of the present invention first finds the same data through the hash value of the data, so that the second data with the same hash value is stored as the first data, so there is no need to directly store the first data, thereby reducing the storage capacity pressure of the SQL database; if it is not found, the first data with too small storage capacity is not split, but directly stored to improve the retrieval speed; and the overlapping part of the first data with a larger storage capacity and the second data is not stored, thereby reducing the storage capacity pressure of the SQL database, and the non-overlapping distribution of the first data and the second data is stored separately as the third data in the SQL database. The present invention can achieve smaller capacity storage and ensure a relatively fast retrieval speed through the above method.

[0059] The following is a further description of an intelligent SQL database processing method of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of the storage phase of an intelligent SQL database processing method. DETAILED DESCRIPTION

[0061] like Figure 1 As shown, the present invention provides an intelligent SQL database processing method, comprising:

[0062] When storing the first data:

[0063] Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database by using a hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data; if so, it is not necessary to store the first data in the SQL database and the first hash value of the first data is updated to the data list, that is, the first data is deemed to have been stored in the SQL database;

[0064] If not, determine whether the capacity of the first data exceeds the first threshold; if not, store the first data in the SQL database and update the first hash value of the first data to the data list; if exceeded, determine whether the second data has content that overlaps with the first data and exceeds the second threshold of the content of the first data; if not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0065] If so, convert the non-overlapping part of the first data and the second data into the third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list;

[0066] When retrieving the first data:

[0067] It is determined according to the data list whether there is a second hash value of the second data which is the same as the first hash value of the first data in the SQL database; if so, the second data is converted into the first data and output; if not, the third data and the second data corresponding to the first data stored in the data list are synthesized into the first data and output.

[0068] The present invention first finds the same data through the hash value of the data, and then stores the second data with the same hash value as the first data, so that the first data does not need to be directly stored, thereby reducing the storage capacity pressure of the SQL database; if it is not found, the first data with too small storage capacity is not split, but directly stored to improve the retrieval speed; and the overlapping part of the first data with a larger storage capacity and the second data is not stored, thereby reducing the storage capacity pressure of the SQL database, and the non-overlapping distribution of the first data and the second data is stored separately as the third data in the SQL database. The present invention can achieve a smaller capacity storage and ensure a relatively fast retrieval speed through the above method.

[0069]

[0070] It should be noted that the content and hash value of the above data list are just examples. The actual hash should be obtained by SHA256 algorithm or SHA512 algorithm to ensure that the probability of coincidence is low and avoid data storage errors caused by different data having the same hash value. In addition, the above content can be video content, photo content, text content, audio content, and code content.

[0071] Among them, the first threshold value can be [1b, positive infinity), preferably 10mb. That is to say, when the second data that is the same as the first data has never been stored in the SQL database, and the capacity of the first data does not exceed 10mb, the first data is directly stored in the SQL database without being split, so as to ensure the retrieval speed. Because of the current retrieval characteristics of the hard disk, when the total capacity of a single retrieval remains unchanged, the more files there are and the smaller the storage capacity of a single file, the slower the retrieval speed. Then, small splits and excessive numbers of files are bound to affect the retrieval speed. Therefore, when the capacity of the first data is too small, it is not recommended to split it. After all, such small-capacity files will not take up too much storage space.

[0072] Among them, the second threshold value can be [0.1%, 99.9%], preferably 10%. That is to say, as long as the capacity of the entire content of the second data and the content of a whole block of the first data overlaps and occupies more than 10% of the total capacity of the first data, this whole block of first data will not be stored, but only recorded in the data list. When it is needed, the second data as the overlapping part and the third data as the non-overlapping part will be integrated into the first data and output. It should be explained that this whole block of overlapping data can no longer be calculated to overlap with other second data. In other words, when the overlapping part of a whole block of first data and one of the second data is determined, it will be deleted, and then it will be checked whether the overlap with other second data exceeds the second threshold, and finally the remaining non-overlapping part will be generated as the third data storage.

[0073] The data list may include the storage locations of the first data, the second data, and the third data on the hard disk, the corresponding hash values, and a guidance method on how to combine the third data into the first data.

[0074] In some embodiments, see Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, and generating a third Hash value of the third data, storing the third data in the SQL database and updating the third Hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0075] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0076] S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304;

[0077] S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301;

[0078] S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

[0079] The present invention selects the available fourth data according to the degree of overlap from large to small and uses the remaining as the third data, which can increase the capacity of the fourth data with the largest capacity as much as possible, so that when the first data is retrieved and synthesized, the larger largest fourth data is used to integrate into the first data, which can improve the processing speed of synthesizing the first data and retrieving the first data. In other words, the first data is split according to the fourth data with the current overlap capacity from large to small, and finally only the third data is left, which helps to improve the retrieval speed.

[0080] It should be noted that the third data not only contains the non-overlapping parts of the first data, but also contains an operating guide on how to use the existing fourth data to re-synthesize the first data. The above operating guide is the data sequence number of the overlapping fourth data and the position of the fourth data.

[0081]

[0082] Among them, it should be noted that, when multiple fourth data and third data together constitute the first data, the third data should include the data sequence numbers of the multiple fourth data and the positions of the multiple fourth data, so as to facilitate the synthesis of the initial first data through the content of the third data and the data sequence numbers of the multiple fourth data in the third data and the positions of the multiple fourth data, thereby reducing the storage capacity of the SQL database and reducing storage pressure.

[0083] Among them, the second threshold is [0.1%, 99.9%], preferably 10%, that is, only the fourth data whose capacity exceeds 10% of the capacity of the first data can be compared to see whether it overlaps with an entire segment of the first data.

[0084] Among them, the above-mentioned entire paragraph overlaps and overlaps an entire paragraph have the same meaning.

[0085] The content of the first data, the second data, and the third data may be code, photos, videos, audio, text, etc., or may be a zip package or other data packages.

[0086] Assume that the first data can be a zip package, and the second data can be a file that is the same as the file in the zip package.

[0087] Alternatively, the first data is a text, for example, a 30-chapter text, and the second data may be the text of one of the chapters.

[0088] In some embodiments, see Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, and generating a third Hash value of the third data, storing the third data in the SQL database and updating the third Hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0089] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0090] S302, converting the fourth data into a format processable by the convolutional neural network;

[0091] S303, constructing and training a convolutional neural network model, wherein the convolutional neural network model is capable of receiving fourth data in a format that can be processed by the convolutional neural network model, and outputting a predicted combination scheme of at least one fourth data, wherein each fourth data in the combination scheme has no intersection with a position that overlaps with the first data;

[0092] S304, using the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data;

[0093] S305, calculating the first data of each combination scheme after deleting the content overlapping with the at least one fourth data, and the remaining part is the third data, and selecting the combination scheme corresponding to the smallest content capacity of the third data as the optimal solution;

[0094] S306. Output the optimal solution and its corresponding third data, wherein the third data includes the data sequence number of the overlapping second data and the position of the second data, and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0095] The present invention utilizes various permutations and combinations of convolutional neural networks (CNNs) and selects, by enumeration, an optimal solution with the smallest capacity of the remaining third data after deleting multiple fourth data from the first data; thereby further reducing the capacity of the final remaining third data to reduce the storage pressure of the SQL database.

[0096] Furthermore, since the second data selected during the training of the convolutional neural network is also selected through conditional screening, it will not occupy a large amount of processing power and calculation time.

[0097] Among them, the second data that overlaps with the first data for a whole segment and has a capacity greater than the second threshold of the content of the first data can be understood as follows: assuming that the content of the first data is 1234567 and the content of the second data is 24, then this second data overlaps with the first data for a whole segment, and this second data cannot be counted as the second data that overlaps with the first data. Assuming that the first data is 123231 and the second data is 23, then this second data can overlap with two whole segments of 23 of the first data, thereby further reducing the capacity of the content of the generated third data. Assuming that only the second data 23 overlaps with the above-mentioned first data 123231, even if the above-mentioned second data 23 overlaps with the above-mentioned first data 123231 at two locations, or the number of overlapping whole segments is two, then it is also counted as a second data or fourth data that overlaps with the first data, which is convenient for the subsequent feedback cycle of the convolutional neural network.

[0098] Among them, converting the fourth data into a format that can be processed by the convolutional neural network is a preprocessing of the fourth data, so that its data form can be recognized by the convolutional neural network model.

[0099] The fourth data and the second data are essentially the same, and when describing an example of their relative relationship, the fourth data and the second data can be interpreted interchangeably.

[0100] The second data that overlaps with the first data should completely overlap with a whole segment of the first data. For example, if the first data is 1234 and the second data is 12345 or 345, the second data does not overlap with the first data. In other words, the first data should completely cover the second data to be considered as one of the overlap conditions.

[0101] The specific forms of the above-mentioned convolutional neural network and its model may be existing technologies and will not be described in detail here.

[0102] In some embodiments, see Figure 1 , and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list, including:

[0103] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0104] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, retaining only the second data of the same file type as the first data and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0105] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0106] Before storing the third data, the present invention verifies whether the fourth data constituting the third data is too much. If it is too much, it will affect the data splitting speed of the first data. Then, in order to make the amount of the fourth data within a suitable range, when it needs to be adjusted, that is, when the amount of the fourth data exceeds the third threshold, the second threshold is increased first, and the cycle is started again from S301, and the number of such cycles is recorded. If the number of cycles exceeds the fourth threshold, it means that the proportion of the capacity of the second data that is simply increased to the first data is too large, and the access threshold of the second data can be raised, that is, it is difficult to synthesize different types of second data and first data. For example, the data type of the second data is a photo, and the data type of the first data is a video; this photo is indeed a whole part of the content of this video, but this different type of data is bound to bring too many second data in splitting and combination, resulting in too slow a retrieval speed and poor user experience. Therefore, if the number of cycles exceeds the fourth threshold and still cannot limit the fourth data to a reasonable third threshold, it should be considered to limit the second data to the same type as the first data and continue to cycle.

[0107] The third threshold may be [1, positive infinity), preferably 100. In other words, we split the first data into: only the second data within 100 can be combined into one first data, so as to avoid excessive data splitting and increase the data synthesis time, improve the data synthesis speed, and improve the user experience.

[0108] Among them, the fourth threshold can be [1, positive infinity), preferably 3. In other words, it is acceptable when it is cycled 3 times, but it also depends on the matching degree of the fourth threshold and the second threshold. The second threshold that is cycled for the fourth threshold should not make the overlapping part of the second data account for more than 100% of the first data. This obviously cannot screen out the error of any second data. We can configure the maximum number of cycles to 3 times through the above-mentioned cycle algorithm. When there are too many fourth data combined into the first data, the threshold of the fourth data is appropriately increased, the number of fourth data is reduced, and the data combination speed is increased. For example, after a maximum of 3 cycles, the threshold of the overlap ratio of the fourth data is increased by 33.1%, which will inevitably reduce the number of fourth data.

[0109] As a variant embodiment of the present invention, see Figure 1 , and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list, including:

[0110] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0111] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, deleting the fourth data with the fifth threshold percentage from the fourth data and the fourth data with the longest storage time away from the fourth data with the largest file capacity from all the fourth data, and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0112] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0113] Before storing the third data, the present invention verifies whether the fourth data constituting the third data is excessive. If it is excessive, it will affect the data splitting speed of the first data. Then, in order to make the amount of the fourth data within a suitable range, when it needs to be adjusted, that is, when the amount of the fourth data exceeds the third threshold, the second threshold is increased first, and the cycle is started again from S301 to record the number of such cycles. If the number of cycles exceeds the fourth threshold, it means that the proportion of the capacity of the second data simply increasing the first data is too large, and the entry threshold of the second data can be raised. That is, assuming that the fifth threshold is 10%, then among all the fourth data, a fourth data file with the largest file capacity is used as the basis (if two or more files with the same capacity appear, a fourth data is randomly designated), and the fourth data with the longest storage time from the fourth data with the largest capacity is 10% of all the fourth data. Then, other fourth data can be deleted from the fourth data based on the maximum single file storage capacity through the time range, thereby reducing the candidate fourth data that can be spliced ​​into the first data, so as to reduce the number of fourth data split from the first data, thereby ensuring the data synthesis speed and reducing the storage pressure of SQL data as much as possible. Among them, deleting within the same storage time range can make the storage time of the alternative fourth data similar, and the storage location in the hard disk is also similar, so that the coordination of these files being retrieved is better and the data retrieval speed is faster. In other words, when there are too many fourth data combined, the present invention selectively deletes some data with a longer storage time, thereby improving the data retrieval speed and user experience. It can be regarded as a feedback loop of a convolutional neural network.

[0114] See also Figure 1 The present invention provides an intelligent SQL database processing system, comprising:

[0115] The data storage module, when storing the first data:

[0116] Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database by using a hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data; if so, it is not necessary to store the first data in the SQL database and the first hash value of the first data is updated to the data list, that is, the first data is deemed to have been stored in the SQL database;

[0117] If not, determine whether the capacity of the first data exceeds the first threshold; if not, store the first data in the SQL database and update the first hash value of the first data to the data list; if exceeded, determine whether the second data has content that overlaps with the first data and exceeds the second threshold of the content of the first data; if not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0118] If so, convert the non-overlapping part of the first data and the second data into the third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list;

[0119] The data retrieval module, when it retrieves the first data:

[0120] It is determined according to the data list whether there is a second hash value of the second data which is the same as the first hash value of the first data in the SQL database; if so, the second data is converted into the first data and output; if not, the third data and the second data corresponding to the first data stored in the data list are synthesized into the first data and output.

[0121] The present invention first finds the same data through the hash value of the data, and then stores the second data with the same hash value as the first data, so that the first data does not need to be directly stored, thereby reducing the storage capacity pressure of the SQL database; if it is not found, the first data with too small storage capacity is not split, but directly stored to improve the retrieval speed; and the overlapping part of the first data with a larger storage capacity and the second data is not stored, thereby reducing the storage capacity pressure of the SQL database, and the non-overlapping distribution of the first data and the second data is stored separately as the third data in the SQL database. The present invention can achieve a smaller capacity storage and ensure a relatively fast retrieval speed through the above method.

[0122]

[0123] It should be noted that the content and hash value of the above data list are just examples. The actual hash should be obtained by SHA256 algorithm or SHA512 algorithm to ensure that the probability of coincidence is low and avoid data storage errors caused by different data having the same hash value. In addition, the above content can be video content, photo content, text content, audio content, and code content.

[0124] Among them, the first threshold value can be [1b, positive infinity), preferably 10mb. That is to say, when the second data that is the same as the first data has never been stored in the SQL database, and the capacity of the first data does not exceed 10mb, the first data is directly stored in the SQL database without being split, so as to ensure the retrieval speed. Because of the current retrieval characteristics of the hard disk, when the total capacity of a single retrieval remains unchanged, the more files there are and the smaller the storage capacity of a single file, the slower the retrieval speed. Then, small splits and excessive numbers of files are bound to affect the retrieval speed. Therefore, when the capacity of the first data is too small, it is not recommended to split it. After all, such small-capacity files will not take up too much storage space.

[0125] Among them, the second threshold value can be [0.1%, 99.9%], preferably 10%. That is to say, as long as the capacity of the entire content of the second data and the content of a whole block of the first data overlaps and occupies more than 10% of the total capacity of the first data, this whole block of first data will not be stored, but only recorded in the data list. When it is needed, the second data as the overlapping part and the third data as the non-overlapping part will be integrated into the first data and output. It should be explained that this whole block of overlapping data can no longer be calculated to overlap with other second data. In other words, when the overlapping part of a whole block of first data and one of the second data is determined, it will be deleted, and then it will be checked whether the overlap with other second data exceeds the second threshold, and finally the remaining non-overlapping part will be generated as the third data storage.

[0126] The data list may include the storage locations of the first data, the second data, and the third data on the hard disk, the corresponding hash values, and a guidance method on how to combine the third data into the first data.

[0127] In some embodiments, see Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, and generating a third Hash value of the third data, storing the third data in the SQL database and updating the third Hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0128] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0129] S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304;

[0130] S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301;

[0131] S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

[0132] The present invention selects the available fourth data according to the degree of overlap from large to small and uses the remaining as the third data, which can increase the capacity of the fourth data with the largest capacity as much as possible, so that when the first data is retrieved and synthesized, the larger largest fourth data is used to integrate into the first data, which can improve the processing speed of synthesizing the first data and retrieving the first data. In other words, the first data is split according to the fourth data with the current overlap capacity from large to small, and finally only the third data is left, which helps to improve the retrieval speed.

[0133] It should be noted that the third data not only contains the non-overlapping parts of the first data, but also contains an operating guide on how to use the existing fourth data to re-synthesize the first data. The above operating guide is the data sequence number of the overlapping fourth data and the position of the fourth data.

[0134]

[0135] Among them, it should be noted that, when multiple fourth data and third data together constitute the first data, the third data should include the data sequence numbers of the multiple fourth data and the positions of the multiple fourth data, so as to facilitate the synthesis of the initial first data through the content of the third data and the data sequence numbers of the multiple fourth data in the third data and the positions of the multiple fourth data, thereby reducing the storage capacity of the SQL database and reducing storage pressure.

[0136] Among them, the second threshold is [0.1%, 99.9%], preferably 10%, that is, only the fourth data whose capacity exceeds 10% of the capacity of the first data can be compared to see whether it overlaps with an entire segment of the first data.

[0137] Among them, the above-mentioned entire paragraph overlaps and overlaps an entire paragraph have the same meaning.

[0138] The content of the first data, the second data, and the third data may be code, photos, videos, audio, text, etc., or may be a zip package or other data packages.

[0139] Assume that the first data can be a zip package, and the second data can be a file that is the same as the file in the zip package.

[0140] Alternatively, the first data is a text, for example, a 30-chapter text, and the second data may be the text of one of the chapters.

[0141] In some embodiments, see Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, and generating a third Hash value of the third data, storing the third data in the SQL database and updating the third Hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0142] S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data;

[0143] S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304;

[0144] S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301;

[0145] S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

[0146] The present invention selects the available fourth data according to the degree of overlap from large to small and uses the remaining as the third data, which can increase the capacity of the fourth data with the largest capacity as much as possible, so that when the first data is retrieved and synthesized, the larger largest fourth data is used to integrate into the first data, which can improve the processing speed of synthesizing the first data and retrieving the first data. In other words, the first data is split according to the fourth data with the current overlap capacity from large to small, and finally only the third data is left, which helps to improve the retrieval speed.

[0147] It should be noted that the third data not only contains the non-overlapping parts of the first data, but also contains an operating guide on how to use the existing fourth data to re-synthesize the first data. The above operating guide is the data sequence number of the overlapping fourth data and the position of the fourth data.

[0148]

[0149]

[0150] Among them, it should be noted that, when multiple fourth data and third data together constitute the first data, the third data should include the data sequence numbers of the multiple fourth data and the positions of the multiple fourth data, so as to facilitate the synthesis of the initial first data through the content of the third data and the data sequence numbers of the multiple fourth data in the third data and the positions of the multiple fourth data, thereby reducing the storage capacity of the SQL database and reducing storage pressure.

[0151] Among them, the second threshold is [0.1%, 99.9%], preferably 10%, that is, only the fourth data whose capacity exceeds 10% of the capacity of the first data can be compared to see whether it overlaps with an entire segment of the first data.

[0152] Among them, the above-mentioned entire paragraph overlaps and overlaps an entire paragraph have the same meaning.

[0153] The content of the first data, the second data, and the third data may be code, photos, videos, audio, text, etc., or may be a zip package or other data packages.

[0154] Assume that the first data can be a zip package, and the second data can be a file that is the same as the file in the zip package.

[0155] Alternatively, the first data is a text, for example, a 30-chapter text, and the second data may be the text of one of the chapters.

[0156] In some embodiments, see Figure 1 , and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list, including:

[0157] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0158] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, retaining only the second data of the same file type as the first data and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0159] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0160] Before storing the third data, the present invention verifies whether the fourth data constituting the third data is too much. If it is too much, it will affect the data splitting speed of the first data. Then, in order to make the amount of the fourth data within a suitable range, when it needs to be adjusted, that is, when the amount of the fourth data exceeds the third threshold, the second threshold is increased first, and the cycle is started again from S301, and the number of such cycles is recorded. If the number of cycles exceeds the fourth threshold, it means that the proportion of the capacity of the second data that is simply increased to the first data is too large, and the access threshold of the second data can be raised, that is, it is difficult to synthesize different types of second data and first data. For example, the data type of the second data is a photo, and the data type of the first data is a video; this photo is indeed a whole part of the content of this video, but this different type of data is bound to bring too many second data in splitting and combination, resulting in too slow a retrieval speed and poor user experience. Therefore, if the number of cycles exceeds the fourth threshold and still cannot limit the fourth data to a reasonable third threshold, it should be considered to limit the second data to the same type as the first data and continue to cycle.

[0161] The third threshold may be [1, positive infinity), preferably 100. In other words, we split the first data into: only the second data within 100 can be combined into one first data, so as to avoid excessive data splitting and increase the data synthesis time, improve the data synthesis speed, and improve the user experience.

[0162] Among them, the fourth threshold can be [1, positive infinity), preferably 3. In other words, it is acceptable when it is cycled 3 times, but it also depends on the matching degree of the fourth threshold and the second threshold. The second threshold that is cycled for the fourth threshold should not make the overlapping part of the second data account for more than 100% of the first data. This obviously cannot screen out the error of any second data. We can configure the maximum number of cycles to 3 times through the above-mentioned cycle algorithm. When there are too many fourth data combined into the first data, the threshold of the fourth data is appropriately increased, the number of fourth data is reduced, and the data combination speed is increased. For example, after a maximum of 3 cycles, the threshold of the overlap ratio of the fourth data is increased by 33.1%, which will inevitably reduce the number of fourth data.

[0163] As a variant embodiment of the present invention, see Figure 1 , and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list, including:

[0164] S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402;

[0165] S402, determining whether the number of cycles exceeds a fourth threshold value, if so, deleting the fourth data with the fifth threshold percentage from the fourth data and the fourth data with the longest storage time away from the fourth data with the largest file capacity from all the fourth data, and jumping to S301, and clearing the number of cycles; if not, jumping to S403;

[0166] S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

[0167] Before storing the third data, the present invention verifies whether the fourth data constituting the third data is excessive. If it is excessive, it will affect the data splitting speed of the first data. Then, in order to make the amount of the fourth data within a suitable range, when it needs to be adjusted, that is, when the amount of the fourth data exceeds the third threshold, the second threshold is increased first, and the cycle is started again from S301 to record the number of such cycles. If the number of cycles exceeds the fourth threshold, it means that the proportion of the capacity of the second data simply increasing the first data is too large, and the entry threshold of the second data can be raised. That is, assuming that the fifth threshold is 10%, then among all the fourth data, a fourth data file with the largest file capacity is used as the basis (if two or more files with the same capacity appear, a fourth data is randomly designated), and the fourth data with the longest storage time from the fourth data with the largest capacity is 10% of all the fourth data. Then, other fourth data can be deleted from the fourth data based on the maximum single file storage capacity through the time range, thereby reducing the candidate fourth data that can be spliced ​​into the first data, so as to reduce the number of fourth data split from the first data, thereby ensuring the data synthesis speed and reducing the storage pressure of SQL data as much as possible. Among them, deleting within the same storage time range can make the storage time of the alternative fourth data similar, and the storage location in the hard disk is also similar, so that the coordination of these files being retrieved is better and the data retrieval speed is faster. In other words, when there are too many fourth data combined, the present invention selectively deletes some data with a longer storage time, thereby improving the data retrieval speed and user experience. It can be regarded as a feedback loop of a convolutional neural network.

[0168] The embodiments described above are merely descriptions of preferred implementation modes of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent SQL database processing method, characterized in that: include When storing the first data: Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database by using a hash algorithm; Determine whether there is a second hash value of the second data that is the same as the first hash value of the first data in the SQL database; if so, it is not necessary to store the first data in the SQL database and the first hash value of the first data is updated to the data list, that is, the first data is deemed to have been stored in the SQL database; If not, determining whether the capacity of the first data exceeds a first threshold, and if not, storing the first data in an SQL database and updating the first hash value of the first data to the data list; If it exceeds, determine whether there is content in the second data that overlaps with the first data and exceeds a second threshold of the content of the first data; if not, store the first data in the SQL database and update the first hash value of the first data to the data list; If so, convert the non-overlapping part of the first data and the second data into the third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list; When retrieving the first data: Determine whether there is a second hash value of second data in the SQL database that is the same as the first hash value of the first data according to the data list, and if so, convert the second data into the first data and output it; If not, the third data and the second data corresponding to the first data stored in the data list are synthesized into the first data and outputted.

2. The intelligent SQL database processing method according to claim 1, characterized in that: Converting the non-overlapping part of the first data and the second data into the third data and generating a third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including: S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data; S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304; S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301; S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

3. The intelligent SQL database processing method according to claim 1, characterized in that: Converting the non-overlapping part of the first data and the second data into the third data and generating a third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including: S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data; S302, converting the fourth data into a format processable by the convolutional neural network; S303, constructing and training a convolutional neural network model, wherein the convolutional neural network model is capable of receiving fourth data in a format that can be processed by the convolutional neural network model, and outputting a predicted combination scheme of at least one fourth data, wherein each fourth data in the combination scheme has no intersection with a position that overlaps with the first data; S304, using the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data; S305, calculating the first data of each combination scheme after deleting the content overlapping with the at least one fourth data, and the remaining part is the third data, and selecting the combination scheme corresponding to the smallest content capacity of the third data as the optimal solution; S306. Output the optimal solution and its corresponding third data, wherein the third data includes the data sequence number of the overlapping second data and the position of the second data, and store the memory of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

4. An intelligent SQL database processing method according to any one of claims 2 or 3, characterized in that: The memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the location of the second data are updated to the data list, including: S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402; S402, determining whether the number of cycles exceeds a fourth threshold value, if so, retaining only the second data of the same file type as the first data and jumping to S301, and clearing the number of cycles; if not, jumping to S403; S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

5. An intelligent SQL database processing method according to any one of claims 2 or 3, characterized in that: The memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the location of the second data are updated to the data list, including: S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402; S402, determining whether the number of cycles exceeds a fourth threshold value, if so, deleting the fourth data with the fifth threshold percentage from the fourth data and the fourth data with the longest storage time away from the fourth data with the largest file capacity from all the fourth data, and jumping to S301, and clearing the number of cycles; if not, jumping to S403; S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

6. An intelligent SQL database processing system, characterized in that: include The data storage module, when storing the first data: Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database by using a hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data; if so, it is not necessary to store the first data in the SQL database and the first hash value of the first data is updated to the data list, that is, the first data is deemed to have been stored in the SQL database; If not, determining whether the capacity of the first data exceeds a first threshold, and if not, storing the first data in an SQL database and updating the first hash value of the first data to the data list; If it exceeds, determine whether there is content in the second data that overlaps with the first data and exceeds a second threshold of the content of the first data; if not, store the first data in the SQL database and update the first hash value of the first data to the data list; If so, convert the non-overlapping part of the first data and the second data into the third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list; The data retrieval module, when it retrieves the first data: Determine whether there is a second hash value of second data in the SQL database that is the same as the first hash value of the first data according to the data list, and if so, convert the second data into the first data and output it; If not, the third data and the second data corresponding to the first data stored in the data list are synthesized into the first data and outputted.

7. An intelligent SQL database processing system according to claim 6, characterized in that: Converting the non-overlapping part of the first data and the second data into the third data and generating a third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including: S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data; S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304; S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301; S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

8. The intelligent SQL database processing system according to claim 6, characterized in that: Converting the non-overlapping part of the first data and the second data into the third data and generating a third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including: S301, obtaining second data that overlaps with the first data by an entire segment and has a capacity greater than a second threshold of the content of the first data, and converting the second data into fourth data; S302, determine whether there is fourth data, if yes, jump to S303, if no, jump to S304; S303, deleting the overlapping content of the fourth data with the largest overlapping capacity from the first data, and jumping to S301; S304, converting the remaining part of the first data into the content of the third data, wherein the third data includes the data sequence number of the overlapping fourth data and the position of the fourth data, and storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data and the position of the fourth data to the data list.

9. An intelligent SQL database processing system according to any one of claims 7 or 8, characterized in that: The memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the location of the second data are updated to the data list, including: S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402; S402, determining whether the number of cycles exceeds a fourth threshold value, if so, retaining only the second data of the same file type as the first data and jumping to S301, and clearing the number of cycles; if not, jumping to S403; S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

10. An intelligent SQL database processing system according to any one of claims 7 or 8, characterized in that: The memory of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the location of the second data are updated to the data list, including: S401, determining whether the number of fourth data overlapping with the first data exceeds the third threshold, if so, increasing the second threshold, and jumping back to S301, and recording the number of cycles, if not, jumping to S402; S402, determining whether the number of cycles exceeds a fourth threshold value, if so, deleting the fourth data with the fifth threshold percentage from the fourth data and the fourth data with the longest storage time away from the fourth data with the largest file capacity from all the fourth data, and jumping to S301, and clearing the number of cycles; if not, jumping to S403; S403, storing the memory of the third data in the SQL database and updating it to the data list, and updating the content of the third data corresponding to the first data, the data sequence number of the overlapping second data and the position of the second data to the data list.

Images