An intelligent SQL database processing method and system

Optimizing the storage and call methods of SQL databases through hashing algorithms and convolutional neural networks, the problem of waste of duplicate data storage is solved, and smaller capacity storage and faster data call is achieved.

CN119474138BActive Publication Date: 2025-08-01BEIJING SHUJU ZHILIAN TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411504344.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-25
Publication Date
2025-08-01
Estimated Expiration
2044-10-25

AI Technical Summary

Technical Problem

When existing SQL databases store and retrieve data, duplicate data cannot be recognized, resulting in wasted storage space and slow recall speed.

Method used

The hash value of the data is generated through the hash algorithm to determine whether the same data exists in the database. If it exists, it will not be stored. Otherwise, it will be split and stored according to the data capacity and overlap. The data combination is optimized using a convolutional neural network to reduce storage space and increase the speed of retrieval.

Benefits of technology

It reduces the storage space pressure of SQL databases and ensures relatively fast data retrieval speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119474138B_ABST
    Figure CN119474138B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of big data processing, and discloses an intelligent SQL database processing method, which includes: when storing the first data, determining whether there is content in the second data that coincides with the first data and exceeds a second threshold of the content of the first data; when retrieving the first data, synthesizing the third data corresponding to the first data stored in the data list and the second data into the first data and outputting it. In the present invention, the overlapping part between the first data with a large storage capacity and the second data is not stored, thereby reducing the storage capacity pressure of the SQL database, and storing the non-overlapping distribution of the first data and the second data as the third data separately in the SQL database. Through the above method, the present invention can achieve a smaller storage capacity and ensure a relatively fast retrieval speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and particularly to a processing method for optimizing an SQL database. Background Art

[0002] SQL (Structured Query Language) is a database language with various functions such as data manipulation and data definition. This language has the characteristic of interactivity, which can provide great convenience for users. The database management system should make full use of the SQL language to improve the working quality and efficiency of the computer application system. The SQL language can not only be independently applied to the terminal, but also be used as a sub-language to provide effective assistance for other programming. In this program application, SQL can optimize the program function together with other programming languages, and then provide users with more and more comprehensive information.

[0003] Currently, the data storage and retrieval of the SQL database are only simple storage and retrieval, unable to identify duplicate data or partial data, thus increasing its storage pressure and occupying a large amount of precious storage space.

[0004] Therefore, there is a need for an intelligent SQL database processing method that can optimize the storage and retrieval methods to reduce the data storage space and ensure a relatively fast retrieval speed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an intelligent SQL database processing method that can optimize the storage and retrieval methods to reduce the data storage space and ensure a relatively fast retrieval speed.

[0006] An intelligent SQL database processing method of the present invention includes

[0007] When storing the first data:

[0008] Generate a first hash value of the first data and a second hash value of the second data already stored in the SQL database through the hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If so, it is not necessary to store the first data in the SQL database and update the first hash value of the first data to the data list, that is, it is regarded that the first data has been stored in the SQL database;

[0009] If not, determine whether the capacity of the first data exceeds a first threshold. If not, store the first data in the SQL database and update the first hash value of the first data to the data list. If it exceeds, determine whether there is content in the second data that coincides with the first data and exceeds a second threshold of the content of the first data. If not, store the first data in the SQL database and update the first hash value of the first data to the data list.

[0010] If so, convert the non - overlapping part of the first data and the second data into third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list.

[0011] When retrieving the first data:

[0012] According to the data list, determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If so, convert the second data into the first data and output it. If not, synthesize the third data and the second data corresponding to the first data stored in the data list into the first data and output it.

[0013] In an intelligent SQL database processing system of the present invention, converting the non - overlapping part of the first data and the second data into third data, generating a third hash value of the third data, storing the third data in the SQL database, updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list includes:

[0014] S301. Obtain second data that coincides with the first data in an entire segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into fourth data;

[0015] S302. Determine whether there is fourth data. If so, jump to S303; if not, jump to S304;

[0016] S303. Delete the content of the first data that coincides with the fourth data with the largest overlapping capacity, and jump to S301;

[0017] S304. Convert the remaining part of the first data into the content of the third data. The third data includes the data sequence number of the overlapping fourth data and the position where the fourth data is located. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data, and the position where the fourth data is located to the data list.

[0018] An intelligent SQL database processing system of the present invention, wherein the non-overlapping part of the first data and the second data is converted into the third data, and the third hash value of the third data is generated. The third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the overlapping part of the first data and the second data is updated to the data list, including:

[0019] S301. Obtain the second data whose content overlapping with the first data in a whole segment and whose capacity is greater than the second threshold of the first data, and convert it into the fourth data;

[0020] S302. Convert the fourth data into a format processable by a convolutional neural network;

[0021] S303. Construct and train a convolutional neural network model, which can receive the fourth data in the format it can process and output at least one combination scheme of the predicted fourth data, wherein each fourth data in the combination scheme has no intersection with the overlapping position of the first data;

[0022] S304. Use the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data;

[0023] S305. Calculate the remaining part as the third data after deleting the content overlapping with the at least one fourth data from the first data for each combination scheme, and select the combination scheme corresponding to the smallest capacity of the content of the third data as the optimal solution;

[0024] S306. Output the optimal solution scheme and its corresponding third data. The third data includes the data serial number of the overlapping second data and the position where the second data is located, and store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the overlapping second data and the position where the second data is located to the data list.

[0025] An intelligent SQL database processing system of the present invention, wherein the content of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data serial number of the overlapping second data and the position where the second data is located are updated to the data list, including:

[0026] S401. Judge whether the number of the fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold and jump back to S301 again, and record the number of loop times once. If it does not exceed, jump to S402;

[0027] S402. Determine whether the above loop count exceeds the fourth threshold. If so, only retain the second data with the same file type as the first data, jump to S301, and clear the loop count; if not, jump to S403;

[0028] S403. Store the content of the third data in the SQL database and update it to the data list. Update the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located to the data list.

[0029] An intelligent SQL database processing system of the present invention, wherein the content of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located are updated to the data list, including:

[0030] S401. Determine whether the number of fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold, and re-jump to S301, and record the loop count once. If it does not exceed, jump to S402;

[0031] S402. Determine whether the above loop count exceeds the fourth threshold. If so, delete the fourth data with the farthest storage time from the file with the largest capacity among the fifth threshold percentages of all fourth data in all fourth data and jump to S301, and clear the loop count; if not, jump to S403;

[0032] S403. Store the content of the third data in the SQL database and update it to the data list. Update the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located to the data list.

[0033] An intelligent SQL database processing system of the present invention includes

[0034] A data storage module, when it stores the first data:

[0035] Generate the first hash value of the first data and the second hash value of the second data already stored in the SQL database through the hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If so, there is no need to store the first data in the SQL database and update the first hash value of the first data to the data list, that is, it is regarded that the first data has been stored in the SQL database;

[0036] If not, determine whether the capacity of the first data exceeds a first threshold. If not, store the first data in the SQL database and update the first hash value of the first data to the data list. If it exceeds, determine whether there is content in the second data that coincides with the first data and exceeds a second threshold of the content of the first data. If not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0037] If there is, convert the non - overlapping part of the first data and the second data into third data, generate a third hash value of the third data, store the third data in the SQL database and update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list;

[0038] When the data retrieval module retrieves the first data:

[0039] According to the data list, determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If there is, convert the second data into the first data and output it. If not, synthesize the third data and the second data corresponding to the first data stored in the data list into the first data and output it.

[0040] In an intelligent SQL database processing system of the present invention, converting the non - overlapping part of the first data and the second data into third data, generating a third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, includes:

[0041] S301. Obtain a second data whose content coincides with the first data in an entire segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into fourth data;

[0042] S302. Determine whether there is fourth data. If so, jump to S303. If not, jump to S304;

[0043] S303. Delete the content of the first data that coincides with the fourth data with the largest overlapping capacity, and jump to S301;

[0044] S304. Convert the remaining part of the first data into the content of the third data. The third data includes the data sequence number of the overlapping fourth data and the position where the fourth data is located, store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping fourth data, and the position where the fourth data is located to the data list.

[0045] An intelligent SQL database processing system of the present invention, wherein the non-overlapping part of the first data and the second data is converted into the third data, and the third hash value of the third data is generated. The third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the overlapping part of the first data and the second data is updated to the data list, including:

[0046] S301. Obtain the second data that overlaps with the entire segment of the first data and has a capacity greater than the second threshold of the content of the first data, and convert it into the fourth data;

[0047] S302. Determine whether there is the fourth data. If so, jump to S303; if not, jump to S304;

[0048] S303. Delete the content that overlaps with the fourth data with the largest overlapping capacity in the first data, and jump to S301;

[0049] S304. Convert the remaining part in the first data into the content of the third data. The third data includes the data serial number of the overlapping fourth data and the position where the fourth data is located. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the overlapping fourth data, and the position where the fourth data is located to the data list.

[0050] An intelligent SQL database processing system of the present invention, wherein the content of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located are updated to the data list, including:

[0051] S401. Determine whether the number of the fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold, and re-jump to S301, and record the number of loop times once. If it does not exceed, jump to S402;

[0052] S402. Determine whether the above loop times exceed the fourth threshold. If so, only retain the second data with the same file type as the first data and jump to S301, and clear the loop times; if not, jump to S403;

[0053] S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located to the data list.

[0054] An intelligent SQL database processing system of the present invention, wherein the content of the third data is stored in the SQL database and updated to the data list, and the content of the third data corresponding to the first data, the data sequence numbers of the overlapping second data, and the positions where the second data is located are updated to the data list, including:

[0055] S401. Determine whether the number of fourth data overlapping with the first data exceeds a third threshold. If it exceeds, increase the second threshold, and re-jump to S301, and record the number of loop times once. If it does not exceed, jump to S402;

[0056] S402. Determine whether the above-mentioned number of loop times exceeds a fourth threshold. If so, delete the fourth data with the farthest storage time from the file capacity among all the fourth data, which accounts for a percentage of the fifth threshold, and jump to S301, and clear the number of loop times; if not, jump to S403;

[0057] S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence numbers of the overlapping second data, and the positions where the second data is located to the data list.

[0058] The difference between an intelligent SQL database processing method of the present invention and the prior art is that the intelligent SQL database processing method of the present invention first finds the same data through the hash value of the data, and thus stores the second data with the same hash value as the first data, so there is no need to directly store the first data, thereby reducing the storage capacity pressure of the SQL database; if not found, the first data with too small storage capacity is not split but directly stored to improve the retrieval speed; and the overlapping part of the first data with a large storage capacity and the second data is not stored either, thereby reducing the storage capacity pressure of the SQL database, and the non-overlapping distribution of the first data and the second data is stored in the SQL database separately as the third data. Through the above method, the present invention can achieve a smaller storage capacity and ensure a relatively fast retrieval speed.

[0059] The following further describes an intelligent SQL database processing method of the present invention with reference to the accompanying drawings. Description of the Drawings

[0060] Figure 1 is a schematic flowchart of the storage stage of an intelligent SQL database processing method. Detailed Embodiment

[0061] As Figure 1 shown, an intelligent SQL database processing method of the present invention includes

[0062] When storing the first data:

[0063] Generate the first hash value of the first data and the second hash value of the second data already stored in the SQL database through a hashing algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If there is, there is no need to store the first data in the SQL database and update the first hash value of the first data to the data list, that is, it is considered that the first data has been stored in the SQL database;

[0064] If not, determine whether the capacity of the first data exceeds a first threshold. If it does not exceed, store the first data in the SQL database and update the first hash value of the first data to the data list; if it exceeds, determine whether there is content in the second data that coincides with the first data and exceeds a second threshold of the content of the first data. If not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0065] If there is, convert the non-coincident part of the first data and the second data into third data, generate the third hash value of the third data, store the third data in the SQL database and update the third hash value of the third data to the data list, and then update the coincident part of the first data and the second data to the data list;

[0066] When retrieving the first data:

[0067] Judge whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data according to the data list. If there is, convert the second data into the first data and output it; if not, synthesize the third data and the second data corresponding to the first data stored in the data list into the first data and output it.

[0068] The present invention first finds the same data through the hash value of the data, so as to store the second data with the same hash value as the first data, and there is no need to directly store the first data, thereby reducing the storage capacity pressure of the SQL database; if not found, the first data with too small storage capacity is not split, but directly stored to improve the retrieval speed; and the overlapping part of the first data with a large storage capacity and the second data is not stored either, thereby reducing the storage capacity pressure of the SQL database, and storing the non-overlapping part of the first data and the second data as the third data separately in the SQL database. The present invention can achieve a smaller-capacity storage and ensure a relatively fast retrieval speed through the above method.

[0069]

[0070] It should be noted that the content and hash values of the above data list are only examples. The actual hash should be obtained by the SHA256 algorithm or the SHA512 algorithm to ensure a low coincidence probability and avoid data storage errors caused by different data having the same hash value. Moreover, the above content can be video content, photo content, text content, audio content, or code content.

[0071] Among them, the first threshold can be [1b, +∞), preferably 10mb. That is to say, when there is a second data that is the same as the first data that has never been stored in the SQL database, and the capacity of this first data does not exceed 10mb, directly store this first data in the SQL database without splitting, so as to ensure the retrieval speed. Because of the retrieval characteristics of the current hard disk, when the total retrieval capacity remains unchanged, the more files and the smaller the storage capacity of each file, the slower the retrieval speed. Then, too small a split and too many file numbers will inevitably affect the retrieval speed. Therefore, when the capacity of the first data is too small, splitting is not recommended, after all, such small-capacity files will not occupy too much storage space.

[0072] Among them, the second threshold can be [0.1%, 99.9%], preferably 10%. That is to say, as long as the capacity of the overlapping part of all the content of a second data and a whole block of content of the first data exceeds 10% of the total capacity of the first data, then this whole block of the first data is not stored, but only recorded in the data list. When retrieval is needed, the second data as the overlapping part and the third data as the non-overlapping part are integrated into the first data and output. It should be explained that this whole block of overlapping data cannot be calculated for overlap with other second data. In other words, after the overlapping part of a whole block of data of the first data and one of the second data is determined, it is deleted, and then it is checked whether it overlaps with other second data by more than the second threshold. Finally, the remaining non-overlapping part is generated as the third data for storage.

[0073] Among them, the data list can be provided with the storage locations of the first data, the second data, and the third data on the hard disk, the corresponding hash values, and the guiding method for how the third data is combined into the first data.

[0074] In some embodiments, refer to Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, and generating the third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0075] S301. Obtain a second data that overlaps with the first data in a whole segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into a fourth data;

[0076] S302. Determine whether there is a fourth piece of data. If so, jump to S303; if not, jump to S304.

[0077] S303. Delete the content that coincides with the fourth piece of data with the largest overlapping capacity in the first piece of data, and jump to S301.

[0078] S304. Convert the remaining part of the first piece of data into the content of the third piece of data. The third piece of data includes the data serial number of the overlapping fourth piece of data and the position where the fourth piece of data is located. Store the content of the third piece of data in the SQL database and update it to the data list. Update the content of the third piece of data corresponding to the first piece of data, the data serial number of the overlapping fourth piece of data, and the position where the fourth piece of data is located to the data list.

[0079] The present invention selects the fourth piece of data according to the degree of overlap from large to small among the available fourth pieces of data and uses the remaining ones as the third piece of data, which can increase the capacity of the fourth piece of data with the largest capacity as much as possible. Thus, when retrieving and synthesizing the first piece of data, using the larger maximum fourth piece of data to integrate into the first piece of data can improve the processing speed of synthesizing and retrieving the first piece of data. That is to say, splitting the first piece of data according to the fourth piece of data with the current overlapping capacity from large to small, and finally only the third piece of data remains, which helps to improve the retrieval speed.

[0080] It should be noted that the third piece of data not only contains the non-overlapping part of the first piece of data, but also contains an operation guide on how to re-synthesize the first piece of data using the existing fourth piece of data. The above operation guide is the data serial number of the overlapping fourth piece of data and the position where the fourth piece of data is located.

[0081]

[0082] It should be noted that when multiple fourth pieces of data and the third piece of data jointly form the first piece of data, the third piece of data should include the data serial numbers of multiple fourth pieces of data and the positions where multiple fourth pieces of data are located, which is convenient for synthesizing the initial first piece of data through the content of the third piece of data and the data serial numbers of multiple fourth pieces of data and the positions where multiple fourth pieces of data are located in the third piece of data, thereby reducing the storage capacity of the SQL database and the storage pressure.

[0083] The second threshold is [0.1%, 99.9%], preferably 10%. That is to say, only the fourth piece of data whose capacity exceeds 10% of the capacity of the first piece of data can be compared for whether it coincides with an entire segment of the first piece of data.

[0084] The above "entire segment coincidence" and "coincide with an entire segment" have the same meaning.

[0085] The content of the first data, the second data, and the third data can be code, photos, videos, audio, text, etc., or a zip package or other data packet.

[0086] Assume that if the first data is a zip package, the second data can be a file that is the same as the file inside the zip package.

[0087] Or, if the first data is a text, for example, a text with 30 chapters, then the second data can be the text of one of the chapters.

[0088] In some embodiments, referring to Figure 1 , convert the non-overlapping parts of the first data and the second data into the third data, and generate the third hash value of the third data. Store the third data in the SQL database and update the third hash value of the third data to the data list, and then update the overlapping parts of the first data and the second data to the data list, including:

[0089] S301. Obtain the second data that overlaps with the entire first data and has a capacity greater than the second threshold of the content of the first data, and convert it into the fourth data;

[0090] S302. Convert the fourth data into a format that can be processed by a convolutional neural network;

[0091] S303. Construct and train a convolutional neural network model, which can receive the fourth data in a format that it can process and output at least one combination scheme of predicted fourth data, where each fourth data in the combination scheme has no intersection with the overlapping position of the first data;

[0092] S304. Use the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data;

[0093] S305. Calculate the remaining part as the third data after deleting the overlapping content of the first data and the at least one fourth data for each combination scheme, and select the combination scheme corresponding to the smallest capacity of the content of the third data as the optimal solution;

[0094] S306. Output the optimal solution scheme and its corresponding third data. The third data includes the data serial number of the overlapping second data and the position where the second data is located, store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located to the data list.

[0095] The present invention uses various permutation and combination forms of a convolutional neural network (CNN). After deleting multiple fourth data by means of the enumeration method from the first data, an optimal solution is selected in which the capacity of the remaining third data is the smallest, thereby further reducing the capacity of the finally remaining third data to reduce the storage pressure on the SQL database.

[0096] Moreover, since the second data selected during the training of the above-mentioned convolutional neural network is also screened by conditions, it will not occupy a large amount of processing power or calculation time.

[0097] Among them, the second data that coincides with a whole segment of the first data and has a capacity greater than the second threshold of the content of the first data can be understood as follows: assuming that the content of the first data is 1234567 and the content of the second data is 24, then this second data does not coincide with a whole segment of the first data, and this second data cannot be regarded as the second data that coincides with the first data. Assume again that the first data is 123231 and the second data is 23. Then this second data can coincide with two whole segments of 23 in the first data, thereby further reducing the capacity of the content of the generated third data. Assume that only this second data 23 coincides with the above-mentioned first data 123231. Even if the position where the above-mentioned second data 23 coincides with the above-mentioned first data 123231 has two places, or in other words, the number of whole segments that coincide is two, then it is also regarded as one second data or fourth data that coincides with the first data, which is convenient for the subsequent feedback loop of the convolutional neural network.

[0098] Among them, converting the fourth data into a format that can be processed by the convolutional neural network is a preprocessing of the fourth data, facilitating the recognition of its data form by the convolutional neural network model.

[0099] Among them, the fourth data and the second data are essentially the same. When giving examples of their relative relationships, the fourth data and the second data can be mutually replaced for explanation.

[0100] Among them, the second data that coincides with the first data should completely coincide with a whole segment of the first data. For example, if the first data is 1234 and the second data is 12345 or 345, then the second data does not coincide with the first data. That is to say, the first data should completely cover the second data to be used as one of the coincidence conditions.

[0101] The specific forms of the above-mentioned convolutional neural network and its model can be the prior art and will not be elaborated here.

[0102] In some embodiments, refer to Figure 1 and store the content of the third data in the SQL database and update it to the data list. Update the content of the third data corresponding to the first data, the data serial number of the coincident second data, and the position where the second data is located to the data list, including:

[0103] S401. Determine whether the number of fourth data overlapping with the first data exceeds a third threshold. If it exceeds, increase the second threshold, re-jump to S301, and record the number of loop times once. If it does not exceed, jump to S402;

[0104] S402. Determine whether the number of loop times exceeds a fourth threshold. If so, only retain the second data with the same file type as the first data and jump to S301, and clear the number of loop times; if not, jump to S403;

[0105] S403. Store the content of the third data in the SQL database and update it to the data list. Update the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located to the data list.

[0106] Before storing the third data, the present invention first verifies whether there are too many fourth data that make up the third data. If there are too many, it will affect the data splitting speed of the first data. Then, in order to keep the number of the fourth data within a suitable range, when adjustment is needed, that is, when the number of the fourth data exceeds the third threshold, the second threshold is first increased, and the loop starts again from S301, and the number of times of such loop is recorded. If the number of loop times exceeds the fourth threshold, it means that the proportion of the capacity of the second data simply increasing in the first data is too large, and the access threshold of the second data can be raised. That is, it is more difficult to synthesize different types of second data and the first data. For example, the data type of the second data is a photo, and the data type of the first data is a video. This photo does overlap with a part of the content of this video. However, this different type of data will inevitably bring too many second data in terms of splitting and combining, resulting in too slow retrieval speed and poor user experience. Therefore, if the number of loop times exceeds the fourth threshold and still cannot limit the fourth data within a reasonable third threshold, it should be considered to limit the second data to the same type as the first data and continue the loop.

[0107] Among them, the third threshold can be [1, +∞), preferably 100. In other words, we split the combined first data into: only the second data within 100 can be combined into a first data, avoiding excessive data splitting and increasing the data synthesis time, improving the data synthesis speed, and enhancing the user experience.

[0108] Among them, the fourth threshold can be [1, positive infinity), preferably 3. In other words, when the loop is 3 times, it is acceptable, but it also depends on the matching degree between the fourth threshold and the second threshold. The second threshold for the fourth threshold loop times should not make the overlapping part of the second data exceed 100% of the first data, which is obviously an error situation where no second data can be selected. We can configure the maximum loop times limit of 3 times through the above loop algorithm. When there are too many fourth data combined into the first data, appropriately increase the threshold of the fourth data to reduce the number of fourth data, thereby improving the data combination speed. For example, at most after 3 loops, increase the threshold of the overlapping ratio of the fourth data by 33.1%, which will inevitably reduce the number of fourth data.

[0109] As a variant embodiment of the present invention, refer to Figure 1 , and store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions where the second data is located to the data list, including:

[0110] S401. Determine whether the number of fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold and re-jump to S301, and record the number of loop times once. If it does not exceed, jump to S402;

[0111] S402. Determine whether the above loop times exceed the fourth threshold. If so, delete the fourth data with the farthest storage time from the file capacity among the fifth threshold percentage of all fourth data and jump to S301, and clear the loop times; if not, jump to S403;

[0112] S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions where the second data is located to the data list.

[0113] Before storing the third data as described above, the present invention first verifies whether the fourth data that makes up the third data is excessive. If it is excessive, it will affect the data splitting speed of the first data. Then, in order to keep the quantity of the above-mentioned fourth data within a suitable range, when it needs to be adjusted, that is, when the quantity of the fourth data exceeds the third threshold, the second threshold is first increased, and the loop starts again from S301, and the number of times of such loop is recorded. If the number of loops exceeds the fourth threshold, it means that the proportion of the capacity of the second data in the first data has increased simply too much, and the access threshold of the second data can be increased. That is, assuming the fifth threshold is 10%, among all the fourth data, based on the fourth data with the largest file capacity (if there are two or more fourth data with the same file capacity, one of them is randomly specified), 10% of the fourth data with the farthest storage time from this largest-capacity fourth data are selected. Then, through the time range, other fourth data can be deleted from the fourth data with the largest single-file storage capacity as the standard, so as to reduce the alternative fourth data that can be spliced into the first data, so as to reduce the quantity of the fourth data split from the first data, thereby being able to ensure the data synthesis speed and reduce the storage pressure of SQL data as much as possible. Among them, deleting within the range of the same storage time can make the storage times of the alternative fourth data similar, and their storage positions in the hard disk are also similar. Then, the coordination of these files being retrieved is better, and the data retrieval speed is faster. In other words, when there is too much fourth data combined in the present invention, some data with a relatively far storage time are selectively deleted to improve the data retrieval speed and user experience. It can be regarded as a feedback loop of a convolutional neural network.

[0114] See Figure 1 , an intelligent SQL database processing system of the present invention includes

[0115] a data storage module, when it stores the first data:

[0116] Generate the first hash value of the first data and the second hash value of the second data already stored in the SQL database through the hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If so, it is not necessary to store the first data in the SQL database and update the first hash value of the first data to the data list, that is, it is regarded that the first data has been stored in the SQL database;

[0117] If not, determine whether the capacity of the first data exceeds a first threshold. If it does not exceed, store the first data in the SQL database and update the first hash value of the first data to the data list; if it exceeds, determine whether there is content in the second data that coincides with the first data and exceeds a second threshold of the content of the first data. If not, store the first data in the SQL database and update the first hash value of the first data to the data list;

[0118] If so, convert the non - coincident part of the first data and the second data into third data, generate a third hash value of the third data, store the third data in the SQL database, update the third hash value of the third data to the data list, and then update the coincident part of the first data and the second data to the data list;

[0119] When the data retrieval module retrieves the first data:

[0120] Judge whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data according to the data list. If so, convert the second data into the first data and output it; if not, synthesize the third data and the second data corresponding to the first data stored in the data list into the first data and output it.

[0121] The present invention first finds the same data through the hash value of the data, so as to store the second data with the same hash value as the first data, and there is no need to directly store the first data, thus reducing the storage capacity pressure of the SQL database; if not found, the first data with too small storage capacity is not split but directly stored to improve the retrieval speed; and the coincident part of the first data with larger storage capacity and the second data is not stored either, thus reducing the storage capacity pressure of the SQL database, and storing the non - coincident part of the first data and the second data as the third data separately in the SQL database. The present invention can achieve a smaller - capacity storage and ensure a relatively fast retrieval speed through the above - mentioned method.

[0122]

[0123] It should be noted that the content and hash value of the above - mentioned data list are only examples. The actual hash should be obtained by the SHA256 algorithm or the SHA512 algorithm to ensure a relatively low coincidence probability and avoid data storage errors caused by different data having the same hash value. And the above content can be video content, photo content, text content, audio content, code content.

[0124] Among them, the first threshold can be [1b, +∞), preferably 10mb. That is to say, when there is a second data that is the same as the first data that has never been stored in the SQL database, and the capacity of this first data does not exceed 10mb, this first data is directly stored in the SQL database without being split, so as to ensure the retrieval speed. Because of the current retrieval characteristics of the hard disk, when the total retrieval capacity remains unchanged, the more files there are and the smaller the storage capacity of each file, the slower the retrieval speed. Then, too small a split and too many file numbers will inevitably affect the retrieval speed. Therefore, when the capacity of the first data is too small, splitting is not recommended, after all, such small-capacity files will not occupy too much storage space.

[0125] Among them, the second threshold can be [0.1%, 99.9%], preferably 10%. That is to say, as long as the overlapping capacity of the entire content of a second data with a whole block of the content of the first data exceeds 10% of the total capacity of the first data, then this whole block of the first data is not stored, but only recorded in the data list. When it is needed for retrieval, the second data as the overlapping part and the third data as the non-overlapping part are integrated into the first data and output. It should be explained that this whole block of overlapping data cannot be calculated for overlap with other second data anymore. In other words, after the overlapping part of a whole block of the first data with one of the second data is determined, it is deleted, and then it is checked whether it overlaps with other second data by more than the second threshold. Finally, the remaining non-overlapping part is generated as the third data for storage.

[0126] Among them, the data list can be provided with the storage locations of the first data, the second data, and the third data on the hard disk, the corresponding hash values, and the guiding method for how the third data is combined into the first data.

[0127] In some embodiments, refer to Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, and generating the third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0128] S301. Obtain a second data that overlaps with the first data in a whole segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into a fourth data;

[0129] S302. Determine whether there is a fourth data. If so, jump to S303; if not, jump to S304;

[0130] S303. Delete the content of the first data that overlaps with the fourth data with the largest overlapping capacity, and jump to S301;

[0131] S304. Convert the remaining part of the first data into the content of the third data. The third data includes the data sequence numbers of the overlapping fourth data and the positions where the fourth data is located. Store the content of the third data in the SQL database and update it to the data list. Update the content of the third data corresponding to the first data, the data sequence numbers of the overlapping fourth data, and the positions where the fourth data is located to the data list.

[0132] In the present invention, the available fourth data is selected according to the degree of overlap from large to small, and the remaining is used as the third data, which can increase the capacity of the fourth data with the largest capacity as much as possible. Therefore, when retrieving the first data and synthesizing the first data, using the larger maximum fourth data to integrate into the first data can improve the processing speed of synthesizing the first data and retrieving the first data. That is to say, the first data is split according to the fourth data with the current overlap capacity from large to small, and finally only the third data remains, which helps to improve the retrieval speed.

[0133] It should be noted that the third data not only contains the non-overlapping part of the first data, but also contains the operation guide on how to re-synthesize the first data using the existing fourth data. The above operation guide is the data sequence number of the overlapping fourth data and the position where the fourth data is located.

[0134]

[0135]

[0136] The second threshold is [0.1%, 99.9%], preferably 10%. That is to say, only the fourth data whose capacity exceeds 10% of the capacity of the first data can be compared for whether it overlaps with an entire segment of the first data.

[0137] The above "entire segment overlap" and "overlap the entire segment" have the same meaning.

[0138] The content of the first data, the second data, and the third data can be code, photo, video, audio, text, etc., or can be a zip package or other data packet.

[0139] Assume that if the first data is a zip package, the second data can be a file that is the same as the file inside the zip package.

[0140] ​Alternatively, if the first data is a text, for example, a text consisting of 30 chapters, then the second data can be the text of one of the chapters.

[0141] In some embodiments, referring to Figure 1 , converting the non-overlapping part of the first data and the second data into the third data, generating a third hash value of the third data, storing the third data in an SQL database, updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including:

[0142] S301. Obtain the second data whose content overlapping with the first data is a whole segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into the fourth data;

[0143] S302. Determine whether there is the fourth data. If so, jump to S303; if not, jump to S304;

[0144] S303. Delete the content overlapping with the fourth data with the largest overlapping capacity in the first data, and jump to S301;

[0145] S304. Convert the remaining part of the first data into the content of the third data. The third data includes the data serial number of the overlapping fourth data and the position where the fourth data is located. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the overlapping fourth data, and the position where the fourth data is located to the data list.

[0146] The present invention selects the fourth data according to the degree of overlap from large to small for the available fourth data and uses the remaining as the third data, which can increase the capacity of the fourth data with the largest capacity as much as possible. Therefore, when retrieving the first data and synthesizing the first data, using the larger maximum fourth data to integrate into the first data can improve the processing speed of synthesizing the first data and retrieving the first data. That is to say, splitting the first data according to the fourth data with the current overlapping capacity from large to small, and finally only the third data remains, which helps to improve the retrieval speed.

[0147] It should be noted that the third data not only contains the non-overlapping part of the first data, but also contains an operation guide on how to re-synthesize the first data using the existing fourth data. The above operation guide is the data serial number of the overlapping fourth data and the position where the fourth data is located.

[0148]

[0149]

[0150] Among them, it should be noted that when multiple fourth data and third data jointly form the first data, the third data should include the data serial numbers of multiple fourth data and the positions where multiple fourth data are located, so as to facilitate synthesizing the initial first data through the content of the third data, the data serial numbers of multiple fourth data in the third data, and the positions where multiple fourth data are located, thereby reducing the storage capacity of the SQL database and reducing the storage pressure.

[0151] Among them, the second threshold is [0.1%, 99.9%], preferably 10%. That is to say, only the fourth data whose capacity exceeds 10% of the capacity of the first data can be compared to see if it coincides with an entire segment of the first data.

[0152] Among them, the above-mentioned "entire segment coincidence" and "coincide with an entire segment" have the same meaning.

[0153] The content of the first data, second data, and third data can be code, photo, video, audio, text, etc., or can be a zip package or other data packet.

[0154] Assume that if the first data can be a zip package, the second data can be a file that is the same as the file inside the zip package.

[0155] Or, if the first data is a text, for example, a text with 30 chapters, then the second data can be the text of one of the chapters.

[0156] In some embodiments, refer to Figure 1 , and store the content of the third data in the SQL database and update it to the data list. Updating the content of the third data corresponding to the first data, the data serial number of the coincident second data, and the position where the second data is located to the data list includes:

[0157] S401. Determine whether the number of fourth data that coincides with the first data exceeds the third threshold. If it exceeds, increase the second threshold, and then jump back to S301 again, and record the number of loop times once. If it does not exceed, then jump to S402;

[0158] S402. Determine whether the above-mentioned number of loop times exceeds the fourth threshold. If so, only retain the second data with the same file type as the first data and jump to S301, and clear the number of loop times; if not, then jump to S403;

[0159] S403. Store the content of the third data in the SQL database and update it to the data list. Update the content of the third data corresponding to the first data, the data serial number of the coincident second data, and the position where the second data is located to the data list.

[0160] Before storing the third data as described above, first verify whether the fourth data that makes up the third data is excessive. If it is excessive, it will affect the data splitting speed of the first data. Then, in order to keep the quantity of the above-mentioned fourth data within a suitable range, when it needs to be adjusted, that is, when the quantity of the fourth data exceeds the third threshold, first increase the second threshold, and start a loop from S301 again, recording the number of times of such a loop. If the number of loops exceeds the fourth threshold, it means that simply increasing the proportion of the capacity of the second data in the first data is too large, and then the admission threshold of the second data can be raised. That is, it is more difficult to synthesize different types of second data and the first data. For example, the data type of the second data is a photo, and the data type of the first data is a video. This photo does indeed overlap with a part of the content of this video, but such different types of data will inevitably lead to an excessive number of second data in terms of splitting and combination, resulting in a slow retrieval speed and a poor user experience. Therefore, if the fourth data still cannot be restricted within a reasonable third threshold when the number of loops exceeds the fourth threshold, it should be considered to limit the second data to the same type as the first data and continue the loop.

[0161] Among them, the third threshold can be [1, +∞), preferably 100. In other words, we split the combined first data into: only second data within 100 can be combined into one first data, avoiding excessive data splitting and increasing the data synthesis time, improving the data synthesis speed, and enhancing the user experience.

[0162] Among them, the fourth threshold can be [1, +∞), preferably 3. In other words, when the loop is 3 times, it is acceptable, but it also depends on the matching degree between the fourth threshold and the second threshold. The second threshold for the fourth loop should not make the overlapping part of the second data account for more than 100% of the first data, which is obviously an incorrect situation where no second data can be selected. We can configure the maximum number of loops to 3 times through the above loop algorithm. When the fourth data for combining the first data is excessive, appropriately raise the threshold of the fourth data to reduce the quantity of the fourth data, thereby improving the data combination speed. For example, after a maximum of 3 loops, raise the threshold of the overlapping proportion of the fourth data by 33.1%, which will inevitably reduce the quantity of the fourth data.

[0163] As a variant embodiment of the present invention, refer to Figure 1 , and store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the overlapping second data, and the position where the second data is located to the data list, including:

[0164] S401. Determine whether the number of fourth data that coincides with the first data exceeds a third threshold. If it exceeds, increase the second threshold, and then jump back to S301 and record the number of loop times once. If it does not exceed, jump to S402;

[0165] S402. Determine whether the above-mentioned number of loop times exceeds a fourth threshold. If so, delete the fourth data with the farthest storage time from the file capacity among the fifth threshold percentages of all fourth data in all fourth data and jump to S301, and clear the number of loop times; if not, jump to S403;

[0166] S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial number of the coincident second data, and the position where the second data is located to the data list.

[0167] Before storing the third data, the present invention first verifies whether there are too many fourth data that make up the third data. If there are too many, it will affect the data splitting speed of the first data. Then, in order to keep the number of the fourth data within a suitable range, when adjustment is needed, that is, when the number of the fourth data exceeds the third threshold, the second threshold is first increased, and the loop starts again from S301, and the number of times of such loop is recorded. If the number of loop times exceeds the fourth threshold, it means that the proportion of the capacity of the second data in the first data has increased too much simply, and the admission threshold of the second data can be raised. That is, assuming the fifth threshold is 10%, then among all the fourth data, taking a fourth data with the largest file capacity as the standard (if there are two or more fourth data with the same file capacity, one fourth data is randomly specified), the number of fourth data that is the farthest from the storage time of the fourth data with the largest capacity and is 10% of all the fourth data is deleted. Then, the number of fourth data that can be used as an alternative to splice the first data can be reduced, so as to reduce the number of fourth data split from the first data, so that both the data synthesis speed can be guaranteed and the storage pressure of the SQL data can be reduced as much as possible. Among them, deleting within the range of the same storage time can make the storage times of the alternative fourth data similar, and the storage positions in the hard disk are also similar. Then, the coordination of these files being retrieved is better, and the data retrieval speed is faster. In other words, when there are too many combined fourth data, the present invention selectively deletes some data with a relatively far storage time, improving the data retrieval speed and user experience. It can be regarded as a feedback loop of a convolutional neural network.

[0168] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent SQL database processing method, characterized in that: including When storing the first data: generating a first hash value of the first data and a second hash value of the second data already stored in the SQL database through a hash algorithm; judging whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If so, it is not necessary to store the first data in the SQL database and update the first hash value of the first data to the data list, that is, it is regarded that the first data has been stored in the SQL database; If not, judge whether the capacity of the first data exceeds a first threshold. If it does not exceed, store the first data in the SQL database and update the first hash value of the first data to the data list; If it exceeds, judge whether there is a second threshold in the second data where the content overlapping with the first data exceeds the content of the first data. If not, store the first data in the SQL database and update the first hash value of the first data to the data list; If so, convert the non-overlapping part of the first data and the second data into a third data, generate a third hash value of the third data, store the third data in the SQL database and update the third hash value of the third data to the data list, and then update the overlapping part of the first data and the second data to the data list; When retrieving the first data: judging according to the data list whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If so, convert the second data into the first data and output it; If not, synthesize the third data and the second data corresponding to the first data stored in the data list into the first data and output it; Converting the non-overlapping part of the first data and the second data into a third data, generating a third hash value of the third data, storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the overlapping part of the first data and the second data to the data list, including: S301. Obtain a second data whose content overlapping with the first data is a whole segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into a fourth data; S302. Convert the fourth data into a format that can be processed by a convolutional neural network; S303. Construct and train a convolutional neural network model that can receive the fourth data in a format that it can process and output at least one combination scheme of predicted fourth data, wherein each fourth data in the combination scheme has no intersection with the overlapping position of the first data; S304. Use the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data; S305. Calculate the remaining part as the third data after deleting the content overlapping with the at least one fourth data from the first data for each combination scheme, and select the combination scheme corresponding to the smallest capacity of the content of the third data as the optimal solution; S306. Output the optimal solution and its corresponding third data. The third data includes the data serial numbers and positions of the overlapping second data, store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions of the second data to the data list.

2. The intelligent SQL database processing method according to claim 1, characterized in that: Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions of the second data to the data list, including: S401. Determine whether the number of fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold, re-jump to S301, and record the number of loops once. If it does not exceed, jump to S402; S402. Determine whether the above number of loops exceeds the fourth threshold. If it is, only retain the second data with the same file type as the first data and jump to S301, and clear the number of loops; if not, jump to S403; S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions of the second data to the data list.

3. The intelligent SQL database processing method according to claim 1, characterized in that: Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions of the second data to the data list, including: S401. Determine whether the number of fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold, re-jump to S301, and record the number of loops once. If it does not exceed, jump to S402; S402. Determine whether the above number of loops exceeds the fourth threshold. If it is, delete the fourth data with the farthest storage time from the fourth data with the largest file capacity in the percentage of the fifth threshold in all fourth data and jump to S301, and clear the number of loops; if not, jump to S403; S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data serial numbers of the overlapping second data, and the positions of the second data to the data list.

4. An intelligent SQL database processing system, characterized in that: including a data storage module. When it stores the first data: Generate the first hash value of the first data and the second hash value of the second data already stored in the SQL database through the hash algorithm; determine whether there is a second hash value of the second data in the SQL database that is the same as the first hash value of the first data. If there is, there is no need to store the first data in the SQL database and update the first hash value of the first data to the data list, that is, it is considered that the first data has been stored in the SQL database; If not, determine whether the capacity of the first data exceeds the first threshold. If it does not exceed, store the first data in the SQL database and update the first hash value of the first data to the data list; If it exceeds, it is determined whether there is content in the second data that coincides with the first data and exceeds the second threshold of the content of the first data. If not, the first data is stored in the SQL database and the first hash value of the first data is updated to the data list; If there is, the non - coincident part of the first data and the second data is converted into the third data, and the third hash value of the third data is generated. The third data is stored in the SQL database and the third hash value of the third data is updated to the data list, and then the coincident part of the first data and the second data is updated to the data list; The data retrieval module, when it retrieves the first data: It is determined according to the data list whether there is a second hash value of the second data with the same first hash value as the first data in the SQL database. If there is, the second data is converted into the first data and output; If not, the third data and the second data corresponding to the first data stored in the data list are combined into the first data and output; Converting the non - coincident part of the first data and the second data into the third data, and generating the third hash value of the third data. Storing the third data in the SQL database and updating the third hash value of the third data to the data list, and then updating the coincident part of the first data and the second data to the data list, includes: S301. Obtain the second data whose content coincides with the first data in a whole segment and whose capacity is greater than the second threshold of the content of the first data, and convert it into the fourth data; S302. Convert the fourth data into a format that can be processed by the convolutional neural network; S303. Construct and train a convolutional neural network model, which can receive the fourth data in a format that it can process and output at least one combination scheme of the predicted fourth data, where each fourth data in the combination scheme has no intersection with the coincident position of the first data; S304. Use the trained convolutional neural network model to generate multiple possible combination schemes of the fourth data; S305. Calculate the remaining part as the third data after deleting the content that coincides with the at least one fourth data from the first data for each combination scheme, and select the combination scheme corresponding to the smallest capacity of the content of the third data as the optimal solution; S306. Output the optimal solution scheme and its corresponding third data. The third data includes the data sequence number of the coincident second data and the position where the second data is located, and store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the coincident second data and the position where the second data is located to the data list.

5. An intelligent SQL database processing system according to claim 4, characterized in that: And store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the coincident second data and the position where the second data is located to the data list, includes: S401. Determine whether the number of fourth data that coincides with the first data exceeds the third threshold. If it exceeds, increase the second threshold, and re - jump to S301, and record the number of loop times once. If it does not exceed, jump to S402; S402. Determine whether the above loop count exceeds the fourth threshold. If so, only retain the second data with the same file type as the first data and jump to S301, and clear the loop count; if not, jump to S403; S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data, and the position where the second data is located to the data list.

6. The intelligent SQL database processing system according to claim 4, wherein: And store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data, and the position where the second data is located to the data list, including: S401. Determine whether the number of fourth data overlapping with the first data exceeds the third threshold. If it exceeds, increase the second threshold, and re-jump to S301, and record the loop count once. If it does not exceed, jump to S402; S402. Determine whether the above loop count exceeds the fourth threshold. If so, delete the fourth data with the farthest storage time from the file capacity among the fifth threshold percentage of all fourth data in all fourth data and jump to S301, and clear the loop count; if not, jump to S403; S403. Store the content of the third data in the SQL database and update it to the data list, and update the content of the third data corresponding to the first data, the data sequence number of the overlapping second data, and the position where the second data is located to the data list.

Citation Information

Patent Citations

  • Data block uploading and storing system and method

    CN103873507A

  • File management method and device based on server, server and program product

    CN118170731A