Data writing processing method and device
By calculating the quality score and access characteristics of the data to be written in the index linked list, determining the index position, and directly writing data, the problem of low data writing efficiency in the index linked list is solved, and more efficient data writing processing is achieved.
Patent Information
- Application Number
- CN202411036925.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-03
AI Technical Summary
When writing data in the index linked list, since there may be multiple levels in the linked list, random probability calculations need to be performed layer by layer, resulting in complex and inefficient data writing process.
By obtaining the quality score of the data to be written, and combining the attribute characteristics of the index linked list and the access characteristics of the data to be written, the index position of the data to be written is determined, and the data writing process is directly carried out, simplifying the writing process and improving efficiency.
It improves the accuracy and efficiency of data writing, reduces the amount of calculation, simplifies the writing process, and improves the convenience of data writing in the index linked list.
Smart Images

Figure CN119065601A_ABST
Abstract
Description
Technical Field
[0001] This document relates to the field of data processing technology, and in particular to a data writing processing method and device. Background Art
[0002] With the continuous development of Internet technology and the continuous advancement of social information, more and more data are stored and queried online, such as storing data in a linked list or searching for data in data stored in a linked list.
[0003] At present, in the process of writing data into a linked list, since the linked list may have multiple levels, the data is usually written layer by layer from the bottom to the top. In the process of writing data in each layer, a random probability calculation is performed through a random function, and whether to write data in this layer is determined according to the calculated random probability, which makes the data writing process more complicated and the data writing efficiency lower. Summary of the invention
[0004] An object of one embodiment of the present specification is to provide a data writing processing method and device to solve the technical problem of how to improve the data writing efficiency of an index linked list.
[0005] To solve the above technical problems, an embodiment of this specification is implemented as follows:
[0006] In a first aspect, an embodiment of the present specification provides a data writing processing method, the method comprising:
[0007] Obtaining data to be written, and calculating a quality score of the data to be written;
[0008] Reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics;
[0009] The data to be written is processed into the index linked list according to the index position.
[0010] In a second aspect, another embodiment of the present specification provides a data writing processing device, the device comprising:
[0011] A data acquisition module, used to acquire the data to be written and calculate the quality score of the data to be written;
[0012] A position determination module, used for reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics;
[0013] A data writing module is used to write the data to be written into the index linked list according to the index position.
[0014] In a third aspect, another embodiment of the present specification provides a data writing processing device, the device comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the data writing processing method as described in the first aspect above.
[0015] In a fourth aspect, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the data writing processing method described in the first aspect above are implemented.
[0016] In a fifth aspect, another embodiment of the present specification provides a computer program product, wherein the computer program product includes a data writing processing program, and the data writing processing program is executed by a processor to implement the steps of the data writing processing method as described in the first aspect above.
[0017] The data writing processing method provided in the present embodiment calculates the quality score of the data to be written after obtaining the data to be written during the process of performing data writing processing in the index linked list. Secondly, the index position for writing the data to be written into the index linked list is determined according to the attribute characteristics of the index linked list, the access characteristics of the data to be written and the quality score of the data to be written. The data writing processing of the data to be written into the index linked list is performed according to the index position. Therefore, the index position of the data to be written is determined from three aspects, namely the quality score of the data to be written, the access characteristics of the data to be written and the attribute characteristics of the index linked list, so as to improve the accuracy of the index position of the data to be written. At the same time, since the index position of the data to be written is directly determined from the above three aspects and the data is written according to the index position, the writing process of the data to be written can be simplified, thereby improving the efficiency and convenience of data writing. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate one or more embodiments of the present specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.
[0019] Figure 1A processing flow chart of a data writing processing method provided by one embodiment of this specification;
[0020] Figure 2 A schematic diagram of a data processing system provided for one embodiment of this specification;
[0021] Figure 3 A schematic diagram of a hash table provided for one embodiment of this specification;
[0022] Figure 4 A processing flow chart of a data writing processing method applied to a table skipping scenario provided by one embodiment of this specification;
[0023] Figure 5 A schematic diagram of a data writing processing device provided in one embodiment of this specification;
[0024] Figure 6 A schematic diagram of the structure of a data writing processing device provided in one embodiment of the present specification. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0026] An embodiment of a data writing processing method provided in this specification is as follows:
[0027] In actual applications, when writing data in an index linked list, since the index linked list may have multiple levels, it is usually determined layer by layer from the bottom to the top whether to write the data. In this process, random probability calculations must be performed through random functions, which makes the data writing process more complicated and the data writing efficiency lower.
[0028] In view of this, the data writing processing method provided in the present embodiment firstly calculates the quality score of the data to be written after acquiring the data to be written, and secondly, determines the index position for writing the data to be written into the index linked list according to the attribute characteristics of the index linked list, the access characteristics of the data to be written and the quality score of the data to be written, and performs data writing processing on the data to be written into the index linked list according to the index position. Therefore, the index position of the data to be written is determined from three aspects: the quality score of the data to be written, the access characteristics of the data to be written and the attribute characteristics of the index linked list, so as to improve the accuracy of the index position of the data to be written. At the same time, since the index position of the data to be written is directly determined from the above three aspects and the data is written according to the index position, it is possible to simplify the complex process of performing random probability calculation layer by layer and reduce the amount of calculation, thereby improving the efficiency and convenience of writing data in the index linked list.
[0029] The data writing processing method provided in this embodiment can be executed by a computer device, which can be a terminal or a server, wherein the terminal can include a mobile phone, a laptop computer, an intelligent interactive device, etc., and the server can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.
[0030] Reference Figure 1 The data writing processing method provided in this embodiment specifically includes the following steps S102 to S106.
[0031] Step S102, obtaining data to be written, and calculating a quality score of the data to be written.
[0032] The data to be written in this embodiment refers to the data to be written into the index linked list; the data content of the data to be written is not specifically limited in this embodiment, and the data to be written can be data of any data content. This embodiment can be applied to a data processing system, such as Figure 2 As shown, the data processing system may include a data storage module, a feature storage module, an index layer calculation module, a data recovery module and a query counting module.
[0033] In specific implementation, after obtaining the data to be written, in order to be able to conveniently write the data to be written into the index linked list, the quality score of the data to be written can be calculated. The higher the quality score of the data to be written, the more likely the data to be written will become more popular data after being written into the index linked list. In the process of calculating the quality score of the data to be written, in order to improve the flexibility and comprehensiveness of calculating the quality score, different score calculation methods can be provided for data to be written of different data categories.
[0034] The data classification of the data to be written includes the data type of the data to be written; the data classification of the data to be written includes text classification and / or image classification. In addition, the data classification of the data to be written may also be other classifications, such as video classification, screen recording classification, etc. The scoring calculation method refers to the method of calculating the quality score of the data to be written; the scoring calculation method includes the scoring calculation method corresponding to the text classification, the scoring calculation method corresponding to the image classification, or the scoring calculation method corresponding to the text classification and the image classification.
[0035] During the specific execution process, the data classification of the data to be written may only include text classification or image classification. In this case, in order to improve the accuracy of the quality score of the data to be written, the data to be written may be scored and calculated according to the scoring calculation method corresponding to the data classification of the data to be written to obtain the quality score of the data to be written. Specifically, in an optional implementation manner provided by this embodiment, in the process of calculating the quality score of the data to be written, the following operations are performed:
[0036] If the data classification of the data to be written is text classification, the data to be written is input into a natural language processing algorithm for score calculation to obtain the quality score;
[0037] If the data classification of the data to be written is image classification, a score calculation is performed on the data to be written using an image scoring algorithm to obtain the quality score.
[0038] In addition, in addition to the above data classification of the data to be written being only text classification or only image classification, the data classification of the data to be written may also include text classification and image classification at the same time, and the importance of text classification and image classification to the quality score of the data to be written is different. In view of this, in order to improve the effectiveness and comprehensiveness of the quality score of the data to be written; in another optional implementation manner provided by this embodiment, if the data classification of the data to be written includes text type and image classification, in the process of calculating the quality score of the data to be written, the following operations are performed:
[0039] Reading text data classified as text classification and image data classified as image classification in the data to be written;
[0040] Inputting the text data into a natural language processing algorithm to perform scoring calculation to obtain a text quality score, and performing scoring calculation on the image data using an image scoring algorithm to obtain an image quality score;
[0041] The quality score is calculated according to the text quality score, the image quality score, and a data ratio of the text data to the image data.
[0042] Among them, the text quality score refers to the quality score of text data; the image quality score refers to the quality score of image data; the data ratio includes the ratio of the data volume of the text data to the data volume of the image data; the image scoring algorithm can be an image aesthetics algorithm.
[0043] Specifically, in the process of calculating the quality score of the data to be written based on the text quality score, the image quality score, and the data ratio of the text data to the image data, the text weight of the text data and the image weight of the image data can be calculated based on the data ratio of the text data to the image data, the first product of the text quality score and the text weight of the text data is calculated, the second product of the image quality score and the image weight of the image data is calculated, and the sum of the first product and the second product is calculated as the quality score of the data to be written.
[0044] It should be supplemented that the above-mentioned another optional implementation method of calculating the quality score of the data to be written can also be performed on the basis of the above-mentioned one optional implementation method of calculating the quality score of the data to be written.
[0045] Step S104, reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics.
[0046] After the quality score of the data to be written is obtained by the above calculation, in this step, the index position for writing the data to be written into the index linked list is determined according to the attribute characteristics of the index linked list, the access characteristics of the data to be written and the quality score of the data to be written. Since the higher the quality score of the data to be written, the more likely the data to be written is to become hot data, that is, the higher the probability of the data to be written being accessed, the index position of the data to be written is determined to be more accurate by combining the quality score of the data to be written, the access characteristics and the attribute characteristics of the index linked list, and the index position of the data to be written is more adapted to the access heat of the data to be written.
[0047] The index linked list described in this embodiment includes a multi-layer index linked list, that is, a linked list including multiple index layers, the index linked list is a ZSET (Sorted Set) data structure, and the index layer refers to the index layer for data query in the index linked list; the index linked list can specifically be a skip list, that is, a jump list, and the index linked list includes multiple index layers, each index layer is a linked list; the attribute characteristics of the index linked list include attribute parameters of the index linked list, and the attribute characteristics of the index linked list include the data volume of the index linked list, the total number of index layers and / or the maximum linked list length, the data volume of the index linked list refers to the amount of data stored in the index linked list, the total number of index layers refers to the total number of index layers included in the index linked list, and the maximum linked list length includes the length of the lowest index layer in the index linked list, that is, the amount of data stored in the lowest index layer, and the lowest index layer stores the full amount of data; Figure 2 As shown, in the index linked list stored in the data storage module, LV 1 is the uppermost index layer, and LV 4 is the lowermost index layer, that is, from LV 1 to LV 4, the levels gradually decrease, that is, LV 3 is the upper layer of LV 4, LV 2 is the upper layer of LV 3, LV 1 is the upper layer of LV 2, LV 4 is the lower layer of LV 3, LV 3 is the lower layer of LV 2, LV 1, LV 2 and LV 3 are the upper layer of LV 4, and LV 4, LV 3 and LV 2 are the lower layer of LV 1.
[0048] The index linked list in this embodiment can be applied to any field. For example, if the index linked list stores the scores of students, the scores are stored according to the student's identity. For another example, if the index linked list stores the sales volume of a company's products, the sales volume is stored according to the product identity. The index linked list in this embodiment can be deployed in a data storage module.
[0049] The access characteristics of the data to be written refer to the relevant characteristics for accessing the data to be written, and the access characteristics may be access parameters; the access characteristics include query frequency and / or write frequency, that is, the frequency of querying the data to be written and / or the frequency of writing the data to be written; optionally, the access characteristics of the data to be written are obtained after the characteristics are initialized. The index position refers to the position in the index linked list where the data to be written is written, and the index position may be an index layer in the index linked list.
[0050] During specific implementation, the index layer calculation module reads the attribute characteristics of the index linked list and the access characteristics of the data to be written from the feature storage module, and determines the index layer identifier for writing the data to be written to the index linked list based on the quality score of the data to be written, the attribute characteristics of the index linked list and the access characteristics of the data to be written.
[0051] In practical applications, since the index linked list is multi-layered, in the process of writing data in the index linked list, it is necessary to determine layer by layer whether to write the data to the previous layer, and each layer needs to calculate the random probability, resulting in low data writing efficiency. In addition, since whether to write the data to the previous layer is determined by random probability, the accuracy of the index position of the determined data is also low. In view of this, in order to improve the efficiency of writing data in the index linked list and improve the accuracy of the index position of the determined data; in an optional implementation manner provided by this embodiment, in the process of determining the index position of the data to be written to the index linked list according to the quality score, the attribute characteristics and the access characteristics, the following operations are performed:
[0052] The quality score, the attribute feature and the access feature are input into an index layer calculation algorithm to perform index layer calculation, and an index layer identifier for writing the data to be written into the index linked list is obtained.
[0053] The index layer identifier refers to an identifier indicating which index layer it is.
[0054] For example, the index layer calculation algorithm includes:
[0055]
[0056] Among them, f(x) represents the index layer calculation algorithm; v represents the total number of index layers of the index linked list; m represents the maximum linked list length of the index linked list; q represents the quality score of the data to be written; s represents the query frequency of the data to be written; w represents the write frequency of the data to be written; x represents the index layer identifier of the data to be written; and n represents the amount of data in the index linked list.
[0057] Step S106: write the data to be written into the index linked list according to the index position.
[0058] After determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics, in this step, the data writing process of the data to be written into the index linked list is performed according to the index position.
[0059] In specific implementation, in order to realize the automation of writing data to the index linked list, the data writing program can be called to perform data writing processing on the data to be written to the index linked list according to the index position. Specifically, the index position of the data to be written can be determined according to the quality score of the data to be written or the data identifier of the data to be written, and the index position can be marked using the quality score of the data to be written or the data identifier of the data to be written, and the marked index position can be associated with the data to be written;
[0060] The data identifier of the data to be written includes an identifier representing the uniqueness of the data to be written; the index position includes a specific position in the index layer, such as the position in the index layer.
[0061] like Figure 2 As shown, assuming that the index layer identifier for writing the data to be written into the index linked list is 2, the data can be stored in the index linked list according to the data identifier, that is, it can be stored in the index linked list in ascending order of the data identifier. Assuming that the data identifier of the data to be written is Data8, according to the data identifier Data8 of the data to be written, the index layer corresponding to the index layer identifier 2 in the index linked list determines that the index position of the data to be written is 2, and the index position 2 is marked as the data identifier Data8 of the data to be written, and the marked index position 2 is associated with the data to be written.
[0062] In actual applications, during data query in an index linked list, if certain data is found in an upper layer, the lower layers will not be queried. Therefore, the highest index layer storing data in the index linked list is important for the query of the data, and the index layer below the highest index layer still stores the data because it is used to assist in the query of other data. In view of this, in order to further improve the data writing efficiency, in an optional implementation manner provided by this embodiment, after inputting the quality score, the attribute feature and the access feature into an index layer calculation algorithm for index layer calculation and obtaining an index layer identifier for writing the data to be written to the index linked list, after the data writing process of the data to be written to the index linked list is performed according to the index position, the following operation is further performed:
[0063] The data to be written is written into each index layer after the index layer corresponding to the index layer identifier in the index chain list through a scheduled task.
[0064] Specifically, after writing the data to be written to the index layer corresponding to the above-mentioned index layer identifier, because the function of querying the data to be written in the index linked list is already available, the data to be written can be asynchronously written in each index layer after the index layer corresponding to the index layer identifier.
[0065] Continuing with the above example, the data to be written is written to the index layers (LV 3, LV 4) after the index layer corresponding to the index layer identifier 2 in the index linked list through a scheduled task. Similarly, according to the data identifier Data8 of the data to be written, in the index layers LV 3 and LV 4 in the index linked list, the index position of the data to be written is determined to be 3, the index position 3 is marked as the data identifier Data8 of the data to be written, and the marked index position 3 is associated with the data to be written.
[0066] In addition, after the data writing process of the data to be written is performed to the index linked list according to the index position, the data writing operation of the data to be written in each index layer after the index layer corresponding to the index layer identifier in the index linked list can also be encapsulated as a message, and the message is written into a message queue, so as to read the message from the message queue and perform message processing. Through the above-mentioned data writing method from the upper layer to the lower layer, multiple random probability calculations are avoided when determining whether to write data from the lower layer to the upper layer, thereby improving the convenience and efficiency of data writing.
[0067] It should be supplemented that after the data writing processing of the data to be written is executed to the index linked list according to the index position or after the data writing of the data to be written is executed in each index layer after the index layer corresponding to the index layer identifier in the index linked list through a scheduled task, the feature storage module can asynchronously update the data volume and / or the maximum linked list length of the index linked list, and asynchronously update the writing frequency of the data to be written; the asynchronous update here includes updating through a scheduled task or encapsulating the update operation into a message and writing it into a message queue to read the message from the message queue and perform message processing.
[0068] In practical applications, there is a need to query data in an index linked list. In the process of querying data in an index linked list, it is often hoped that data with higher popularity is stored in the upper layer of the index linked list as much as possible, and data with lower popularity is stored in the lower layer of the index linked list as much as possible, so as to improve the efficiency of data query. In view of this, in the process of querying data, the update access heat of the query data can be calculated, and the access heat of the query data in the access heat table is replaced with the update access heat. In an optional implementation manner provided in this embodiment, the following operations are also performed:
[0069] Calculate the update access heat of the query data according to the index layer of the query data in the index chain list, the quality score of the query data and the query frequency;
[0070] Based on the updated access heat of the query data, the access heat of the query data is updated in the access heat table.
[0071] Access heat refers to the heat of accessing the query data, that is, the query heat of querying the query data; the higher the access heat, the more times the query data is queried. The access heat table includes a table storing query data and corresponding access heat; the access heat table can be stored in Figure 2 The query counting module shown can calculate the updated access heat of the query data through the query counting module and update the access heat of the query data in the access heat table based on the updated access heat of the query data. The query data includes query data corresponding to the first query type, the first query type can be a batch query type, and the query data includes batch query data.
[0072] Specifically, after detecting a query instruction for querying data, the query counting module can asynchronously update the query frequency of the query data, that is, asynchronously update the query frequency of the query data in the query frequency table, and the query frequency table contains the hash value of each data and the corresponding query frequency; the feature storage module can read the query frequency of the query data and update the query frequency of the query data; after detecting a query instruction for querying data, the feature storage module can also directly update the query frequency of the query data; asynchronous update refers to updating through a scheduled task or a message;
[0073] After detecting the query instruction of the query data, the index layer identifier of the index layer where the query data is located in the index linked list, the quality score of the query data and the query frequency can be input into the heat calculation algorithm to perform heat calculation to obtain the updated access heat of the query data, and replace the access heat of the query data in the access heat table with the updated access heat of the query data.
[0074] For example, heat calculation algorithms include:
[0075]
[0076] Among them, h represents the update access popularity of the query data; s represents the query frequency of the query data; q represents the quality score of the query data; and x represents the index level of the query data in the index linked list.
[0077] Since the index linked list is suitable for batch data query, in order to improve the convenience of data query, the above-mentioned query data can be obtained in the following way: if the query type of the data query is the first query type, the data query can be performed in the index linked list based on the query identification range (such as the sorting position range) to obtain the query data; optionally, the first query type is determined when the query instruction for the data query carries the query identification range, and the first query type here can be a batch query type.
[0078] In actual application scenarios, there may be some cold data with low heat that occupies the upper layer of the index linked list for a long time, consuming more memory resources. The upper layer with better query efficiency does not necessarily store data with higher heat. In view of this, in order to avoid the cold data with low heat occupying the upper layer of the index linked list for a long time, causing resource waste on the upper layer of the index linked list, and improving the resource utilization of the index linked list; in an optional implementation manner provided by this embodiment, the following operations are also performed:
[0079] Calculating data redundancy of the index linked list according to the amount of data in each index layer in the index linked list and the total number of index layers in the index linked list;
[0080] The index linked list is cleaned up based on the data redundancy.
[0081] The data quantity of each index layer refers to the quantity of data stored in each index layer; and the data redundancy of the index linked list includes the redundancy degree of the data stored in the index linked list.
[0082] Specifically, the amount of data in each index layer of the index linked list and the total number of index layers of the index linked list may be input into a redundancy calculation algorithm to perform redundancy calculation, thereby obtaining data redundancy of the index linked list, and performing data cleaning processing on the index linked list based on the data redundancy; the data redundancy of the index linked list may be calculated and the data cleaning processing may be performed as follows: Figure 2 The data recovery module shown is executed.
[0083] For example, redundancy calculation algorithms include:
[0084]
[0085] Among them, k represents data redundancy; n represents the total number of index levels in the index list; L 1 , L 2 ...L n Represents the amount of data in each index layer in the index linked list, that is, the amount of data stored in each index layer.
[0086] On this basis, in order to clean up the cold data in the index linked list in time and release the storage resources in the index linked list, in an optional implementation manner provided by this embodiment, in the process of performing data cleaning processing on the index linked list based on the data redundancy, the following operations are performed:
[0087] Detecting whether the data redundancy is greater than a preset redundancy threshold;
[0088] If not, no action is required;
[0089] If yes, a cleaning step is calculated according to the amount of data and the total number of index layers, and data is cleaned in the index chain list according to the access popularity of each data in the index chain list and the cleaning step.
[0090] Optionally, the cleaning step includes the number of cleanings performed on the data in the index linked list; optionally, the access heat of each data in the index linked list is read from the access heat table. The preset redundancy threshold refers to a pre-set redundancy threshold, and the preset redundancy threshold may be 0.8.
[0091] Specifically, in the process of calculating the cleaning step length according to the data quantity and the total number of index layers, the data quantity of each index layer in the index linked list and the total number of index layers in the index linked list can be input into the step length calculation algorithm for step length calculation to obtain the cleaning step length.
[0092] For example, the step size calculation algorithm includes:
[0093]
[0094] Among them, p represents the cleaning step length; n represents the total number of index levels in the index linked list; L 1 , L 2 ...L n Represents the amount of data at each index level in the index linked list.
[0095] In order to further clean up some cold data in the index linked list from the upper layer, release the storage resources of the index linked list, ensure that the data with higher popularity is stored in the upper layer of the index linked list as much as possible through the cleaning mechanism, improve the query efficiency of the data with higher popularity, and reduce the impact of the increase in data volume on the performance of the index linked list by reducing the occupation of storage resources; in an optional implementation manner provided by this embodiment, in the process of cleaning up data in the index linked list according to the access popularity of each data in the index linked list and the cleaning step length, the following operations are performed:
[0096] According to the priority of each index layer in the index chain table, detecting whether the access heat of the data of each index layer is less than a preset heat threshold;
[0097] If so, the target data of the target index layer in the index linked list whose access heat is less than the preset heat threshold, or the target data and the target data of at least one index layer after the target index layer are deleted, and an updated index linked list is obtained after the number of data detections reaches the cleaning step.
[0098] Optionally, the detection priority of the upper index layer is higher than the detection priority of the lower index layer, such as Figure 2As shown, the data of the LV 1 index layer is detected first, and then the data of the LV 2 index layer and the data of the LV 3 index layer are detected; each index layer in the index chain list includes other index layers except the bottom index layer in the index chain list, that is, except Figure 2 The LV 1 index layer, LV 2 index layer and LV 3 index layer other than the LV 4 index layer in the index chain list shown. The data detection number includes the number of data whose access heat has been detected in the index chain list; the updated index chain list refers to the index chain list after cleaning or deletion.
[0099] Specifically, in the process of detecting whether the access heat of the data of each index layer is less than the preset heat threshold according to the priority of each index layer in the index linked list, it is possible to detect whether the access heat of the data of each index layer is less than the preset heat threshold in order from the upper index layer to the lower index layer.
[0100] In addition, the target data of the target index layer whose access heat is less than the preset heat threshold in the index linked list may be deleted according to the preset number of layers, or the target data and the target data of at least one index layer after the target index layer may be deleted, and an updated index linked list may be obtained after the number of data detections reaches the cleaning step.
[0101] After the above-mentioned detection of whether the access heat of the data of each index layer is less than the preset heat threshold according to the priority of each index layer in the index chain list is performed, if the execution result is no, the following operations may be performed:
[0102] Detecting whether the access heat of the data in each index layer is less than a second preset heat threshold;
[0103] If so, the target data of the target index layer in the index linked list whose access heat is less than the second preset heat threshold, or the target data and the target data of at least one index layer after the target index layer are deleted according to the second preset number of layers, and an updated index linked list is obtained after the number of data detections reaches the cleaning step; optionally, the second preset heat threshold is greater than the preset heat threshold, and the second preset number of layers is less than the preset number of layers.
[0104] like Figure 2As shown, in the order from the upper index layer to the lower index layer, detect whether the access heat of Data1 of the LV 1 index layer is less than the preset heat threshold. If so, delete Data1 of the LV 1 index layer in the index linked list according to the preset layer number 1, or delete Data1 of the LV 1 index layer and Data1 of the LV 2 index layer in the index linked list according to the preset layer number 2; if the access heat of Data1 of the LV 1 index layer is greater than or equal to the preset heat threshold, detect whether the access heat of Data1 of the LV 1 index layer is less than the second preset heat threshold. If less than, delete Data1 of the LV 1 index layer and Data1 of the LV 2 index layer in the index linked list according to the second preset layer number 2, or delete Data1 of the LV 1 index layer, Data1 of the LV 2 index layer and Data1 of the LV 3 index layer in the index linked list according to the second preset layer number 3, and then continue to detect the access heat of Data76 of the LV 1 index layer until the number of data detections reaches the cleaning step to obtain an updated index linked list.
[0105] After the data cleaning process is performed on the index linked list based on the data redundancy, a data query may be performed at the index layer in the index linked list based on the data identifier of the query data to obtain relevant files of the query data associated with the data identifier.
[0106] In this embodiment, data can be stored in the index linked list according to quality score or data identifier. In addition, data can also be stored according to other types, such as storing data in the index linked list according to quality score. In this case, data query can be performed in the index linked list based on the ranking range of quality score, and the ranking range here is, for example, the 5th to 7th place in quality score; specifically, the distance between the data and the next data can be recorded, and the distance number can be obtained by continuously accumulating span, and when it is detected that the distance number is greater than the starting query identifier, jump to the next index layer of the current index layer in the index linked list, and continue to accumulate span until all data in the ranking range is queried; the starting query identifier refers to the starting identifier for data query, such as querying data with quality scores of 5th to 7th, the starting query identifier is 5;
[0107] For example, if you need to query the data with quality scores from 5th to 7th in the index list:
[0108] Starting from the top layer, span:1+span:4==5. Assuming that span:1 corresponds to a quality score of 0 and span:4 corresponds to a score of 10, it can be calculated that the score corresponding to the 5th place is 20. Then, starting from the score of 20, the bottom layer continuously obtains two data, and uses score:20 and the two obtained data as the data for the 5th to 7th places, i.e., the query data. The average time complexity of the data query here is O[(LogN)+M], where N represents the amount of data in the index linked list and M represents the amount of data obtained by the query.
[0109] As mentioned above, if the query type for data query is the first query type, data query can be performed in the index linked list based on the query identifier range to obtain query data; in addition, in actual applications, the time complexity of performing a single data query through the index linked list is relatively high, so in order to improve the flexibility of data query and improve the efficiency of data query; in an optional implementation manner provided by this embodiment, the following operations are also performed:
[0110] If the query type of the data query is a specific query type, a hash value of the query identifier of the data query is calculated;
[0111] An index position mapped by the query identifier in the hash table is determined based on the hash value, and query data corresponding to the query identifier is acquired from the mapped index position.
[0112] The specific query type may be the second query type, and the second query type may be a single query type; optionally, the specific query type is determined when a query instruction for data query carries a query identifier, and the query identifier may be a query key, i.e., a key for data query in a hash table, such as a query identifier of 27. An index position may be a slot in a hash table, and a hash value may be an integer or a number in other forms. A hash table refers to a table for data storage.
[0113] like Figure 3 The hash table shown contains 6 slots or buckets, each slot can store a key-value pair, each key-value pair consists of a key and a corresponding value. Assume that the hash function maps the key to the index position of the slot, for example, key 5 is mapped to slot 0, key 8 is mapped to slot 1, key 12 is mapped to slot 2... By storing the key-value pairs in the corresponding slots, the efficiency of data query in the hash table is achieved.
[0114] Any kind of data can be stored in a hash table. For example, the key in a hash table can be a student's identity and the value can be the student's score. The keys in a hash table can be arranged in ascending order or descending order. Another example is that the key in a hash table can be a product ID and the value can be the product sales volume. Another example is that the key in a hash table can be a student's identity and the value can be the student's physical examination data, and so on.
[0115] Specifically, in the process of calculating the hash value of the query identifier for data query, the hash value of the query identifier for data query may be calculated by a hash function.
[0116] On this basis, in order to accurately map the query identifier to the index position of the hash table through the hash value to avoid exceeding the storage range of the hash table; in an optional implementation manner provided by this embodiment, in the process of determining the index position mapped by the query identifier in the hash table based on the hash value, the following operations are performed:
[0117] Performing modulo processing on the table capacity of the hash table according to the hash value to obtain a remainder;
[0118] The remainder is used as the index position mapped by the query identifier in the hash table.
[0119] The table capacity of the hash table refers to the total capacity of the hash table that can be used for data storage.
[0120] For example, the hash value of the query key calculated by the hash function is 123. The table capacity 100 of the hash table is modulo the hash value 123 of the query key to obtain a remainder 23. The remainder 23 is used as the index position mapped by the query key in the hash table, that is, the query key is mapped to the slot with index position 23 in the hash table, and the query data corresponding to the query key can be obtained in the slot corresponding to the index position. The keys can be evenly distributed in the hash table through a hash function with high performance, avoiding the occurrence of multiple keys mapping to the same slot. At the same time, the table capacity of the hash table can be greater than the preset capacity threshold, so that a low conflict rate can be maintained when storing large amounts of data.
[0121] It should be supplemented that each optional implementation method in step S102 to step S108 provided in this embodiment can be executed independently as needed, or can be combined and referenced with each other. At the same time, each specific execution step in each optional implementation method can also be executed independently, and this embodiment does not make specific limitations.
[0122] In summary, the data writing processing method provided in the present embodiment first obtains the data to be written. If the data classification of the data to be written is text classification, the data to be written is input into a natural language processing algorithm for scoring calculation to obtain a quality score of the data to be written. If the data classification of the data to be written is image classification, the data to be written is scored and calculated by an image scoring algorithm to obtain a quality score of the data to be written. If the data classification of the data to be written includes text classification and image classification, the text data classified as text classification and the image data classified as image classification in the data to be written are read, the text data is input into a natural language processing algorithm for scoring calculation to obtain a text quality score, and the image data is scored and calculated by an image scoring algorithm to obtain an image quality score. The quality score of the data to be written is calculated according to the text quality score, the image quality score and the data ratio of the text data to the image data. By providing different scoring calculation methods for data of different data classifications, the flexibility of calculating the quality score of the data to be written is improved, and the effectiveness of the quality score of the data to be written is improved.
[0123] Secondly, the attribute characteristics of the index linked list and the access characteristics of the data to be written are read, the quality score, attribute characteristics and access characteristics are input into the index layer calculation algorithm for index layer calculation, and the index layer identifier for writing the data to be written to the index linked list is obtained, and the data to be written is written to the index linked list according to the index layer corresponding to the index layer identifier. Since the index layer of the data to be written is determined directly from the three aspects of the quality score, attribute characteristics and access characteristics, and the data to be written is written into the index layer, the writing process of the data to be written can be simplified, thereby improving the efficiency and convenience of data writing.
[0124] The following combination Figure 4 , taking the application of the data writing processing method provided by this embodiment in the table jump scenario as an example, the data writing processing method provided by this embodiment is further described. Figure 4 The data writing processing method applied to the table skipping scenario specifically includes the following steps.
[0125] Step S402, obtaining the data to be written, and performing score calculation on the data to be written according to the score calculation method corresponding to the data classification of the data to be written, to obtain a quality score of the data to be written.
[0126] Step S404, read the attribute characteristics of the skip list and the access characteristics of the data to be written, and input the quality score, attribute characteristics and access characteristics of the data to be written into the index layer calculation algorithm for index layer calculation, and obtain the index layer identifier for writing the data to be written into the skip list.
[0127] Step S406 , writing the data to be written into the skip list according to the index layer corresponding to the index layer identifier.
[0128] Step S408 , writing the data to be written into each index layer after the index layer corresponding to the index layer identifier in the skip list through the scheduled task.
[0129] Step S410, calculating the update access heat of the query data according to the index layer where the query data is located in the skip list, the quality score of the query data, and the query frequency.
[0130] Step S412: based on the updated access heat of the query data, update the access heat of the query data in the access heat table.
[0131] Step S414, calculating the data redundancy of the skip list according to the amount of data in each index layer in the skip list and the total number of index layers in the skip list.
[0132] Step S416, detecting whether the data redundancy is greater than a preset redundancy threshold;
[0133] If yes, execute the following step S418;
[0134] If not, no action will be taken.
[0135] Step S418, calculate the cleaning step length according to the data quantity of each index layer and the total number of index layers, read the access heat of each data in the skip list from the access heat table, and clean the data in the skip list based on the access heat and the cleaning step length.
[0136] It should be noted that any one of steps S402 to S418 or any combination of multiple steps can be replaced by the corresponding technical means provided in the above steps S102 to S106 according to the needs of implementation and deployment, and will not be described one by one here.
[0137] Figure 5 A schematic diagram of a data writing processing device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device comprises:
[0138] The data acquisition module 502 is used to acquire the data to be written and calculate the quality score of the data to be written;
[0139] A position determination module 504 is used to read the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determine the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics;
[0140] The data writing module 506 is used to write the data to be written into the index linked list according to the index position.
[0141] The data writing processing device provided in this embodiment firstly obtains the data to be written by running the data acquisition module 502, and calculates the quality score of the data to be written, then runs the position determination module 504, reads the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determines the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics, and finally runs the data writing module 506 to write the data to be written into the index linked list according to the index position, thereby improving the data writing efficiency of the index linked list.
[0142] The data writing processing device provided in one embodiment of the present specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.
[0143] Furthermore, an embodiment of the present specification also provides a data writing processing device, Figure 6 A schematic diagram of a data writing processing device provided in an embodiment of the present specification is shown in FIG. Figure 6 As shown, the device includes: a memory 601, a processor 602, a bus 603 and a communication interface 604. The memory 601, the processor 602 and the communication interface 604 communicate through the bus 603, and the communication interface 604 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.
[0144] Figure 6 In the embodiment, the memory 601 stores computer executable instructions that can be run on the processor 602. When the computer executable instructions are executed by the processor 602, the following process is implemented:
[0145] Obtaining data to be written, and calculating a quality score of the data to be written;
[0146] Reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics;
[0147] The data to be written is processed into the index linked list according to the index position.
[0148] The data writing processing device provided in this embodiment, through the cooperation of the memory 601, the processor 602, the bus 603 and the communication interface 604, first obtains the data to be written and calculates the quality score of the data to be written; secondly, according to the attribute characteristics of the index linked list, the access characteristics of the data to be written and the quality score, determines the index position for writing the data to be written to the index linked list; finally, the data writing processing of the data to be written to the index linked list is performed according to the index position to improve the data writing efficiency of the index linked list.
[0149] The data writing processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.
[0150] Furthermore, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:
[0151] Obtaining data to be written, and calculating a quality score of the data to be written;
[0152] Reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics;
[0153] The data to be written is processed into the index linked list according to the index position.
[0154] The computer-readable storage medium provided in this embodiment first obtains the data to be written and calculates the quality score of the data to be written; secondly, determines the index position for writing the data to be written to the index linked list according to the attribute characteristics of the index linked list, the access characteristics of the data to be written, and the quality score; finally, writes the data to be written to the index linked list according to the index position to improve the data writing efficiency of the index linked list.
[0155] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0156] The computer-readable storage medium provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.
[0157] Furthermore, another embodiment of the present specification also provides a computer program product, which implements the following process when executed by a processor:
[0158] Obtaining data to be written, and calculating a quality score of the data to be written;
[0159] Reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics;
[0160] The data to be written is processed into the index linked list according to the index position.
[0161] The computer program product provided in this embodiment first obtains the data to be written and calculates the quality score of the data to be written; secondly, determines the index position for writing the data to be written into the index linked list according to the attribute characteristics of the index linked list, the access characteristics of the data to be written and the quality score; finally, writes the data to be written into the index linked list according to the index position to improve the data writing efficiency of the index linked list.
[0162] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.
[0163] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0164] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0168] Memory may include non-permanent storage in a computer-readable storage medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable storage medium.
[0169] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0172] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A data writing processing method, characterized in that: The method comprises: Obtaining data to be written, and calculating a quality score of the data to be written; Reading the attribute characteristics of the index linked list and the access characteristics of the data to be written, and determining the index position for writing the data to be written into the index linked list according to the quality score, the attribute characteristics and the access characteristics; The data to be written is processed into the index linked list according to the index position.
2. The data writing method according to claim 1, characterized in that: The method further comprises: Calculate the update access heat of the query data according to the index layer of the query data in the index chain list, the quality score of the query data and the query frequency; Based on the updated access heat of the query data, the access heat of the query data is updated in the access heat table.
3. The data writing method according to claim 1, characterized in that: The method further comprises: Calculating data redundancy of the index linked list according to the amount of data in each index layer in the index linked list and the total number of index layers in the index linked list; The index linked list is cleaned up based on the data redundancy.
4. The data writing method according to claim 3, characterized in that: The performing data cleaning processing on the index linked list based on the data redundancy includes: Detecting whether the data redundancy is greater than a preset redundancy threshold; If yes, a cleaning step is calculated according to the amount of data and the total number of index layers, and data is cleaned in the index chain list according to the access popularity of each data in the index chain list and the cleaning step.
5. The data writing method according to claim 4, characterized in that: The cleaning of data in the index linked list according to the access popularity of each data in the index linked list and the cleaning step length includes: According to the priority of each index layer in the index chain table, detecting whether the access heat of the data of each index layer is less than a preset heat threshold; If so, the target data of the target index layer in the index linked list whose access heat is less than the preset heat threshold, or the target data and the target data of at least one index layer after the target index layer are deleted, and an updated index linked list is obtained after the number of data detections reaches the cleaning step.
6. The data writing method according to claim 1, characterized in that: The step of determining, according to the quality score, the attribute feature, and the access feature, an index position for writing the data to be written into the index linked list comprises: Inputting the quality score, the attribute feature and the access feature into an index layer calculation algorithm to perform index layer calculation, and obtaining an index layer identifier for writing the data to be written into the index linked list; After the step of performing the data writing process of the data to be written into the index linked list according to the index position is executed, the method further includes: The data to be written is written into each index layer after the index layer corresponding to the index layer identifier in the index chain list through a scheduled task.
7. The data writing method according to claim 1, characterized in that: The method further comprises: If the query type of the data query is a specific query type, a hash value of the query identifier of the data query is calculated; An index position mapped by the query identifier in the hash table is determined based on the hash value, and query data corresponding to the query identifier is acquired from the mapped index position.
8. The data writing method according to claim 7, characterized in that: The determining, based on the hash value, an index position of the query identifier mapped in the hash table comprises: Performing modulo processing on the table capacity of the hash table according to the hash value to obtain a remainder; The remainder is used as the index position mapped by the query identifier in the hash table.
9. The data writing method according to claim 1, characterized in that: The calculating the quality score of the data to be written includes: If the data classification of the data to be written is text classification, the data to be written is input into a natural language processing algorithm for score calculation to obtain the quality score; If the data classification of the data to be written is image classification, a score calculation is performed on the data to be written using an image scoring algorithm to obtain the quality score.
10. The data writing method according to claim 1, characterized in that: The calculating the quality score of the data to be written includes: Reading text data classified as text classification and image data classified as image classification in the data to be written; Inputting the text data into a natural language processing algorithm to perform scoring calculation to obtain a text quality score, and performing scoring calculation on the image data using an image scoring algorithm to obtain an image quality score; The quality score is calculated according to the text quality score, the image quality score, and a data ratio of the text data to the image data.
Citation Information
Cited By
Data storage method and device
CN119806417A
