Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps that if a current data packet corresponding to current to-be-merged data exceeds the capacity of a first storage space packet, corresponding index merging information is obtained based on a feature value corresponding to a target data packet cached in a current first storage space, and the index merging information is stored in a second storage space; performing space processing on the first storage space, and storing the current to-be-merged data into the current data group in the unoccupied space; updating index merging information based on the characteristic value corresponding to the current data group; sorting the index merging information based on the characteristic value to obtain sorted index merging information; and merging the data corresponding to the same characteristic value in the sorted index merging information to obtain corresponding final merged data. By adopting the method, the data processing efficiency can be improved under the condition of not causing memory overflow.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of big data technology, massive amounts of data have been generated. In order to extract effective information from massive data, massive data are usually processed.

[0003] However, in traditional methods, due to the excessive amount of data, memory overflow is often possible when merging massive data (for example, merging 10,000 data items into 500 items according to preset rules). This will affect the stability of the corresponding system, as well as the efficiency of data processing and the accuracy of the processing results. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of data processing without causing memory overflow in order to address the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Get the current data to be merged;

[0007] If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, the target data group currently cached in the first storage space is obtained, and the corresponding index merging information is obtained based on the feature value corresponding to the target data group, and the index merging information is stored in the second storage space;

[0008] Performing space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged;

[0009] Updating index merging information based on a feature value corresponding to the current data group cached in the first storage space;

[0010] Sorting the index merging information based on the characteristic value to obtain sorted index merging information;

[0011] The data corresponding to the same feature value in the sorted index merging information are merged to obtain the corresponding final merged data.

[0012] In a second aspect, the present application further provides a data processing device, comprising:

[0013] Acquisition module, used to obtain the current data to be merged;

[0014] a first merging module configured to, if a current data group corresponding to the data to be merged exceeds the capacity of the first storage space group, obtain a target data group currently cached in the first storage space, obtain corresponding index merging information based on a feature value corresponding to the target data group, and store the index merging information in the second storage space;

[0015] a processing module, configured to perform space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged;

[0016] An updating module, configured to update index merging information based on a feature value corresponding to a current data group cached in the first storage space;

[0017] A sorting module is used to sort the index merging information based on the feature value to obtain sorted index merging information;

[0018] The second merging module is used to merge the data corresponding to the same feature value in the sorted index merging information to obtain the corresponding final merged data.

[0019] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0020] Get the current data to be merged;

[0021] If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, the target data group currently cached in the first storage space is obtained, and the corresponding index merging information is obtained based on the feature value corresponding to the target data group, and the index merging information is stored in the second storage space;

[0022] Performing space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged;

[0023] Updating index merging information based on a feature value corresponding to the current data group cached in the first storage space;

[0024] Sorting the index merging information based on the characteristic value to obtain sorted index merging information;

[0025] The data corresponding to the same feature value in the sorted index merging information are merged to obtain the corresponding final merged data.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0027] Get the current data to be merged;

[0028] If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, the target data group currently cached in the first storage space is obtained, and the corresponding index merging information is obtained based on the feature value corresponding to the target data group, and the index merging information is stored in the second storage space;

[0029] Performing space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged;

[0030] Updating index merging information based on a feature value corresponding to the current data group cached in the first storage space;

[0031] Sorting the index merging information based on the characteristic value to obtain sorted index merging information;

[0032] The data corresponding to the same feature value in the sorted index merging information are merged to obtain the corresponding final merged data.

[0033] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0034] Get the current data to be merged;

[0035] If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, the target data group currently cached in the first storage space is obtained, and the corresponding index merging information is obtained based on the feature value corresponding to the target data group, and the index merging information is stored in the second storage space;

[0036] Performing space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged;

[0037] Updating index merging information based on a feature value corresponding to the current data group cached in the first storage space;

[0038] Sorting the index merging information based on the characteristic value to obtain sorted index merging information;

[0039] The data corresponding to the same feature value in the sorted index merging information are merged to obtain the corresponding final merged data.

[0040] The above-mentioned data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain the current data to be merged. If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, that is, because a lot of data has been cached in the first storage space, the current data to be merged cannot be cached in the first storage space for subsequent processing, the cached data in the first storage space is stored, and then the first storage space is spatially processed to obtain unoccupied space, and the current data to be merged is stored in the unoccupied space for subsequent processing. In the process of storing the target data group cached in the first storage space, the corresponding index merging information is obtained based on the feature value corresponding to the target data group, and the index merging information is stored in the second storage space. In this way, the results of the processed data in the first storage space can be stored, and then other groups that exceed the capacity of the first storage space group can continue to be processed, thereby reducing the possibility of memory overflow. Afterwards, the index merge information corresponding to all the data to be merged is sorted according to the eigenvalues ​​to obtain the sorted index merge information, and the groups corresponding to the same eigenvalues ​​in the sorted index merge information are merged to obtain the final merge result corresponding to each eigenvalue. Since the index merge information has been sorted, when the group corresponding to the next different eigenvalue is processed, it means that the group corresponding to the previous eigenvalue has been merged, that is, the final merge result corresponding to the previous eigenvalue is obtained. In this way, the occupancy rate of the first storage space can be reduced, the possibility of memory overflow can be reduced, the stability of the system can be improved, and the efficiency of merging massive data can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 1 is a flow chart of a data processing method in one embodiment;

[0043] Figure 2 A schematic diagram of index merging information in one embodiment;

[0044] Figure 3 is a flow chart of a data processing method in another embodiment;

[0045] Figure 4 is a schematic diagram of data to be processed in one embodiment;

[0046] Figure 5This is a schematic diagram corresponding to the target merging result in one embodiment;

[0047] Figure 6 A schematic diagram of an association relationship between merged identification information and data identification information in one embodiment;

[0048] Figure 7 is a flowchart of a data processing method in yet another embodiment;

[0049] Figure 8 is a structural block diagram of a data processing device in one embodiment;

[0050] Figure 9 is a diagram of the internal structure of a computer device in one embodiment;

[0051] Figure 10 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] The data processing method provided in the embodiments of the present application can be applied to a terminal, a server, or a system comprising a terminal and a server, and implemented through interaction between the terminal and the server. If applied to a system comprising a terminal and a server, the terminal communicates with the server via a network. The data storage system can store data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The terminal can include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, projectors, etc. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, etc. The head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In an exemplary embodiment, Figure 1 As shown, a data processing method is provided, which is described by taking the method applied to a server as an example, and includes the following steps 102 to 112. Among them:

[0055] Step 102: Obtain the current data to be merged.

[0056] In this embodiment, it is necessary to merge massive amounts of data. Exemplarily, this embodiment can merge tabular data. Tabular data can be, for example, reports, documents, and other data. Tabular data includes at least one dimension and the dimension values ​​corresponding to each dimension. For example, for a database table, the data in the table includes multiple fields and the field values ​​corresponding to each field. The merge process is used to merge data units in the processed data that meet preset rules. The preset rule can be, for example, that the dimension values ​​corresponding to the target dimension are the same. For example, the merge process can be that at least two data rows with the same dimension values ​​corresponding to the target dimension in the processed data are merged into one data row according to the target dimension. The preset rule can also be, for example, that the dimension value corresponding to the target dimension is within a preset range.

[0057] Because the amount of data to be processed exceeds a preset value (related to the capacity of the first storage space), the first storage space cannot process all of this data at once. Forcing the first storage space to process all of this data at once could cause memory overflow, leading to a series of abnormalities such as system downtime. Therefore, in this embodiment, the data to be processed is processed in batches to reduce the possibility of memory overflow. The data to be processed can be pre-stored in the second storage space and then read into the first storage space for processing.

[0058] Among them, the current data to be merged refers to a piece of data in the data to be processed that is currently being processed. If the data to be processed is a form, the current data to be merged is a data row in the form. Before obtaining the current data to be merged, a large amount of data has been obtained from the data to be processed, and these data have been read into the first storage space and some related processing has been performed. Since it is uncertain whether the processing results corresponding to these data will need to be accessed later, some results related to these data are also temporarily stored in the first storage space cache, thereby compressing the available space of the first storage space. In this embodiment, after the previous data to be merged is read into the first storage space and related processing is performed, the current data to be merged is obtained from the data to be processed, so that the current data to be merged is read into the first storage space and related processing is performed.

[0059] Specifically, in order to divide the various data to be merged that belong to the same category (for example, the dimension values ​​corresponding to the target dimension are the same category) in the data to be processed into the same group and merge them, a corresponding cache space can be allocated to each group in the first storage space. When a data to be merged is obtained, the data group to which the data to be merged corresponds in the first storage space is determined, and the data to be merged is allocated to the corresponding data group to be merged with other data to be merged corresponding to the data group, thereby obtaining a merge result corresponding to each data group. When the number of data groups corresponding to the data to be processed is less than or equal to the capacity of the cacheable groups in the first storage space, the merging process performed on the data to be processed can refer to the above process. In other words, the final merge result of the data to be processed can be obtained without performing spatial processing on the first storage space.

[0060] It's understandable that merging isn't necessarily used in data consolidation scenarios; it can also be applied in other scenarios, such as querying data. For example, if you need to query the sales figures for each material in a particular month, you need to merge the relevant data for that month to summarize the sales figures for each material in that month.

[0061] When the number of data packets corresponding to the data to be processed is greater than the capacity of packets that can be cached in the first storage space, the following steps may be referred to:

[0062] Step 104: If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, obtain the target data group cached in the current first storage space, obtain the corresponding index merging information based on the feature value corresponding to the target data group, and store the index merging information in the second storage space.

[0063] Among them, the number of target data groups currently cached in the first storage space has reached the upper limit of the capacity of the groups that can be cached in the first storage space. If the current data group corresponding to the current data to be merged is not obtained in the first storage space, and there is no unoccupied space in the first storage space, a certain cache space is allocated for the current data group. In this case, in order to process the current data to be merged, it is necessary to perform space processing on the first storage space to free up some memory to cache the current data group. In order to reduce the possibility of errors in the final merging result due to data loss or omission, before clearing the space of the first storage space, it is necessary to pre-store the target data groups currently cached in the first storage space. Exemplarily, in order to store the target data groups cached in the first storage space, corresponding index merging information can be obtained based on the feature value corresponding to the target data group, and the index merging information can be stored in the second storage space.

[0064] The second storage space is different from the first storage space for caching each group. The index merging information is used to record the relationship between each target data group cached in the first storage space and the corresponding feature value. The index merging information can be, for example, Figure 2 The data table shown in FIG. Wherein, the characteristic values ​​correspond to the target data groups one by one. The characteristic values ​​corresponding to the data to be merged corresponding to the target data group are consistent. The characteristic values ​​corresponding to the target data group can be determined based on the characteristic values ​​corresponding to the data to be merged that match the target data group.

[0065] When the amount of data to be processed far exceeds the data capacity that can be cached in the first storage space, the first storage space may need to be spatially processed multiple times. In this case, the index merge information is used to record the relationship between each target data group cached in the first storage space and the corresponding characteristic value before each spatial processing. For example, the upper limit of the group capacity that can be cached in the first storage space is 3, but the data to be processed corresponds to 10 groups. In this case, the space needs to be cleaned up at least 3 times, and the index merge information needs to record the group data cached in the first storage space before each space cleanup to avoid data loss.

[0066] Step 106 , performing space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged.

[0067] The purpose of performing space processing on the first storage space is to free up space in the first storage space to cache the current data group corresponding to the current data to be merged, so as to process the current data to be merged.

[0068] In this embodiment, the current data to be merged may include at least one data unit. If the current data to be merged is multiple data units, these data units are read into the corresponding current data groups in the first storage space after spatial processing, and then the data are merged in the first storage space.

[0069] In one implementation, performing space processing on the first storage space may include: clearing the first storage space.

[0070] In another embodiment, performing space processing on the first storage space may include determining a target space in the first storage space and clearing the target space. The target space may correspond to a cache space occupied by at least one target data group. The target data group cached in the target space may be a group that has not been updated or accessed for a long period of time. Exemplarily, the target space may be determined based on access attributes corresponding to the target data group. The access attributes include at least one of the number of accesses and the time of the most recent access.

[0071] Step 108: Update the index merging information based on the feature value corresponding to the current data group cached in the first storage space.

[0072] For example, if the current data to be merged have all been read into the first storage space and relevant processing has been performed in the corresponding groups, since the index merge information corresponds to all groups corresponding to the data to be processed, this embodiment needs to update the index merge information based on the characteristic values ​​corresponding to the current data groups cached in the first storage space.

[0073] Step 110 : sort the index merging information based on the feature values ​​to obtain sorted index merging information.

[0074] Step 112: Merge the data corresponding to the same feature value in the sorted index merging information to obtain the corresponding final merged data.

[0075] After obtaining the final merged data, the final merged data can be stored in a second storage space. The second storage space is a storage space different from the first storage space. The first storage space belongs to the memory. The second storage space can belong to a storage space other than the first storage space in the memory, or it can belong to the hard disk, depending on the type of database storing the final merged data.

[0076] In some cases, due to the space limitation of the first storage space, it may not be possible to match all the data to be merged to the corresponding data group at one time, so it is necessary to clear the first storage space and continue to write the remaining unprocessed data to be merged to the first storage space for subsequent processing. Before the first storage space is cleared, the target data groups stored in the first storage space will be stored in the index merge information. Due to the uncertainty of the data to be merged, multiple data to be merged belonging to the same characteristic value may be written into the index merge information in different batches. In other cases, due to the limitation of the space occupied by a group in the memory, when there are more data to be merged corresponding to a certain group, the memory may allocate space occupied by multiple groups for the data group. In the above cases, there will be a situation in the index merge information where one characteristic value corresponds to multiple data. For example, when the index merge information is a form, the first row and the tenth row of the form both store data information corresponding to the target data group with a characteristic value of 1. It can be understood that the final merge result needs to merge the data information corresponding to the first row and the data information corresponding to the second row to obtain the final merge result with a characteristic value of 1.

[0077] That is, to obtain the final merge result, the server needs to read each data in the index merge information into the memory again for merging. Considering the limitation of the first storage space capacity, the index merge information needs to be sorted to reduce the possibility of memory overflow.

[0078] For example, after sorting the index merge information, the data corresponding to each eigenvalue is sequentially read into the first storage space for processing. For example, data corresponding to an eigenvalue of 3 is read into the first storage space only after all data corresponding to an eigenvalue of 1 in the index merge information is read into the first storage space. This prevents a situation where data corresponding to an eigenvalue of 1 is read into the first storage space, then data corresponding to an eigenvalue of 3 is read into the first storage space, and then data corresponding to an eigenvalue of 1 is read into the first storage space again.

[0079] That is to say, in this embodiment, after all the data corresponding to a certain target characteristic value are read into the first storage space and the relevant merging processing is performed, the grouping space allocated for the target characteristic value in the first storage space can be cleaned up, thereby reducing the possibility of the memory being full. And because the index merging information has been sorted, the data corresponding to the target characteristic value will not appear later, so the correctness of the final merging result can also be ensured. In addition, because the index merging information has been sorted, when the data in the index merging information is stored in the corresponding grouping space in the memory, the query of the position of the grouping space where the current data is located can be reduced. For example, when the current data is stored in the memory, if the characteristic value corresponding to the current data is the same as that of the previous data, the current data can be stored according to the spatial position of the previous data, without having to query the position of the grouping space corresponding to the current data in the memory every time a data is read, thereby improving the efficiency of data merging.

[0080] In the above data processing method, by obtaining the current data to be merged, if the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, that is, because more data has been cached in the first storage space, the current data to be merged cannot be cached in the first storage space for subsequent processing, the cached data in the first storage space is stored, and then the first storage space is space-processed to obtain unoccupied space, and the current data to be merged is stored in the unoccupied space for subsequent processing. In the process of storing the target data group cached in the first storage space, the corresponding index merging information is obtained based on the feature value corresponding to the target data group, and the index merging information is stored in the second storage space. In this way, the results of the processed data in the first storage space can be stored, and then other groups that exceed the capacity of the first storage space group can continue to be processed, thereby reducing the possibility of memory overflow. Afterwards, the index merge information corresponding to all the data to be merged is sorted according to the eigenvalues ​​to obtain the sorted index merge information, and the groups corresponding to the same eigenvalues ​​in the sorted index merge information are merged to obtain the final merge result corresponding to each eigenvalue. Since the index merge information has been sorted, when the group corresponding to the next different eigenvalue is processed, it means that the group corresponding to the previous eigenvalue has been merged, and the final merge result corresponding to the previous eigenvalue can be stored in the second storage space. In this way, the occupancy rate of the first storage space can be reduced, the possibility of memory overflow can be reduced, the stability of the system can be improved, and the efficiency of merging massive data can be improved.

[0081] In an exemplary embodiment, Figure 3 As shown, the corresponding index merging information is obtained based on the feature value corresponding to the target data group, including steps 302 to 307. Wherein: the data to be processed is taken as an example of a form to be processed, and the data to be merged is taken as an example of a data row to be merged in the form to be processed. The form to be processed includes field values ​​corresponding to multiple fields. Figure 4 The form to be processed is a business record table with multiple fields (for example, PK, sequence number, accounting date, material batch number, unit price, and other dimension fields). Each row of data in the business record table is a data row to be merged. If rows with PK values ​​of 1, 2, and 3 are in one category, and rows with PK values ​​of 4, 5, and 6 are in another category, then rows with PK values ​​of 1, 2, and 3 need to be merged, and rows with PK values ​​of 4, 5, and 6 need to be merged.

[0082] Step 302: Acquire a feature value corresponding to the target data to be merged based on the field value corresponding to the target field in the target data to be merged.

[0083] In this embodiment, the data to be processed is merged according to the target merge rule. The target merge rule can be pre-configured. The target merge rule can include the configured target field and target condition. That is to say, the data to be merged whose field value corresponding to the target field in the data to be processed meets the target condition can be classified into one category for merging. For example, the data rows in the business record table with the same material batch number and the accounting date belonging to the same month need to be merged. The configured target field can include a rule field and a merge field. The rule field is used to group the data to be merged. The merge field is used for summarization. For example, Figure 4 As shown, the rule fields are: accounting date, material batch number, and the merged fields are: unit price amount. Figure 4 For the corresponding expected merge results, please refer to Figure 5 .exist Figure 5 The first row of data is Figure 4 The first three rows of data in the corresponding merge results. It can be determined that the unit price amount is a merge field, and the summary Figure 4 The unit price amount of the first three rows of data (100+200+300=600).

[0084] Among them, the characteristic value is a numerical value obtained by calculating the field value corresponding to the rule field included in the target field. Compared with comparing the data one by one to group the data, this embodiment obtains the characteristic value by calculating the field value corresponding to the rule field in each target data to be merged. The target data to be merged with the same characteristic value are cached in the memory into the same target data group. For example, it is necessary to obtain the summary amount corresponding to different materials on different dates, the corresponding rule fields are "date" and "material model", and the corresponding merge field is "amount". The field value corresponding to the "date" field of each data row and the field value corresponding to the "material model" field are calculated by the preset formula, and the various amounts of the same material on the same date are divided according to the calculation results. In this way, the comparison dimension can be simplified, and the comparison of the value of each field of each data is simplified to the comparison of the characteristic value corresponding to each data, thereby improving the efficiency of grouping.

[0085] Exemplarily, the feature value may be a hash value obtained by performing a hash calculation on the field value corresponding to the target field.

[0086] Step 304: Determine the characteristic value corresponding to the target data group based on the characteristic value corresponding to the target data to be merged.

[0087] It can be understood that the field value corresponding to the target field in the target data to be merged is calculated, and only the target data to be merged with the same calculation result will be read into the same target data group in the memory. Therefore, the characteristic value corresponding to each target data to be merged corresponding to the target data group is consistent. Therefore, the characteristic value corresponding to the target data group can be determined based on the characteristic value corresponding to the target data to be merged. Optionally, the characteristic value corresponding to the target data to be merged can be consistent with the characteristic value corresponding to the target data group. Optionally, the characteristic value corresponding to the target data to be merged can also satisfy a specific relationship with the characteristic value corresponding to the matching target data group. For example, the target data to be merged corresponding to the characteristic value of 1 is stored in the target data group corresponding to the characteristic value of 0, and the target data to be merged corresponding to the characteristic value of 2 is stored in the target data group corresponding to the characteristic value of 1.

[0088] It is understood that steps 302 and 304 can also be executed when the target data to be merged is stored in the memory. That is, after the target data to be merged is obtained, the corresponding target characteristic value is determined based on the target data to be merged. If a target data group corresponding to the target characteristic value is cached in the first storage space, the target data to be merged is stored in the space corresponding to the target data group in the memory. If the target data group corresponding to the target characteristic value is not cached in the first storage space, if there is unoccupied space in the first storage space, space will be allocated to the target data group in the unoccupied space, and the target data to be merged will be stored in the target data group, and the characteristic value corresponding to the target data group will be set to the target characteristic value corresponding to the target data to be merged.

[0089] Step 306: Perform merging processing on at least one to-be-merged data corresponding to the target data group according to the target field to obtain a target merging result corresponding to the target data group.

[0090] The fields of the target merge result are consistent with the target fields corresponding to the target data group.

[0091] Exemplarily, the target field includes a merge field, and at least one to-be-merged data corresponding to the target data group is merged according to the merge field to obtain a target merge result corresponding to the target data group. Specifically, the target merge result can be determined based on the target field. That is, the fields of the target merge result belong to the target field, and other fields that are not target fields are not used as fields of the target merge result. In the case where the field values ​​of the rule fields are the same, the feature values ​​will be the same. The value of the field corresponding to the rule field in the target merge result is the field value of the rule field, and the value of the field corresponding to the merge field is the sum of the field values ​​of the merge field in at least one to-be-merged data.

[0092] In some other embodiments, since the characteristic value is calculated based on the field value corresponding to the rule field according to a preset formula, the field values ​​corresponding to the rule field may not be exactly the same even if the characteristic value is the same. In other words, the field values ​​corresponding to the target field in multiple target data to be merged belonging to the same target data group may not be completely consistent. For example, if the target condition is the same month, the corresponding dates may be different. Therefore, the field value corresponding to the rule field in the target merge result can be determined based on the common characters of the field value corresponding to the rule field in at least one of the data to be merged.

[0093] It is understood that this process can be executed during space processing or when the target data to be merged is stored in the corresponding target data group in the first storage space. That is, when the target data to be merged is stored in the target data group, the target data to be merged and the temporary merge result corresponding to the target data group are merged according to the target field. The temporary merge result is the result of merging the other target data to be merged stored in the target data group before the target data to be merged is stored in the first storage space according to the target field.

[0094] Step 307: Obtain index merging information based on the target merging result and the feature values ​​corresponding to the target data group.

[0095] In order to compress the space occupied by the index merging information stored in the second storage space, the index merging information can be obtained based on the target merging result and the feature value corresponding to the target data group. Figure 5 The merged records in the second storage space can be compressed to obtain index merge information (refer to Figure 2 ).

[0096] Exemplarily, the key information corresponding to the target merge result is stored in the index merge information. The key information may be information that stores the target merge result using JSON or other serialization formats. This can compress the storage space of the index merge information. In some cases, multiple target merge results corresponding to the same category may be stored in the index merge information. In order to obtain the final merge result, these target merge results need to be merged. Therefore, in order to distinguish multiple target merge results belonging to the same category, it is necessary to store the feature values ​​corresponding to each target merge result in the index merge information. The feature value corresponding to the target merge result is the feature value corresponding to the corresponding target data group.

[0097] In this embodiment, by merging at least one data to be merged corresponding to the target data group according to the target field, a target merging result corresponding to the target data group is obtained, and the data cached in the current first storage space can be stored, which is convenient for subsequent space processing of the first storage space to free up space to process the current data to be merged, reducing the possibility of memory overflow, and by obtaining index merging information based on the target merging result and the characteristic value corresponding to the target data group, the storage space corresponding to the index merging information can be compressed, and the remaining target data groups that need to be merged and the corresponding target merging results that have not yet been read into the first storage space for processing can also be determined through the characteristic value.

[0098] In some embodiments, in order to facilitate tracing the merging process, for example, tracing which merged results are based on which merged processes, an association table can be established to store the association between the target merged results and the corresponding target merged data. Figure 3 As shown, after merging at least one to-be-merged data corresponding to the target data group according to the target field to obtain the target merging result corresponding to the target data group, the following steps are included:

[0099] Step 308: configure merge identification information for the target merge result. Different target merge results match different merge identification information.

[0100] Step 310: Obtain at least one piece of data identification information corresponding to at least one piece of data to be merged corresponding to the target data group.

[0101] Step 312: Establish an association relationship between the merge identification information and at least one piece of data identification information, and store the association relationship in the second storage space.

[0102] Step 314: store the target merging result, the merging identification information corresponding to the target merging result, and the feature value corresponding to the target data group into the index merging information.

[0103] like Figure 5As shown, the field value corresponding to the PK field is the merge identification information corresponding to each target merge result. When the target merge results corresponding to the target data group are stored in the index merge information table, merge identification information can be configured for each target merge result. The merge identification information can be used to distinguish different target merge results corresponding to the same feature value. Compared with only storing the association relationship between the feature value and the corresponding target merge result in the merge index table, an additional merge identification information field is set to store the merge identification information corresponding to each target merge result, so that the different target merge results corresponding to the same feature value can be traced back to which update of the index merge information was obtained. In some cases, the merge identification information can also be used to verify the correctness of the data in the index merge information. For example, if the same feature value corresponds to the same merge identification information in the index merge information, it may be because the index merge information is wrong. Relevant deduplication can be performed to improve the correctness of the index merge information, thereby ensuring the correctness of the final merge information.

[0104] In addition, in order to trace which original records (i.e. the original data to be merged) correspond to each target merge result, and to facilitate the expansion of some detailed query functions, an additional association relationship can be established between the target merge result and the original record (refer to Figure 6 Exemplarily, at least one piece of data identification information corresponding to at least one piece of data to be merged corresponding to the target data group is obtained. Data identification information is information used to uniquely identify the data to be merged. Different pieces of data to be merged correspond to different pieces of data identification information, and therefore, the corresponding piece of data to be merged can be queried based on the data identification information. The data identification information corresponding to at least one piece of data to be merged corresponding to the same target data group is associated with the merge identification information corresponding to the target merge result corresponding to the target data group, thereby obtaining an association relationship between each target merge result and each piece of original data to be merged.

[0105] In addition to being used for traceability, recording the relationship between each target merge result and each original data to be merged can also be used for subsequent verification to reduce the situation where data to be merged that correspond to the same feature value but have different field values ​​corresponding to the rule field in the target field are grouped together for merging. In other words, when the data in the index merge information is subsequently read into memory for a second merge, the association relationship can be used to obtain the various data to be merged corresponding to the target merge result, and it can be determined whether the field values ​​corresponding to the target fields in each data to be merged are consistent, or whether the target characters of the field values ​​corresponding to the target fields in each data to be merged are consistent. For example, for the target field "date", the target characters can be "year" and "month", that is, to determine whether the year and month of the field values ​​corresponding to the "date" field in each data to be merged are consistent. If they are inconsistent, they cannot be grouped together for the final merge. This further improves the accuracy of the final merge result.

[0106] That is, querying the index merge table can obtain different target merge results corresponding to the same feature value, and querying the association relationship can obtain each to-be-merged data corresponding to each target merge result.

[0107] In this embodiment, by establishing a relationship between the target merge result and the original data to be merged, it is possible to easily trace the source of the final merge result, and by indexing the merge information and the association relationship, hierarchical query can be performed to improve the efficiency of the query.

[0108] After the above steps, all the data to be merged in the data to be processed have been read into the memory to complete the preliminary merge. If there is no target merge result with the same eigenvalue in the index merge information, then each target merge result in the index merge information is the final merge result. Exemplarily, each key information in the index merge information can be deserialized into a target merge result, and the final merge result is obtained based on each target merge result. The final merge result can be, for example, a merge form. If there is a target merge result with the same eigenvalue in the index merge information, it means that during the preliminary merge process, due to the excessive amount of data to be processed, which far exceeds the grouping capacity that the memory can cache, the memory space has been processed more than once, and the corresponding index merge information has been updated more than once, so the target merge results with the same eigenvalue may be stored in the index merge information during different update processes. In this case, it is necessary to perform a secondary merge on each target merge result corresponding to the index merge information, merge the target merge results corresponding to the same eigenvalue, and obtain the final merge result corresponding to each eigenvalue. For details, please refer to the following content.

[0109] In an exemplary embodiment, Figure 3 As shown, merging the data corresponding to the same feature value in the sorted index merging information includes the following steps:

[0110] Step 316: Obtain at least one target merging result corresponding to the same feature value from the sorted index merging information.

[0111] Step 318: If at least one target field combination corresponding to each target merge result is not exactly the same, then at least one target merge result is grouped according to the target field combination to obtain at least one merged group, and each target merge result in each merged group is merged to obtain a final merged result corresponding to the same feature value.

[0112] The target field combination corresponding to each target merge result in each merge group is the same. A target field combination refers to the set of field values ​​corresponding to the target field.

[0113] In this embodiment, by sorting the index merge information, the possibility of memory overflow during the secondary merge can be avoided and the efficiency of the secondary merge can be improved. Furthermore, by verifying at least one target field combination corresponding to at least one target merge result corresponding to the same feature value, and excluding incompletely consistent target field combinations from the merge, the accuracy of the final merge result can be further improved.

[0114] Specifically, the steps after obtaining at least one target merge result corresponding to the same feature value from the sorted index merge information include: obtaining a first target merge result from at least one target merge result corresponding to the first same feature value; caching a first merge cache group corresponding to the first target merge result in a first storage space, and storing the first target merge result in the first merge cache group; sequentially obtaining a second target merge result from the sorted index merge information; if the target field combination corresponding to the second target merge result is inconsistent with the target field combination corresponding to the first target merge result, caching a second merge cache group corresponding to the second target merge result in the first storage space, and storing the second target merge result in the second merge cache group. group; if the target field combination corresponding to the second target merge result is consistent with the target field combination corresponding to the first target merge result, the second target merge result is stored in the first merge cache group; continue to obtain the third target merge result until at least one target merge result corresponding to the same feature value is stored in the corresponding merge cache group; merge the target merge results in the merge cache group to obtain the final merge result corresponding to the first same feature value, and store the final merge result corresponding to the first same feature value in the second storage space; continue to obtain the first target merge result from at least one target merge result corresponding to the second same feature value until the target merge results corresponding to each feature value in the index merge information are merged.

[0115] In this embodiment, by reading each target merge result in the index merge information into the memory, each target merge result corresponding to each eigenvalue can be processed centrally, so that when processing the target merge result corresponding to the next different eigenvalue, it can be indicated that all the data to be merged corresponding to the previous eigenvalue have been merged, so the final merge result corresponding to the previous eigenvalue can be transferred to the second storage space, so that some space can be cleared to process the merging of the target merge result corresponding to the next eigenvalue, thereby reducing the memory occupancy.

[0116] In a specific embodiment, Figure 7As shown in the example, the summary data tables corresponding to various business documents are being merged. Due to the large amount of data in the summary data tables, the number of groups corresponding to the summary data tables exceeds the upper limit of the memory cache capacity. For example, the number of groups corresponding to the summary data tables is 100, while the upper limit of the memory cache capacity is 20. Therefore, some data processing is necessary to reduce the possibility of memory overflow.

[0117] In summary, the data processing process of this embodiment is divided into two parts. The following is a description of the first merging process and the second merging process.

[0118] During the first merge, the server performs a preliminary merge on each data row in the summary data table, and then processes each data row in the summary data table in turn.

[0119] Specifically, the current data to be merged is obtained, wherein the current data to be merged may be data in a unified data format obtained by processing various business document data.

[0120] Based on the field value corresponding to the target field in the current data to be merged, the characteristic value corresponding to the current data to be merged is obtained, including: performing a hash calculation on the field value corresponding to the target field in the current data to be merged, and determining that the calculation result is the characteristic value corresponding to the current data to be merged.

[0121] If the current data group corresponding to the data to be merged is already cached in memory, that is, if a current data group with the same characteristic value as the data to be merged is already cached in memory, the data to be merged is read into the cached current data group. Accordingly, the server can merge the temporary merge result corresponding to the current data group with the data to be merged, updating the temporary merge result. The server then proceeds to obtain the next piece of data to be merged from the summary data table.

[0122] If the current data group corresponding to the current data to be merged is not cached in the memory, it can be determined whether the current data group can still be cached in the memory. If so, space is allocated in the memory for the current data group. Then, the current data to be merged can be stored in the current data group. The steps for outputting the final merged result data can be found in the above description and will not be repeated here. If not, that is, if space is allocated in the memory for the current data group, the number of data groups cached in the memory will exceed the upper limit of the group capacity corresponding to the memory (that is, the number of merged outputs exceeds the upper limit of the memory), then the memory space needs to be cleared. For example, the data of 20 groups cached in the memory are cleared to make room for processing the data to be merged corresponding to the 21st group.

[0123] To avoid data loss, before performing space processing on the memory, it is necessary to obtain the target data group currently cached in the first storage space, obtain the corresponding index merge information based on the feature value corresponding to the target data group, and store the index merge information on the hard disk. In other words, the group records are cached using a group index table. The group index table contains the hash values ​​of all the field values ​​involved in the grouping, as well as the complete record of the merged group.

[0124] During the first merge, after clearing the data cached in memory for groups 1-20, the memory continues to cache subsequent groups. Due to data uncertainty, during the second caching process, the memory may cache data for groups 21-30, with the remaining 10 groups cached in memory corresponding to 10 of the groups 1-20. Therefore, if the current data to be merged corresponds to group 31, space must be cleared again to cache the data to be merged corresponding to group 31 and update the index merge information. This shows that the index merge information may store multiple target merge results corresponding to the same feature value. For example, the first 20 rows of the index merge information store the target merge results for groups 1-20, the 21st row stores the target merge result for group 21, and the 22nd row may store the target merge result for group 19. The target merge result for group 19 is different from the target merge result for group 19 in the first 20 rows, requiring a subsequent second merge.

[0125] After all data in the summary data table are read into memory and corresponding target merge results exist in the index merge information, that is, after all data have been merged for the first time, a second merge can be performed based on the index merge information.

[0126] During the second merging process, the characteristic values ​​corresponding to the target data group are determined based on the characteristic values ​​corresponding to the target data to be merged, so that the target merging results corresponding to the same characteristic values ​​can be sorted together.

[0127] Similarly, the target merge results are obtained from the index merge information in sequence. That is, the first merge data sorted by hash value is obtained from the group index table to perform a secondary merge. Specifically, the first target merge result is obtained from at least one target merge result corresponding to the first identical feature value. The first merge cache group corresponding to the first target merge result is cached in the first storage space, and the first target merge result is stored in the first merge cache group. The second target merge result in the sorted index merge information is obtained in sequence. If the target field combination corresponding to the second target merge result is inconsistent with the target field combination corresponding to the first target merge result, the second merge cache group corresponding to the second target merge result is cached in the first storage space, and the second target merge result is stored in the second merge cache group. If the target field combination corresponding to the second target merge result is consistent with the target field combination corresponding to the first target merge result, the second target merge result is stored in the first merge cache group. Continue to obtain the third target merge result until at least one target merge result corresponding to the same feature value is stored in the corresponding merge cache group. The target merge results in the merge cache group are merged to obtain a final merge result corresponding to the first identical feature value, and the final merge result corresponding to the first identical feature value is stored in the second storage space. The first target merge result is obtained from at least one target merge result corresponding to the second identical feature value, until all target merge results corresponding to each feature value in the index merge information are merged. In this way, the final merge result corresponding to the summary data table is obtained.

[0128] In this embodiment, multiple threads or service nodes can be enabled to perform a merge concurrently and out of order, and the merge results are uniformly stored in the index merge information to maximize the horizontal scalability of the group aggregation. During the second merge process, multiple threads or multiple service nodes can also be enabled to handle the secondary merges corresponding to different hash values ​​respectively. In this way, whether it is the first merge or the second merge, the upper limit of memory usage is controllable, thereby reducing the possibility of memory overflow. In addition, since the present application is based on characteristic values ​​to determine the data groups corresponding to each data to be merged, it can support the unification of multiple business document records generated in any batches and at any multiple time points into a large merge task for merging, thereby solving the problem that document records generated across batches and across time points cannot be merged together. In other words, the first merge process and the second merge process can be executed discontinuously. For example, if a task is to merge a month's business data, then you can perform the first merge process on the data of the first 10 days on the 10th day and store the result in the index merge information. Then, on the 20th day, perform the first merge process on the data of the middle 10 days. Finally, on the last day, perform the first merge process on the data of the remaining 10 days and perform the second merge process on the updated index merge information. In this way, data merging across time points and batches can be achieved.

[0129] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0130] Based on the same inventive concept, the present application also provides a data processing device for implementing the aforementioned data processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0131] In an exemplary embodiment, Figure 8As shown, a data processing device 800 is provided, comprising: an acquisition module 801, a first merging module 802, a processing module 803, an updating module 804, a sorting module 805 and a second merging module 806, wherein:

[0132] The acquisition module 801 is used to acquire the current data to be merged.

[0133] The first merging module 802 is used to obtain the target data group cached in the current first storage space if the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, obtain the corresponding index merging information based on the feature value corresponding to the target data group, and store the index merging information in the second storage space.

[0134] The processing module 803 is configured to perform space processing on the first storage space, and store the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged.

[0135] The updating module 804 is configured to update the index merging information based on the characteristic value corresponding to the current data group cached in the first storage space.

[0136] The sorting module 805 is configured to sort the index merging information based on the feature values ​​to obtain sorted index merging information.

[0137] The second merging module 806 is configured to merge the data corresponding to the same feature value in the sorted index merging information to obtain the corresponding final merged data.

[0138] In some embodiments, the target data group corresponds to at least one target data to be merged. In terms of obtaining corresponding index merging information based on the characteristic value corresponding to the target data group, the first merging module 802 is further used to: obtain the characteristic value corresponding to the target data to be merged based on the field value corresponding to the target field in the target data to be merged; determine the characteristic value corresponding to the target data group based on the characteristic value corresponding to the target data to be merged; merge the at least one data to be merged corresponding to the target data group according to the target field to obtain the target merging result corresponding to the target data group; and obtain index merging information based on the target merging result and the characteristic value corresponding to the target data group.

[0139] In some embodiments, after merging at least one data to be merged corresponding to the target data group according to the target field to obtain the target merging result corresponding to the target data group, the first merging module 802 is also used to: configure merge identification information for the target merging result, different target merging results match different merge identification information; obtain at least one data identification information corresponding to at least one data to be merged corresponding to the target data group; establish an association relationship between the merge identification information and the at least one data identification information, and store the association relationship in the second storage space; store the target merging result, the merge identification information corresponding to the target merging result, and the characteristic value corresponding to the target data group in the index merging information.

[0140] In some embodiments, in terms of merging data corresponding to the same characteristic value in the sorted index merge information, the second merge module 806 is also used to: obtain at least one target merge result corresponding to the same characteristic value from the sorted index merge information; if the at least one target field combination corresponding to each target merge result is not exactly the same, then group the at least one target merge result according to the target field combination to obtain at least one merged group, and merge each target merge result in each merged group to obtain a final merge result corresponding to the same characteristic value; wherein, the target field combination corresponding to each target merge result in each merged group is the same.

[0141] In some embodiments, after obtaining at least one target merge result corresponding to the same feature value from the sorted index merge information, the second merge module 806 is further used to: obtain a first target merge result from at least one target merge result corresponding to the first same feature value; cache a first merge cache group corresponding to the first target merge result in the first storage space, and store the first target merge result in the first merge cache group; sequentially obtain the second target merge result in the sorted index merge information; if the target field combination corresponding to the second target merge result is inconsistent with the target field combination corresponding to the first target merge result, cache a second merge cache group corresponding to the second target merge result in the first storage space, and store the second target merge result in the second storage space. merge cache group; if the target field combination corresponding to the second target merge result is consistent with the target field combination corresponding to the first target merge result, store the second target merge result in the first merge cache group; continue to obtain the third target merge result until at least one target merge result corresponding to the same feature value is stored in the corresponding merge cache group; merge the target merge results in the merge cache group to obtain the final merge result corresponding to the first same feature value, and store the final merge result corresponding to the first same feature value in the second storage space; continue to obtain the first target merge result from at least one target merge result corresponding to the second same feature value until the target merge results corresponding to each feature value in the index merge information are merged.

[0142] In some embodiments, in performing space processing on the first storage space and storing the current data to be merged in the current data group in the unoccupied space, the processing module 803 is also used to: obtain the storage position corresponding to each target data group in the first storage space; determine the space to be processed in the first storage space based on the storage position; clear the space to be processed to obtain unoccupied space; and store the current data to be merged in the current data group in the unoccupied space.

[0143] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0144] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and a first storage space memory. The non-volatile storage medium stores an operating system, a computer program and a database. The first storage space memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as each data to be merged, index merging information, and association relationship tables. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0145] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and a first storage space memory. The non-volatile storage medium stores an operating system and a computer program. The first storage space memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near field communication (NFC), or other technologies. When executed by the processor, the computer program implements a data processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0146] Those skilled in the art will understand that Figure 9 or Figure 10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0147] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0149] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0152] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Get the current data to be merged; If the current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, then obtain the target data group currently cached in the first storage space, obtain corresponding index merging information based on the feature value corresponding to the target data group, and store the index merging information in the second storage space; Performing space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged; updating the index merging information based on the characteristic value corresponding to the current data group cached in the first storage space; sorting the index merging information based on the characteristic value to obtain sorted index merging information; The data corresponding to the same characteristic value in the sorted index merging information are merged to obtain the corresponding final merged data.

2. The method according to claim 1, characterized in that The target data group corresponds to at least one target data to be merged, and obtaining corresponding index merging information based on a feature value corresponding to the target data group includes: Based on the field value corresponding to the target field in the target data to be merged, obtaining the feature value corresponding to the target data to be merged; Determining the characteristic value corresponding to the target data group based on the characteristic value corresponding to the target data to be merged; Performing merging processing on at least one to-be-merged data corresponding to the target data group according to the target field to obtain a target merging result corresponding to the target data group; The index merging information is obtained based on the target merging result and the feature value corresponding to the target data group.

3. The method according to claim 2, characterized in that After performing merging processing on the at least one to-be-merged data corresponding to the target data group according to the target field to obtain a target merging result corresponding to the target data group, the method includes: Configuring merge identification information for the target merge result, where different target merge results match different merge identification information; Acquire at least one piece of data identification information corresponding to at least one piece of data to be merged corresponding to the target data group; Establishing an association relationship between the merge identification information and the at least one data identification information, and storing the association relationship in the second storage space; The obtaining the index merging information based on the target merging result and the feature value corresponding to the target data group includes: The target merging result, the merging identification information corresponding to the target merging result, and the feature value corresponding to the target data group are stored in the index merging information.

4. The method according to claim 1, wherein The merging of data corresponding to the same characteristic values ​​in the sorted index merging information includes: Obtaining at least one target merging result corresponding to the same feature value from the sorted index merging information; If at least one target field combination corresponding to each target merging result is not completely the same, the at least one target merging result is grouped according to the target field combination to obtain at least one merged group, and each target merging result in each merged group is merged to obtain a final merged result corresponding to the same feature value; Wherein, the target field combination corresponding to each target merging result in each of the merging groups is the same.

5. The method according to claim 4, characterized in that After obtaining the at least one target merging result corresponding to the same feature value from the sorted index merging information, the method further includes: obtaining a first target merging result from the at least one target merging result corresponding to the first same feature value; caching a first merged cache group corresponding to the first target merged result in a first storage space, and storing the first target merged result in the first merged cache group; Sequentially obtain the second target merge result in the sorted index merge information; If the target field combination corresponding to the second target merge result is inconsistent with the target field combination corresponding to the first target merge result, cache a second merge cache group corresponding to the second target merge result in the first storage space, and store the second target merge result in the second merge cache group; If the target field combination corresponding to the second target merge result is consistent with the target field combination corresponding to the first target merge result, storing the second target merge result in the first merge cache group; Continue to obtain the third target merge result until at least one target merge result corresponding to the same feature value is stored in the corresponding merge cache group; Merging the target merged results in the merged cache group to obtain a final merged result corresponding to the first identical feature value, and storing the final merged result corresponding to the first identical feature value in the second storage space; Continue to obtain the first target merging result from at least one target merging result corresponding to the second identical eigenvalue until all target merging results corresponding to each eigenvalue in the index merging information are merged.

6. The method according to claim 1, wherein The performing space processing on the first storage space to store the current data to be merged into the current data group in the unoccupied space includes: Obtaining a storage location corresponding to each target data group in the first storage space; determining a space to be processed in the first storage space based on the storage location; Clearing the space to be processed to obtain unoccupied space; The current data to be merged is stored in the current data group in the unoccupied space.

7. A data processing device, characterized in that: The device comprises: Acquisition module, used to obtain the current data to be merged; a first merging module configured to, if a current data group corresponding to the current data to be merged exceeds the capacity of the first storage space group, obtain a target data group currently cached in the first storage space, obtain corresponding index merging information based on a feature value corresponding to the target data group, and store the index merging information in the second storage space; a processing module, configured to perform space processing on the first storage space, storing the current data to be merged into the current data group in the unoccupied space, so as to process the current data to be merged; an updating module, configured to update the index merging information based on a feature value corresponding to the current data group cached in the first storage space; a sorting module, configured to sort the index merging information based on the feature values ​​to obtain sorted index merging information; The second merging module is used to merge the data corresponding to the same feature value in the sorted index merging information to obtain the corresponding final merged data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images