A high performance computing data processing system

Abstract

The application relates to the technical field of data processing, in particular to a high-performance computing data processing system. The system comprises a collection module, a filtering module and a storage module. The application can reduce the influence on the subsequent data processing process, improve the data quality and processing efficiency by deleting the data with a running time of 0 and the data containing missing values through real-time monitoring of input log data. Different storage methods are selected according to the relationship between the newly-added data and the stock data. The newly-added data is directly stored by appending when the newly-added data has no influence on the stock data, so that the computing processing speed is improved, the real-time performance and processing efficiency are increased. The associated data is determined by comparing the newly-added data with the historical data, and the relevant data is stored in the form of complex objects by using a preprocessing method, so that the data history information is retained, the data processing efficiency and performance are improved, the data quality and consistency are guaranteed, and the overall performance and stability of the system are optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a high-performance computing data processing system. BACKGROUND

[0002] High-performance computing refers to the field of using powerful computer systems and high-speed networks to perform large-scale data processing and complex computing tasks. High-performance computing systems are usually composed of multiple processors, large amounts of memory and high-speed storage devices to process large amounts of data and perform complex computing tasks simultaneously. This technology plays an important role in scientific research, engineering design, weather forecasting, financial modeling, etc. It is usually used to handle tasks that require a large amount of computing resources and storage space, such as climate simulation, genomics research, fluid dynamics simulation, etc. These systems can speed up computing, improve data processing efficiency, and help researchers and engineers solve complex problems.

[0003] Chinese patent publication No. CN105302915A discloses a high-performance data processing system based on memory computing. The technical point is to select a data table, automatically extract related fields of the data table, extract related data from the data source according to the metadata defined by the data acquisition interface definition module, store the data in the form of bytecode in the memory through dictionary method, shaping encoding and compression algorithm, and extract records in statistics as needed. Therefore, in the existing high-performance computing data processing technology, the dependency relationship between data cannot be managed, thus the data consistency cannot be guaranteed, and it is difficult to load all input data into memory, resulting in low access performance. SUMMARY

[0004] Therefore, the present application provides a high-performance computing data processing system to overcome the problem that the dependency relationship between data cannot be managed in the prior art, thus the data consistency cannot be guaranteed, and the storage efficiency of the system is low.

[0005] To achieve the above purpose, the present application provides a high-performance computing data processing system, comprising,

[0006] The acquisition module is used to acquire a plurality of input order transaction log sets;

[0007] The filtering module is used to delete each order transaction log set respectively to obtain the to-be-stored data respectively;

[0008] For any order transaction log set, the data with a running time of 0 is deleted, and the data with missing values in the feature data segment is deleted to obtain the to-be-stored data;

[0009] A storage module, which has a plurality of historical data segments, historical update data corresponding to each of the historical data segments and storage nodes, is used to segment the to-be-stored data set to obtain to-be-stored data segments, and each of the to-be-stored data segments is stored on a corresponding storage node, and the storage module comprises,

[0010] An analysis unit is used to analyze any of the to-be-stored data segments, and if the to-be-stored data segment is new data, it is analyzed whether the data has an impact on the inventory data.

[0011] A processing unit is used to make new modifications and delete modifications to the associated data.

[0012] When the analysis unit determines that the to-be-stored data segment has an impact on the associated data, the processing unit makes new modifications to the associated data, and when the analysis unit determines that the to-be-stored data segment has no impact on the associated data, the processing unit makes delete modifications to the associated data.

[0013] Further, the analysis unit analyzes any of the to-be-stored data segments, obtains a historical data segment consistent with a real-time flow path of the to-be-stored data segment and a corresponding storage node, matches the to-be-stored data segment with the corresponding historical data segment,

[0014] If the to-be-stored data segment and the historical data segment do not have matching data, it is determined that the to-be-stored data segment is new data, and the analysis unit determines whether the new data has an impact on the memory amount of the inventory data according to the real-time memory amount.

[0015] If the to-be-stored data segment and the historical data segment have matching data, the matching data is obtained as associated data, and the analysis unit determines whether the to-be-stored data segment has an impact on the content of the inventory data according to the associated data.

[0016] Further, when the analysis unit determines that the to-be-stored data segment and the historical data segment have matching data, the matching data is obtained as associated data, the analysis unit determines whether the to-be-stored data segment has an impact on the content of the inventory data according to the associated data, compares a standard similarity with a real-time similarity,

[0017] If the real-time similarity is less than the standard similarity, the analysis unit determines that the to-be-stored data segment has an impact on the content of the associated data.

[0018] If the real-time similarity is greater than or equal to the standard similarity, the analysis unit determines that the to-be-stored data segment has no impact on the associated data, indicating repetition, and the processing unit makes delete modifications to the associated data.

[0019] Further, the analysis unit determines whether the to-be-stored data segment is new data or has an impact on the content of the associated data, and obtains a real-time memory usage of the to-be-stored data segment, and compares the real-time memory usage with a standard memory usage,

[0020] If the real-time memory usage is less than or equal to the standard memory usage, the analysis unit determines that the new data has no impact on the memory usage of the inventory data or the to-be-stored data segment has no impact on the memory usage of the associated data.

[0021] If the real-time memory usage is greater than the standard memory usage, the analysis unit determines that the new data has an impact on the memory usage of the inventory data or the to-be-stored data segment has an impact on the memory usage of the associated data.

[0022] Further, the storage module includes an appending unit, and when the analysis unit determines that the new data has no impact on the memory usage of the inventory data, the appending unit directly appends the to-be-stored data segment.

[0023] Further, when the analysis unit determines that the to-be-stored data segment has an impact on the associated data, the processing unit performs new modification on the associated data, including,

[0024] modifying a historical end version number corresponding to the associated data to a current version number;

[0025] storing the to-be-stored data segment as an increase-modification field on a corresponding storage node, and the end version number and the start version number of the increase-modification field are both the current version number.

[0026] Further, when the analysis unit determines that the to-be-stored data segment has no impact on the associated data, the processing unit performs deletion modification on the associated data, including,

[0027] modifying a historical end version number corresponding to the associated data to a current version number;

[0028] deleting the to-be-stored data segment.

[0029] Further, when the analysis unit determines that the new data has an impact on the memory usage of the inventory data or the to-be-stored data segment has an impact on the memory usage of the associated data, the to-be-stored data segment is scheduled to another storage node, and an associated key mark is performed on the corresponding inventory data.

[0030] Further, the filtering module deletes data with a running time of 0, including,

[0031] acquiring the timestamp of each data segment in any of the order transaction logs, for any data segment,

[0032] if the running time is 0, deleting the corresponding data segment;

[0033] if the running time is not 0, recording the corresponding data segment as a feature data segment.

[0034] Further, the filtering module deletes data with missing values in the feature data segment, comprising,

[0035] for any of the feature data segments, acquiring the cumulative character length of the standard character segment as the real-time character length, calculating the real-time missing degree according to the real-time character length, and judging the real-time missing degree according to the standard missing degree,

[0036] if the real-time missing degree is less than the standard missing degree, not deleting the feature data segment;

[0037] if the real-time missing degree is greater than or equal to the standard missing degree, judging that the data feature has missing values, and deleting the feature data segment.

[0038] Compared with the prior art, the beneficial effects of the present application are that by monitoring the input log data in real time, the data with a running time of 0 and the data containing missing values are deleted first, which can reduce the impact on the subsequent data processing process, improve the data quality and processing efficiency, and by selecting different storage methods according to the relationship between the new data and the inventory data, the system directly appends the new data for storage when the new data has no impact on the inventory data, improves the calculation processing speed, improves the real-time performance and processing efficiency, by comparing the new data with the historical data, determines the related data, and uses the preprocessing method to organize the related data into complex objects for storage, to retain the data history information, ensure the consistency of the data, and at the same time avoid cross-node access to data, reduce network interaction, and improve the data access performance. The system combines real-time monitoring, data cleaning, storage optimization and other technical means, aims to improve the data processing efficiency and performance, and through different processing methods for different types of data, the system can better cope with complex data processing requirements, while ensuring the quality and consistency of the data, to optimize the overall performance and stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is the connection diagram of the high-performance computing data processing system of the embodiment of the present application;

[0040] Figure 2 is the logic judgment diagram of the appending unit of the embodiment of the present application for directly appending the storage of the data segment to be stored;

[0041] Figure 3The logic decision diagram for the processing unit of the embodiment of the present application to newly add, modify and delete the associated data;

[0042] Figure 4 The logic decision diagram for the embodiment of the present application to obtain the data to be stored. DETAILED DESCRIPTION

[0043] In order to make the objects and advantages of the present application clearer, the present application will be further described below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0044] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood by those skilled in the art that the embodiments are only used to explain the technical principles of the present application and are not used to limit the protection scope of the present application.

[0045] It should be noted that, in the description of the present application, the terms of direction or position relationship such as "upper", "lower", "left", "right", "inner", "outer" and the like are based on the direction or position relationship shown in the drawings, which is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0046] In addition, it should also be noted that, in the description of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0047] Please refer to Figure 1 As shown in the figure, it is a connection diagram of the high-performance computing data processing system of the embodiment of the present application. The present application provides a high-performance computing data processing system, comprising,

[0048] The acquisition module is used to acquire a plurality of input order transaction log sets;

[0049] The filtering module is used to delete each order transaction log set respectively to obtain the data to be stored respectively;

[0050] Among them, for any order transaction log set, the data with a running time of 0 is deleted, and the data with missing values in the feature data segment is deleted to obtain the data to be stored;

[0051] A storage module has a plurality of historical data segments, historical update data corresponding to each of the historical data segments, and storage nodes, which are used to segment the to-be-stored data set to obtain to-be-stored data segments, and to store each of the to-be-stored data segments on a corresponding storage node, the storage module comprising,

[0052] An analysis unit analyzes any of the to-be-stored data segments, and if the to-be-stored data segment is new data, analyzes whether the to-be-stored data segment has an impact on inventory data;

[0053] A processing unit is used to make new modifications and delete modifications to the associated data;

[0054] When the analysis unit determines that the to-be-stored data segment has an impact on the associated data, the processing unit makes new modifications to the associated data, and when the analysis unit determines that the to-be-stored data segment has no impact on the associated data, the processing unit makes delete modifications to the associated data.

[0055] By monitoring the input log data in real time, first deleting data with a running time of 0 and data containing missing values, the influence on the subsequent data processing process can be reduced, the data quality and processing efficiency can be improved, and by selecting different storage methods according to the relationship between the new data and the inventory data, when the new data has no impact on the inventory data, the system directly appends the new data for storage, improves the calculation processing speed, and improves the real-time performance and processing efficiency. By comparing the new data with the historical data, the associated data is determined, and a preprocessing method is used to organize the related data into complex objects for storage, so as to retain the historical information of the data, ensure the consistency of the data, and at the same time avoid cross-node access to the data, reduce network interaction, and improve the data access performance. The system combines real-time monitoring, data cleaning, storage optimization and other technical means to improve the data processing efficiency and performance, and through different processing methods for different types of data, the system can better cope with complex data processing requirements, while ensuring the quality and consistency of the data, to optimize the overall performance and stability of the system.

[0056] Specifically, the analysis unit analyzes any of the to-be-stored data segments, obtains a historical data segment consistent with a real-time flow path of the to-be-stored data segment and a corresponding storage node, and matches the to-be-stored data segment with the corresponding historical data segment,

[0057] If there is no matching data between the to-be-stored data segment and the historical data segment, it is determined that the to-be-stored data segment is new data, and the analysis unit determines whether the new data has an impact on the memory size of the inventory data according to the real-time memory size;

[0058] If the to-be-stored data segment matches the historical data segment, the matching data is recorded as the associated data, and the analysis unit determines whether the to-be-stored data segment has an impact on the content of the inventory data according to the associated data.

[0059] Specifically, when the analysis unit determines that the to-be-stored data segment matches the historical data segment, the matching data is recorded as the associated data, and the analysis unit determines whether the to-be-stored data segment has an impact on the content of the inventory data according to the associated data, calculates the real-time similarity between the associated data and the to-be-stored data segment, and compares the standard similarity with the real-time similarity.

[0060] If the real-time similarity is less than the standard similarity, the analysis unit determines that the to-be-stored data segment has an impact on the content of the associated data.

[0061] If the real-time similarity is greater than or equal to the standard similarity, the analysis unit determines that the to-be-stored data segment has no impact on the associated data, indicating that the data is repeated, and the processing unit deletes or modifies the associated data.

[0062] Specifically, when the analysis unit determines that the to-be-stored data segment is new data or the to-be-stored data segment has an impact on the content of the associated data, the analysis unit obtains the real-time memory usage of the to-be-stored data segment, compares the standard memory usage with the real-time memory usage, and determines whether the to-be-stored data segment has an impact on the memory usage of the associated data.

[0063] If the real-time memory usage is less than or equal to the standard memory usage, the analysis unit determines that the new data has no impact on the memory usage of the inventory data or the to-be-stored data segment has no impact on the memory usage of the associated data.

[0064] If the real-time memory usage is greater than the standard memory usage, the analysis unit determines that the new data has an impact on the memory usage of the inventory data or the to-be-stored data segment has an impact on the memory usage of the associated data.

[0065] Referring to FIG. 1, Figure 2 FIG. 1 is a logical determination diagram of the additional storage of the to-be-stored data segment by the additional unit according to an embodiment of the present application;

[0066] Specifically, the storage module includes an additional unit, and when the analysis unit determines that the new data has no impact on the memory usage of the inventory data, the additional unit directly stores the to-be-stored data segment.

[0067] Referring to FIG. 1, Figure 3 FIG. 1 is a logical determination diagram of the additional storage of the to-be-stored data segment by the additional unit according to an embodiment of the present application;

[0068] Specifically, when the analysis unit determines that the to-be-stored data segment has an impact on the associated data, the processing unit performs an addition modification on the associated data, including,

[0069] modifying the historical end version number corresponding to the associated data to the current version number;

[0070] storing the to-be-stored data segment as an addition modification field on the corresponding storage node, and the end version number and the start version number of the addition modification field are both the current version number.

[0071] Specifically, when the analysis unit determines that the to-be-stored data segment has no impact on the associated data, the processing unit performs a deletion modification on the associated data, including,

[0072] modifying the historical end version number corresponding to the associated data to the current version number;

[0073] deleting the to-be-stored data segment.

[0074] Specifically, when the analysis unit determines that the to-be-stored data segment has an impact on the memory amount of the inventory data or the memory amount of the associated data, the to-be-stored data segment is scheduled to another storage node, and the associated key marker is marked with the corresponding inventory data.

[0075] Referring to Figure 4 , which is a logical determination diagram of the to-be-stored data obtained by the embodiment of the present application;

[0076] Specifically, the filtering module deletes the data with a running time of 0, including,

[0077] acquiring the time stamp of each data segment in any of the order transaction logs, and for any data segment,

[0078] if the running time is 0, the corresponding data segment is deleted;

[0079] if the running time is not 0, the corresponding data segment is marked as a feature data segment.

[0080] Specifically, the filtering module deletes the data with missing values in the feature data segment, including,

[0081] for any of the feature data segments, acquiring the cumulative character length of the standard character segment as the real-time character length, calculating the real-time missing degree according to the real-time character length, and determining the real-time missing degree according to the standard missing degree,

[0082] if the real-time missing degree is less than the standard missing degree, the feature data segment is not deleted;

[0083] If the real-time missing degree is greater than or equal to the standard missing degree, it is determined that the data feature has missing values, and the feature data segment is deleted.

[0084] The standard character segment is a field where a blank value and a placeholder are located.

[0085] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will fall within the protection scope of the present application.

[0086] The above description is only the preferred embodiments of the present application and is not used to limit the present application; for those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A high performance computing data processing system, comprising: Comprising, a collection module for collecting a plurality of input order transaction log sets; a filtering module for deleting each order transaction log set respectively to obtain respective to-be-stored data; wherein, for any order transaction log set, data with a running time of 0 is deleted, and data with missing values in the feature data segment is deleted to obtain to-be-stored data; a storage module having a plurality of historical data segments, historical update data corresponding to each historical data segment, and storage nodes, for segmenting the to-be-stored data set to obtain to-be-stored data segments, and for storing each to-be-stored data segment on a corresponding storage node respectively, the storage module comprising an analysis unit for analyzing any to-be-stored data segment, and if the to-be-stored data segment is new data, analyzing whether it has an impact on the inventory data; the storage module comprising an appending unit for directly appending the to-be-stored data segment for storage when the analysis unit determines that the new data has no impact on the memory size of the inventory data; a processing unit for adding, modifying, or deleting associated data; wherein, when the analysis unit determines that the to-be-stored data segment has an impact on the associated data, the processing unit adds to the associated data, and when the analysis unit determines that the to-be-stored data segment has no impact on the associated data, the processing unit deletes the associated data; the analysis unit analyzes any to-be-stored data segment, obtains a historical data segment consistent with the real-time flow path of the to-be-stored data segment and a corresponding storage node, matches the to-be-stored data segment with the corresponding historical data segment, and if there is no matching data between the to-be-stored data segment and the historical data segment, determines that the to-be-stored data segment is new data, and the analysis unit determines whether the new data has an impact on the memory size of the inventory data according to the real-time memory size; if there is matching data between the to-be-stored data segment and the historical data segment, the matching data is recorded as associated data, and the analysis unit determines whether the to-be-stored data segment has an impact on the content of the inventory data according to the associated data; the analysis unit obtains the real-time memory size of the to-be-stored data segment when it determines that the to-be-stored data segment is new data or that the to-be-stored data segment has an impact on the content of the associated data, compares the standard memory size with the real-time memory size, and if the real-time memory size is less than or equal to the standard memory size, the analysis unit determines that the new data has no impact on the memory size of the inventory data or that the to-be-stored data segment has no impact on the memory size of the associated data; if the real-time memory size is greater than the standard memory size, the analysis unit determines that the new data has an impact on the memory size of the inventory data or that the to-be-stored data segment has an impact on the memory size of the associated data; The analysis unit obtains matching data as associated data when determining that the to-be-stored data segment and the historical data segment exist matching data, determines whether the to-be-stored data segment has an influence on the content of the inventory data according to the associated data, calculates a real-time similarity between the associated data and the to-be-stored data segment, compares the real-time similarity with a standard similarity, and determines that the to-be-stored data segment has an influence on the content of the associated data when the real-time similarity is less than the standard similarity. If the real-time similarity is greater than or equal to the standard similarity, the analysis unit determines that the to-be-stored data segment has no influence on the associated data, indicating repetition, and the processing unit deletes or modifies the associated data. When the analysis unit determines that the to-be-stored data segment has an influence on the associated data, the processing unit adds or modifies the associated data, including modifying a historical end version number corresponding to the associated data to a current version number. The to-be-stored data segment is stored as an added or modified field on a corresponding storage node, and the end version number and the start version number of the added or modified field are both the current version number.

2. The high performance computing data processing system of claim 1, wherein, When the analysis unit determines that the to-be-stored data segment has no influence on the associated data, the processing unit deletes or modifies the associated data, including modifying a historical end version number corresponding to the associated data to a current version number. The to-be-stored data segment is deleted. When the analysis unit determines that the to-be-stored data segment has an influence on the inventory data or the associated data, the to-be-stored data segment is scheduled to another storage node, and a corresponding associated key is marked.

3. The high performance computing data processing system of claim 1, wherein, The filtering module deletes data with a running time of 0, including obtaining a timestamp of each data segment in any order transaction log set, for any data segment, 4. The high performance computing data processing system of claim 1, wherein, If the running time is 0, the corresponding data segment is deleted. If the running time is not 0, the corresponding data segment is marked as a feature data segment. The filtering module deletes data with missing values in the feature data segment, including, for any feature data segment, obtaining a cumulative character length of a standard character segment as a real-time character length, calculating a real-time missing degree according to the real-time character length, determining the real-time missing degree according to a standard missing degree, and not deleting the feature data segment if the real-time missing degree is less than the standard missing degree.

5. The high performance computing data processing system of claim 1, wherein, If the real-time missing degree is greater than or equal to the standard missing degree, it is determined that the data feature has missing values, and the feature data segment is deleted. ​

Images