Data processing method, computer equipment, storage medium and program product

Abstract

The invention discloses a data processing method, computer equipment, a storage medium and a program product, and relates to the technical field of computers, the data structure type of a data object is determined according to a target field of the data object, the data structure type comprises first structure data and second structure data, the first structure data comprises a data definition language, and the second structure data comprises a second definition language; the second structure data comprises the first structure data and the data content, the corresponding data processing strategies are acquired according to different data structure types, and the data of different data structure types are processed, so that the problems of poor flexibility and pressure on memory resources caused by acquisition of a large amount of data during docking of system tasks are solved; the technical effect of improving the performance of the data processing system is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a computer device, a storage medium, and a program product. Background Art

[0002] As a cutting-edge technology in the field of artificial intelligence, data processing has made significant progress in recent years. In the existing technology, when the data processing system is carrying out data preparation work, it will occupy too much memory when obtaining a large amount of data, which will cause great pressure on the system memory resources. In addition, the traditional data processing system relies on the database to load data, which not only has poor performance, but also causes high development costs and slows down the overall progress of the project when the business process is complicated. As a result, it has poor flexibility when connecting to multiple system services and is difficult to achieve efficient expansion. Therefore, there is an urgent need for a data processing method that can improve the performance of the data processing system. Summary of the invention

[0003] The present application provides a data processing method, a computer device, a storage medium, and a program product to at least solve the problem of system performance degradation caused by an excessive number of simultaneously processed inference requests in the related art.

[0004] The present application provides a data processing method, comprising: Acquire data corresponding to a target field in a data object, and determine a data structure type of the data object according to the data corresponding to the target field, wherein the data structure type includes first structure data and second structure data, wherein the first structure data includes a data definition language, and the second structure data includes the first structure data and data content; In response to the data structure type being first structure data, obtaining a first data cache object and a first data processing strategy corresponding to the first structure data, and processing the first structure data based on cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters; In response to the data structure type being second structure data, a second data processing strategy and a first data cache object corresponding to the first structure data in the second structure data are obtained, and the second structure data is processed based on the cache system parameters of the first data cache object and the second data processing strategy.

[0005] The present application also provides a computer device, including: a memory for storing a computer program; and a processor for implementing the steps of the data processing method in the following implementation mode when executing the computer program.

[0006] Acquire data corresponding to a target field in a data object, and determine a data structure type of the data object according to the data corresponding to the target field, wherein the data structure type includes first structure data and second structure data, wherein the first structure data includes a data definition language, and the second structure data includes the first structure data and data content; In response to the data structure type being first structure data, obtaining a first data cache object and a first data processing strategy corresponding to the first structure data, and processing the first structure data based on cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters; In response to the data structure type being second structure data, a second data processing strategy and a first data cache object corresponding to the first structure data in the second structure data are obtained, and the second structure data is processed based on the cache system parameters of the first data cache object and the second data processing strategy.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method in the following implementation are implemented.

[0008] Acquire data corresponding to a target field in a data object, and determine a data structure type of the data object according to the data corresponding to the target field, wherein the data structure type includes first structure data and second structure data, wherein the first structure data includes a data definition language, and the second structure data includes the first structure data and data content; In response to the data structure type being first structure data, obtaining a first data cache object and a first data processing strategy corresponding to the first structure data, and processing the first structure data based on cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters; In response to the data structure type being second structure data, a second data processing strategy and a first data cache object corresponding to the first structure data in the second structure data are obtained, and the second structure data is processed based on the cache system parameters of the first data cache object and the second data processing strategy.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of the data processing method in the following embodiments when the computer program is executed by a processor.

[0010] Acquire data corresponding to a target field in a data object, and determine a data structure type of the data object according to the data corresponding to the target field, wherein the data structure type includes first structure data and second structure data, wherein the first structure data includes a data definition language, and the second structure data includes the first structure data and data content; In response to the data structure type being first structure data, obtaining a first data cache object and a first data processing strategy corresponding to the first structure data, and processing the first structure data based on cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters; In response to the data structure type being second structure data, a second data processing strategy and a first data cache object corresponding to the first structure data in the second structure data are obtained, and the second structure data is processed based on the cache system parameters of the first data cache object and the second data processing strategy.

[0011] Through the data processing method provided by the present application, the data structure type of the data object is determined according to the target field of the data object, the data structure type includes first structure data and second structure data, the first structure data includes a data definition language, and the second structure data includes the first structure data and data content. According to different data structure types, corresponding data processing strategies are obtained, and data of different data structure types are processed, which solves the problems of poor flexibility when docking system tasks and the pressure on memory resources caused by obtaining a large amount of data, and achieves the technical effect of improving the performance of the data processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application; Figure 2 A schematic diagram of the first structure data provided in an embodiment of the present application; Figure 3 A schematic diagram of a second structure data provided in an embodiment of the present application; Figure 4 A schematic diagram of a first data cache object provided in an embodiment of the present application; Figure 5 A flowchart of a data processing method provided by another embodiment of the present application; Figure 6 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application; Figure 7 A structural block diagram of a data processing device provided in an embodiment of the present application; Figure 8 This is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0017] like Figure 1 As shown, an embodiment of the present application provides a data processing method, which specifically includes the following steps: Step 101: Obtain data corresponding to a target field in a data object, and determine a data structure type of the data object according to the data corresponding to the target field. The data structure type includes first structure data and second structure data. The first structure data includes a data definition language, and the second structure data includes the first structure data and data content.

[0018] Specifically, before obtaining the data corresponding to the target field in the data object, it also includes: receiving the data object, and determining whether the data object meets the preset structural requirements. Receiving the data object and determining whether the data object meets the preset structural requirements include: obtaining metadata information transmitted by the client system; searching for the data object from the client system according to the metadata information; in response to receiving the data object transmitted by the client system, determining whether the data object structure is consistent with the preset data object structure; if consistent, it is considered that the data object whose structure is consistent with the preset data object structure meets the preset structural requirements.

[0019] This solution involves a client system, a target database, and a data processing system formed by a message middleware and ChronicleMap set in this application.

[0020] Among them, the data processing system caches the target database data, and the client system will send metadata information to the data processing system. The data processing system searches for data objects based on the metadata information sent by the client system. The data objects here can be changed data objects. The changed data objects correspond to the changed data information of the target database relative to the target database data cached by the data processing system. The changed data information here includes the field names and field contents contained in the changed data.

[0021] The data processing system in the present application defines a unified data structure. When a data object received from a client system satisfies the unified data structure defined by the data processing system, the data object is considered to be a data object that meets preset requirements.

[0022] The unified data structure here may be a data structure including multiple set field names and multiple field contents in a data object.

[0023] If the received data object does not meet the requirements of the preset structure, the data object will not be processed and will be rejected.

[0024] If the received data object meets the requirements of the preset structure, the data corresponding to the target field in the data object is obtained.

[0025] Because a unified data structure is defined in this application, the data objects received by the data processing system are all data including multiple field names and multiple field contents, and the field contents here can be descriptive information of the changed data. Among them, the multiple field names include a data type field (the target field here), and the data structure type of the data object can be determined according to the field content corresponding to the data type field. The data structure type of the data object set in this application includes first structure data and second structure data. The first structure data includes data definition language, and the second structure data includes data definition language and data content. Data Definition Language (DDL) is an important component of the SQL language, which is mainly responsible for data structure definition and database object definition. In this application, the first structure data can be expressed as DDL data.

[0026] Different structural data types in this application contain different data information and correspond to different data processing strategies, which can meet the requirements of more customer systems.

[0027] Specifically, in the present application, the first structure data is set to include a first field set and data corresponding to the first field set, and the second structure data includes a second field set and data corresponding to the second field set. The method also includes: determining the first structure data and determining the second structure data, and determining the first structure data and determining the second structure data include: obtaining a first field set, and determining the first structure data based on the first field set; the first field set includes data source, sub-data type group, main data type group, metadata ID, data operation type and data type field; obtaining a second field set, and determining the second structure data based on the second field set; the second field set includes data source, sub-data type group, main data type group, metadata ID, data ID, data content, data operation type and data type field.

[0028] Specifically, the data structure of the first structure data can be as follows: Figure 2 As shown: the first structure data (DDL data object in the figure) is composed of the first field set and the business data corresponding to the first field set. The first field set includes source (data source, data source type in the figure), groupID (sub data type group, data type group in the figure), parentID (main data type group, data type hierarchy relationship in the figure), metadata ID (metadata ID, metadata id in the figure, used to find metadata objects in the metadata pool), optionType (operation type, ADD, DELETE, RELOAD, CLEAR operations in the figure) and data Type (data type, not shown in the figure) and other fields.

[0029] The data structure of the second structure data can be as follows Figure 3 As shown: The second structure data (illustrated data object) is composed of the second field set and the business data corresponding to the second field set. The second field set includes source (data source, illustrated data source type), groupID (sub data type group, illustrated data type group), parentID (main data type group, illustrated data type hierarchy), metadata ID (metadata ID, illustrated metadata id, used to find metadata objects in the metadata pool), data ID (illustrated data id), data (illustrated data content, mainly MAP type data, used to store metadata information), optionType (operation type, illustrated ADD, DELETE, RELOAD, operation) and data Type (data type, not shown) and other fields.

[0030] Among them, source (data source) corresponds to the business type and is used to distinguish data from different business systems. For example, the data of a data table in a business system can be marked in the form of "XXX (representing the business system) -XXX data table (representing the data table)". As long as the sources of different business systems are different, they can be distinguished.

[0031] groupID (sub-data type group) is used to indicate a data group, and its purpose is to distinguish different data groups. For example, when receiving batches of data from different sources, the data from the same data source can be grouped into one group, such as the watch list data from the MySQL database as a group of data; the transaction log data from the MySQL database can also be a group of data.

[0032] parentID (primary data type group) is the primary ID of groupID, which enables different groups to be recursively nested to adapt to the situation where a business system has multiple primary and child type data groups.

[0033] Metadata ID is used to obtain metadata information by ID, which can describe the number and type of fields in the data set and provide functional support for encapsulation cache. Metadata information is data about data, mainly information about data properties, used to support functions such as indicating storage location, historical data, resource search, file records, etc. Metadata is a kind of electronic catalog. In order to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, so as to achieve the purpose of assisting data retrieval.

[0034] optionType (operation type) refers to the operation on cache construction information, where the add operation and the reload operation have the same function, both of which are used to load data.

[0035] Data Type is used to distinguish data types and complete different business operations according to different data types. For example, the data type part of the first structure data can be expressed as "data Type=DDL". In this example, it means that the target data type of the first structure data is DDL data. The data type part of the second structure data can be expressed as "data Type=data". It means that the target data type of the second structure data is data data.

[0036] The data field is of Map type. A MAP type field is a data structure that stores key-value pairs, where the key and value can be of any type. It provides a flexible way to store and query structured data. The key is the field name, and the value is the field content. When the first structure data does not reach the cache system, or the first structure data reaches the cache system after the second structure data reaches the cache system, the cache information cannot be found using the "three elements". In this case, the field information is obtained through the metadata ID field, and a default first default structure data is created. At the same time, a cache Map and a cache file are created based on the default structure data for storing data. This is a fault-tolerant mechanism design. The cache system here is a system that caches the first structure data and the second structure data, and can specifically be a cache system based on ChronicleMap.

[0037] ChronicleMap is a high-performance Java container library based on memory-mapped files (mMap), designed for high-performance and low-latency application scenarios. It supports large amounts of data and high concurrent access, and is particularly suitable for scenarios that require fast read and write operations. ChronicleMap supports persisting data to disk, so that data will not be lost even if the process terminates. And the design takes into account multi-process access scenarios, making it suitable for use in a multi-process environment.

[0038] The field names in data correspond to the field names in metadata. The fields in data can only be less than the fields in metadata. VALUE is the actual value of the changed data corresponding to the data object. Metadata only describes the name and content of the field. In other words, metadata is the description information of the changed data.

[0039] From the above description, it can be seen that the difference between the first field set and the second field set of the first structure data and the second structure data is that the second field set is provided with data ID (data ID) and data (data content) fields based on the first field set.

[0040] The first structure data and the second structure data are set here so that they can still be used and can still be used to access data even when there is no specific data. That is, the first structure data is data description information, which can tell the client system what data can be obtained from here, but it does not contain the content of the data information obtained by the client system. The second structure data is data description information and the content of the data information. Through the second structure data, the address of the specific data information required by the client system can be obtained, and the real cache data content can be obtained based on the address.

[0041] Step 102: In response to the data structure type being first structure data, obtain a first data cache object and a first data processing strategy corresponding to the first structure data, and process the first structure data based on cache system parameters and the first data processing strategy, wherein the first data cache object includes cache system parameters.

[0042] In the present application, the first structure data is also converted into a data cache object to obtain a first data cache object. The specific structure of the first data cache object can be as follows: Figure 4 As shown: The first data cache object includes multiple fields such as source (data source), groupID (sub-data type group), parentID (main data type group), metadata ID (metadata ID) and ChronicleMap (cache system parameters).

[0043] Among them, source (data source) corresponds to the business type and its function is to distinguish data from different business systems.

[0044] groupID (sub-data type group) is used to indicate a data group in order to distinguish different data groups.

[0045] parentID (primary data type group) is the primary ID of groupID, which enables different groups to be recursively nested to adapt to the situation where a business system has multiple primary and child type data groups.

[0046] The metadata ID is used to obtain metadata information based on the ID. It can describe the number and type of fields in the data set and provide functional support for the encapsulation cache.

[0047] ChronicleMap (cache system parameters) is a key parameter for creating a cache system, including the maxBloatFactor parameter, which indicates the maximum expansion factor during abnormal bursts, the averageKeySize parameter, which indicates the average string size of each field name of the metadata sent by the client system, the averageValueSize parameter, which indicates the average string length of the description content of the metadata sent by the client system, the entries parameter, which indicates the storage space capacity of the cache system, and the createPersistedToPath parameter, which indicates the storage path of the first structure data.

[0048] Here, when the first structure data is stored for the first time, the storage path is usually a public parameter location specified by different projects. After the storage path is determined, it can be concatenated with the name of the cache file to complete the definition of the cache file path. The cache file name is composed of the string concatenation of the three elements corresponding to the cache file: source (data source), groupID (sub-data type group), and parentID (main data type group).

[0049] The cache pool of the first structure data, that is, the storage space of the first structure data, is established when the cache system is started. It is an independent ChronicleMap object specifically used to store DDL cache data. Since the cache system adopts a singleton system design locally, it ensures that there is only one DDL cache Map and the corresponding local file. All data cache operations obtain data from the DDL cache Map and then construct the corresponding ChronicleMap object. Each of the three elements corresponds to a DDL cache object and is stored in the DDL cache Map, as well as a data cache ChronicleMap named after the three elements and the corresponding local file.

[0050] See also Figure 5 , obtain cache system parameters in the first data cache object, the cache system parameters include the first data cache address; obtain the data operation type corresponding to the first structure data, the data operation type corresponding to the first structure data includes a first data clear operation (clear), a first data delete operation (delete) and a first data add operation (add); if the data operation type corresponding to the first structure data is the first data clear operation, the cache data at the first data cache address is deleted; if the data operation type corresponding to the first structure data is the first data delete operation, the metadata in the first data cache object is deleted, and the cache data at the first data cache address is deleted; if the data operation type corresponding to the first structure data is the first data add operation, the cache data corresponding to the first structure data is obtained, and the cache data corresponding to the first structure data is added to the cache system based on the first data cache address.

[0051] Step 103: In response to the data structure type being the second structure data, obtain the second data processing strategy and the first data cache object corresponding to the first structure data in the second structure data, and process the second structure data based on the cache system parameters of the first data cache object and the second data processing strategy.

[0052] Please continue reading Figure 5, obtain cache system parameters in the first data cache object, the cache system parameters include a first data cache address; determine a second data cache address based on the first data cache address; obtain a data operation type corresponding to the second structure data, the data operation type corresponding to the second structure data includes a second data addition operation and a second data deletion operation; if the data operation type corresponding to the second structure data is a second data deletion operation, obtain the second data cache address, and delete the cache data in the second data cache address; if the data operation type corresponding to the second structure data is a second data addition operation, obtain the second cache data corresponding to the second structure data, and add the second cache data corresponding to the second structure data to the cache system based on the second data cache address.

[0053] In this application, after determining whether it is DDL data (first structure data) or cache data (second structure data), the specific operation can be obtained according to the operation type field in the corresponding structure data. The figure shows three operation types of clean, delete, and add corresponding to DDL data, and two operation types of delete and add corresponding to cache data.

[0054] This application establishes a cache object for the DDL data object (first structure data) to obtain a DDL data cache object (first data cache object), and establishes a data object for the cache data (second structure data) to obtain a second data cache object. Since the DDL data cache object includes the ChronicleMap parameter, this parameter includes the cache address, so that the DDL data cache object can be added or deleted at the cache address according to the corresponding operation type.

[0055] There are two operations for cache data: adding and deleting. If adding, get the ChronicleMap object from the DDL data cache object. This object is the cache address of the DDL cache data. Add the DDL cache data cache address to the address lookup instruction to get the address of the second data cache object. Delete and add the second cache data at this cache address.

[0056] In one embodiment, the target cache data size of the second cache data and the remaining size of the target storage space of the cache system are obtained; the remaining size of the target storage space is compared with the target cache data size; if the remaining size of the target storage space is less than or equal to the target cache data size, the second cache data is directly stored in the target storage space; if the remaining size of the target storage space is greater than the target cache data size, the target storage space is expanded based on an expansion algorithm.

[0057] Since the second cache data includes data content, when adding the second cache data, there is a process of determining whether the added data will overflow, including a capacity expansion algorithm and establishing a new cache address.

[0058] Specifically, a first storage space size preset range, a second storage space size preset range, and a third storage space size preset range may be obtained; the target cache data size is compared with the first storage space size preset range, the second storage space size preset range, and the third storage space size preset range respectively; if the target cache data size is within the first storage space size range, the target storage space is expanded based on the first expansion formula; if the target cache data size is within the second storage space size range, the target storage space is expanded based on the second expansion formula; if the target cache data size is within the third storage space size range, the target storage space is expanded based on the third expansion formula; The first expansion formula is as follows: ; The second expansion formula is as follows: ; The third expansion formula is as follows: ; Among them, C MAX A is the target storage space size after expansion, B is the target storage space size before expansion, and D is a constant, which can be set according to actual needs.

[0059] The present application also sets an expansion threshold, which can be obtained. The expansion threshold is the maximum value of the target storage space size. When the target cache data size is greater than the expansion threshold, a new storage space is created and a timestamp is added to the new storage space. The second cache data is imported into the new storage space.

[0060] When the cache data capacity overflows, an exception is obtained through the try method. The DDL cache object is obtained by searching the source (data source), groupID (sub data type group), and parentID (main data type group) fields, and a ChronicleMap is rebuilt and the original data is copied to the new Map. After modifying the path name, the original object and file are deleted. The expansion algorithm is as follows: Each time the capacity is expanded, the size of the file currently storing data must be read.

[0061] If the current file size is less than 128kb, follow the expansion algorithm: max=max+(max*0.5), where max is the current maximum storage capacity.

[0062] If the current file size is between 128kb and 1mb, max=max+(max*0.25), where max is the current maximum storage capacity.

[0063] If the current file size is between 1mb and 512mb, max=max+2000. max is the current maximum storage quantity. At this time, expansion will make the file too large, wasting disk space. At the same time, there will not be too much cache of this type of data, and frequent expansion operations will not be triggered.

[0064] If the current file size exceeds 512mb, data will be rejected. At this time, the amount of data will generally exceed 10 million or even 100 million, which will put great pressure on disk expansion. At the same time, it is no longer used for caching, but for persistent data, which is not supported by this system.

[0065] In one embodiment, the time when the first structure data arrives at the cache system is obtained, and it is determined whether the time when the first structure data arrives at the cache system meets the preset time requirement; if not, the second structure data sends metadata, and the first default structure data is created based on the metadata sent by the second structure, and a default cache address and a default cache file are created based on the first default structure data.

[0066] In actual applications, in response to receiving the first structure data, it will be determined whether the cache system includes the first default structure data corresponding to the first structure data; if included, the first default structure data is verified based on the first structure data to obtain the difference information between the first structure data and the first default structure data; and the first default structure information is corrected based on the difference information.

[0067] The data objects received in this application all have DDL objects (first structure data) and cache objects (second structure data). Usually, the DDL object is first obtained and converted into a DDL cache object and stored in the cache system, and then the data object is stored in the cache system according to the cache address of the DDL cache object. In actual operation, the DDL object of a data is not stored in the cache system, or the DDL data object of the data has not been stored, but the data object is stored in the cache system. At this time, metadata is obtained from the data object, and a default DDL data object is created based on the metadata. After the DDL data object of the data is received, the received DDL data object is compared with the DDL data object created based on the metadata, and the default DDL data object is corrected.

[0068] This setting is because in actual operation, there is a situation where the first structure data cannot be received. After the first data reaches the cache system, the absence of the first structure data will cause data loss. Since the first structure data is not received, the system will forcibly send a metadata ID. At this time, the metadata ID is obtained based on the data object. The metadata ID includes metadata information. The first default structure data is created based on the metadata information. After the system receives the first structure data, it will compare the received first structure data with the first default data object created according to the metadata, and correct the default DDL data object, which is equivalent to a fault-tolerant mechanism to reduce the cost of trial and error.

[0069] In one embodiment, in response to receiving a data object sent by a client system, the data object is stored in a message queue of a message middleware; and the data object in the message queue is sent to a cache system for processing according to the level of the message queue.

[0070] The number of processor cores of a data processing system can be obtained, and the number of consumer threads can be determined based on the number of processor cores of the data processing system; wherein, determining the number of consumer threads based on the number of processor cores of the data processing system includes: obtaining a number threshold range of a first processor core number, a number threshold range of a second processor core number, and a number threshold range of a third processor core number; comparing the number of processor cores with the number threshold range of the first processor core number, the number threshold range of the second processor core number, and the number threshold range of the third processor core number, respectively; if the number of processor cores is within the number threshold range of the first processor core number, determining the number of consumer threads based on a first calculation formula; if the number of processor cores is within the number threshold range of the second processor core number, determining the number of consumer threads based on a second calculation formula; if the number of processor cores is within the number threshold range of the third processor core number, determining the number of consumer threads based on a third calculation formula.

[0071] The first calculation formula is as follows: ; The second calculation formula is as follows: ; The third calculation formula is as follows: ; Among them, K is the number of consumer threads and Y is the number of processor cores.

[0072] The kafka client of this application can be set up in multiple ways, which are uniformly managed by a thread pool and use only one topic name. All producers only need to send data to the same topic according to the specified data structure. The thread pool creates consumer threads based on the current number of CPU cores, monitors each thread, and outputs logs of received data. Try to improve data consumption capabilities and the rate of parsing local files by improving hardware configuration. This is especially critical in virtual machine deployment and cloud environment deployment scenarios, because such environments will frequently change configurations as needed. In this way, while improving the configuration, the cache system can also provide corresponding performance.

[0073] Specifically, the algorithm for creating consumer thread data is as follows: In order to ensure system performance while also ensuring Kafka's consumption performance, it is necessary to set the number of partitions of the Kafka topic according to the number of CPUs, and allocate the same number of consumer threads as the number of partitions.

[0074] k represents the current number of threads, and cpu represents the current number of cores.

[0075] If the current number of CPU cores is less than 32, the formula for the number of threads is: k=cpu / 8+1.

[0076] If the current CPU is greater than 32 but less than 128, the formula for the number of threads is: k=cpu / 32+4.

[0077] If the current CPU has more than 128 cores and is a supercomputer, the formula for the number of threads is: k = CPU / 128 + 8. Too many consumer threads will not improve consumption efficiency, but will slow down the efficiency of the CPU in executing its main business.

[0078] Let k represent the current number of threads and cpu represent the current number of cores, which can be expressed as the following piecewise function:

[0079] In a feasible implementation, it may be determined whether the data object written into the message queue is complete; wherein, when the data object written into the message queue is incomplete, a data compensation mechanism is triggered.

[0080] Since MQ systems such as RabbitMQ and RocketMQ are used as message queues to implement asynchronous processing, although messages (events) can be stored on disk, even if there is a problem with MQ, the message data will not be lost, but the asynchronous process may lose messages in the message sending, transmission, processing and other links. In addition, no MQ middleware can ensure 100% availability, and it is necessary to consider how the asynchronous process continues when it is unavailable. Therefore, for the asynchronous processing process, compensation must be considered or the establishment of a master-slave dual-active process. Based on this, the present invention sets a data compensation mechanism. When the data object written to the message queue is incomplete, the data compensation mechanism is triggered to compensate the data object written to the message queue.

[0081] Optionally, the data object not written into the message queue can be compared with its corresponding data object after being written into the message queue. When the comparison shows that there is a difference between the two, the data object written into the message queue is considered incomplete. And the system will feedback the specific difference between the data object not written into the message queue and its corresponding data object after being written into the message queue to the user. The user can be reminded by an alarm or other means that the data object currently written into the message queue is incomplete.

[0082] Exemplarily, when it is confirmed that the data object written into the message queue is incomplete, the data compensation mechanism is triggered and a data compensation instruction is generated. In response to receiving the data compensation instruction, the difference information between the data object written into the message queue and the corresponding data object after being written into the message queue can be captured and the scenario parameters of the difference information can be matched according to the feedback information of the data object itself that is not written into the message queue. The difference information is timely compensated based on the information itself to a certain extent. By setting up the data compensation mechanism, the reliability of message transmission is improved and the correct transmission of the message is ensured. In addition, the abnormal messages (data objects after being written into the message queue) can also be stored, generally stored in the message middleware cache queue. The messages can be retrieved and resent as needed to avoid the loss of messages after the abnormality occurs, so as to improve the reliability and robustness of subsequent data transmission.

[0083] like Figure 6 As shown, the present application provides a data processing system, and the overall design of the data processing system is divided into four layers, including a data channel layer, a data conversion encapsulation layer, a data cache layer, and a data storage layer.

[0084] The data channel layer includes message middleware components (Kafka channel shown in the figure), interface channels, unified communication data structures, and thread pool management.

[0085] The data channel layer of this application is responsible for receiving data, which is mainly implemented by Kafka. Kafka is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern network. This data is usually solved by processing logs and log aggregation due to throughput requirements. This is a viable solution for log data and offline analysis systems like Hadoop, but with the limitation of real-time processing. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time messages through clusters.

[0086] Since the channel layer completely relies on Kafka, this application sets a timer to push the full data when synchronizing data with Kafka. This is to avoid the situation where the cache system fails to start after the Kafka service is started, and it takes too long to pull data before the Kafka data expires, which ultimately causes data loss and cache data missing and unusable.

[0087] The data channel layer is responsible for receiving data, which is mainly implemented by Kafka. It converts the received JSON format data into the specified data structure. A unified data structure will be defined in the data channel layer. When receiving data, the structure layer directly uses the data structure and completes the receiving operation through the API interface. The role of Kafka is to convert the received JSON data into the aforementioned unified data structure.

[0088] The main task of the data conversion encapsulation layer is to convert the received data into the data type for storage. After receiving the DDL data, it needs to be converted into a DDL data cache object to describe the parameters for building the ChronicleMap cache pool.

[0089] The data cache layer mainly undertakes three tasks: first, calling the storage layer interface according to the DDL cache object, creating and managing the ChronicleMap object; second, after receiving the cache data, accurately finding the created ChronicleMap object and writing the data into it; third, being able to quickly locate and search the cache and obtain the cache content.

[0090] The data storage layer directly uses the ChronicleMap object to implement functions, and uses its ability to write files to achieve the purpose of caching local files.

[0091] This application solves the design of data cache localization by utilizing the file writing characteristics and high-performance capabilities of ChronicleMap. It also optimizes the design of ChronicleMap usage issues, such as the processing of overflow data, the processing of data files, and the management design of multiple data caches; the design of a unified data receiving method utilizes the distributed characteristics of Kafka, and only needs to connect to Kafka to complete data reception and caching, which improves scalability and decouples the data generation end and the data usage end.

[0092] The embodiment of the present application provides a data processing device, the data processing device is specifically as follows Figure 7 As shown, the data processing device includes: a receiving module 20 and a processing module 21.

[0093] The receiving module 20 is used to obtain the data corresponding to the target field in the data object, and determine the data structure type of the data object according to the data corresponding to the target field. The data structure type includes first structure data and second structure data. The first structure data includes a data definition language, and the second structure data includes the first structure data and data content.

[0094] The processing module 21 is used for obtaining a first data cache object and a first data processing strategy corresponding to the first structure data in response to the data structure type being the first structure data, and processing the first structure data based on the cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters; In response to the data structure type being the second structure data, a second data processing strategy and a first data cache object corresponding to the first structure data in the second structure data are obtained, and the second structure data is processed based on the cache system parameters of the first data cache object and the second data processing strategy.

[0095] For the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, which will not be described in detail here.

[0096] The embodiment of the present application also provides a computer device, such as Figure 8 As shown, the computer device includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above data processing method embodiments.

[0097] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when running.

[0098] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0099] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] The above is a detailed introduction to a data processing method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire data corresponding to a target field in a data object, and determine a data structure type of the data object according to the data corresponding to the target field, wherein the data structure type includes first structure data and second structure data, wherein the first structure data includes a data definition language, and the second structure data includes the first structure data and data content; In response to the data structure type being first structure data, obtaining a first data cache object and a first data processing strategy corresponding to the first structure data, and processing the first structure data based on cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters; In response to the data structure type being second structure data, a second data processing strategy and a first data cache object corresponding to the first structure data in the second structure data are obtained, and the second structure data is processed based on the cache system parameters of the first data cache object and the second data processing strategy.

2. The data processing method according to claim 1, characterized in that: Before obtaining the data corresponding to the target field in the data object, the method further includes: receiving the data object, and determining whether the data object meets the preset structural requirements. The receiving the data object, and determining whether the data object meets the preset structural requirements include: Obtain metadata information transmitted by the client system; searching for a data object from a client system according to the metadata information; In response to receiving a data object transmitted by a client system, determining whether a data object structure is consistent with a preset data object structure; If they are consistent, it is considered that the data object whose structure is consistent with the preset data object structure meets the preset structure requirements.

3. The data processing method according to claim 1, characterized in that: The first structure data includes a first field set and data corresponding to the first field set, the second structure data includes a second field set and data corresponding to the second field set, the second field set includes the first field set, and the method further includes: determining the first structure data and determining the second structure data: The determining of the first structure data and the determining of the second structure data include: Acquire a first field set, and determine first structure data based on the first field set; The first field set includes data source, sub-data type group, main data type group, metadata ID, data operation type and data type field; Acquire a second field set, and determine second structure data based on the second field set; The second field set includes data source, sub-data type group, main data type group, metadata ID, data ID, data content, data operation type and data type field.

4. The data processing method according to claim 1, characterized in that: The first data cache object and the first data processing strategy corresponding to the first structure data are obtained, and the first structure data is processed based on the cache system parameters and the first data processing strategy, wherein the first data cache object includes the cache system parameters including: Obtaining cache system parameters in the first data cache object, wherein the cache system parameters include a first data cache address; Acquire a data operation type corresponding to the first structure data, where the data operation type corresponding to the first structure data includes a first data clearing operation, a first data deleting operation, and a first data adding operation; If the data operation type corresponding to the first structure data is a first data clearing operation, deleting the cache data at the first data cache address; If the data operation type corresponding to the first structure data is a first data deletion operation, then deleting the metadata in the first data cache object, and deleting the cache data at the first data cache address; If the data operation type corresponding to the first structure data is a first data adding operation, the cache data corresponding to the first structure data is acquired, and the cache data corresponding to the first structure data is added to the cache system based on the first data cache address.

5. The data processing method according to claim 1, characterized in that: The acquiring the second data processing strategy and the first data cache object corresponding to the first structure data in the second structure data, and processing the second structure data based on the cache system parameters of the first data cache object and the second data processing strategy includes: Acquire a cache system parameter in a first data cache object, the cache system parameter including a first data cache address; determine a second data cache address based on the first data cache address; Acquire a data operation type corresponding to the second structure data, where the data operation type corresponding to the second structure data includes a second data adding operation and a second data deleting operation; If the data operation type corresponding to the second structure data is a second data deletion operation, obtaining a second data cache address, and deleting cached data in the second data cache address; If the data operation type corresponding to the second structure data is a second data adding operation, the second cache data corresponding to the second structure data is acquired, and the second cache data corresponding to the second structure data is added to the cache system based on the second data cache address.

6. The data processing method according to claim 5, characterized in that: The adding the second cache data corresponding to the second structure data to the cache system based on the data cache address includes: Obtaining a target cache data size of the second cache data and a remaining size of a target storage space of the cache system; Compare the remaining size of the target storage space with the target cache data size; If the remaining size of the target storage space is less than or equal to the target cache data size, directly storing the second cache data in the target storage space; If the remaining size of the target storage space is greater than the target cache data size, the target storage space is expanded based on an expansion algorithm.

7. The data processing method according to claim 6, characterized in that: The expanding the target storage space based on the expansion algorithm includes: Obtaining a first storage space size preset range, a second storage space size preset range, and a third storage space size preset range; Compare the target cache data size with the first storage space size preset range, the second storage space size preset range, and the third storage space size preset range respectively; If the target cache data size is within the first storage space size range, the target storage space is expanded based on the first expansion formula; If the target cache data size is within the second storage space size range, expanding the target storage space based on the second expansion formula; If the target cache data size is within the third storage space size range, expanding the target storage space based on a third expansion formula; The first expansion formula is as follows: ； The second expansion formula is as follows: ； The third expansion formula is as follows: ； Among them, C MAX A is the target storage space size after expansion, B is the target storage space size before expansion, and D is a constant, which can be set according to actual needs.

8. The data processing method according to claim 7, characterized in that: The method further comprises: Obtaining a capacity expansion threshold, where the capacity expansion threshold is a maximum value of a target storage space size; When the target cache data size is larger than the expansion threshold, a new storage space is created and a timestamp is added to the new storage space; Import the second cache data into the new storage space.

9. The data processing method according to claim 1, characterized in that: The method further comprises: Obtaining the time when the first structure data arrives at the cache system, and determining whether the time when the first structure data arrives at the cache system meets a preset time requirement; If not, the second structure data is made to send metadata, the first default structure data is created based on the metadata sent by the second structure data, and the default cache address and the default cache file are created based on the first default structure data.

10. The data processing method according to claim 9, characterized in that: The method further comprises: In response to receiving the first structure data, determining whether the cache system includes first default structure data corresponding to the first structure data; If included, verifying the first default structure data based on the first structure data to obtain difference information between the first structure data and the first default structure data; The first default structure information is modified based on the difference information.

11. The data processing method according to claim 2, characterized in that: The method further comprises: In response to receiving the data object sent by the client system, storing the data object in a message queue of the message middleware; The data objects in the message queue are sent to the cache system for processing according to the level of the message queue.

12. The data processing method according to claim 1, characterized in that: The method further comprises: Obtaining the number of processor cores of a data processing system, and determining the number of consumer threads based on the number of processor cores of the data processing system; Wherein, determining the number of consumer threads based on the number of processor cores of the data processing system includes: Obtaining a quantity threshold range of a first processor core number, a quantity threshold range of a second processor core number, and a quantity threshold range of a third processor core number; Comparing the number of processor cores with a first processor core number threshold range, a second processor core number threshold range, and a third processor core number threshold range, respectively; If the number of the processor cores is within a first processor core threshold, determining the number of consumer threads based on a first calculation formula; If the number of the processor cores is within a second processor core threshold, determining the number of consumer threads based on a second calculation formula; If the number of processor cores is within a third processor core number threshold range, determining the number of consumer threads based on a third calculation formula; The first calculation formula is as follows: ； The second calculation formula is as follows: ； The third calculation formula is as follows: ； Among them, K is the number of consumer threads and Y is the number of processor cores.

13. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 12 are implemented.

Images