A data processing method, device, system and storage medium

Abstract

The present application discloses a method, device, system and storage medium for data processing, specifically, firstly loading at least one data source interface, secondly, obtaining information to be processed based on the data source interface, further, loading an information processing plug-in based on the classification label carried by the information to be processed, and generating processed information corresponding to the information to be processed after processing, and finally, converting the processed information into a corresponding storage format and storing it. The embodiment of the present application quickly accesses the data source interface, configures the reception, processing and storage of data, adapts to small and large data processing needs, and improves computing and storage efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, system and storage medium for data processing. Background Art

[0002] In the field of engineering deployment of real-time data processing, the industry currently generally uses tools such as Spark Streaming, Flink, Kafka, and Pulsar for real-time data calculations. However, these tools need to be deployed in specific environments. During development, different tools require familiarity with different application programming interfaces (APIs), as well as understanding of related knowledge and tuning methods, which has a certain learning threshold. At the same time, most open source tools rely on numerous open source software packages, and due to too many software versions, version conflicts are very likely to occur. Summary of the invention

[0003] The embodiment of the present application provides a method for data processing, which overcomes the problem that software versions are not universal in real-time computing and improves the computing and storage efficiency of data.

[0004] The method includes:

[0005] Load at least one data source interface;

[0006] Acquire information to be processed based on the data source interface;

[0007] Based on the classification label carried by the information to be processed, load the information processing plug-in, and generate processed information corresponding to the information to be processed after processing;

[0008] The processed information is converted into a corresponding storage format and stored.

[0009] Optionally, a first quantity of the to-be-processed information acquired within a preset time period is recorded, and a second quantity of the processed information acquired within the preset time period is recorded;

[0010] Compare the numerical values ​​of the first number and the second number, and when the second number is smaller than the first number and the difference between the second number and the first number is smaller than a preset threshold, adjust the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads.

[0011] Optionally, a filter is loaded, and the processed information marked with a filter tag is filtered by the filter.

[0012] In another embodiment of the present invention, a data processing device is provided, the device comprising:

[0013] A loading module, used to load at least one data source interface;

[0014] An acquisition module, used for acquiring information to be processed based on the data source interface;

[0015] A generating module, configured to load an information processing plug-in based on the classification label carried by the information to be processed, and generate processed information corresponding to the information to be processed after the information to be processed is processed;

[0016] The storage module is used to convert the processed information into a corresponding storage format and store it.

[0017] Optionally, the device further comprises:

[0018] A collection module, used to record a first amount of the to-be-processed information obtained within a preset time period, and to record a second amount of the processed information obtained within the preset time period;

[0019] An adjustment module is used to compare the numerical values ​​of the first number and the second number, and when the second number is smaller than the first number and the difference between the second number and the first number is smaller than a preset threshold, adjust the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads.

[0020] Optionally, the device further comprises:

[0021] The filtering module is used to load a filter and filter the processed information marked with a filtering tag through the filter.

[0022] In another embodiment of the present invention, a data processing system is provided, the system comprising a data source layer, a data channel layer, a data processing layer, a data adaptation layer and a data storage layer, wherein the data source layer is used to provide at least one data source interface and transmit information to be processed through the data channel layer;

[0023] The data processing layer is used to process the information to be processed obtained through the data channel layer to generate processed information;

[0024] The data adaptation layer is used to convert the processed information into a corresponding storage format;

[0025] The data storage layer is used to store the processed information in a storage medium corresponding to the storage format.

[0026] Optionally, the data processing system further comprises a system management layer, wherein the system management layer is used to:

[0027] interacting with the data source layer and the data channel layer to record a first amount of the to-be-processed information acquired within a preset time period, and interacting with the data channel layer to record a second amount of the processed information acquired within the preset time period;

[0028] Compare the numerical values ​​of the first number and the second number, and when the second number is less than the first number and the difference between the second number and the first number is less than a preset threshold, adjust the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads through the data channel layer.

[0029] Optionally, the data processing system further comprises a data filtering layer, which is used to load a filter and filter the processed information marked with a filtering tag after being processed by the data processing layer through the filter.

[0030] In another embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, and when the instructions are executed by a processor, the processor executes each step in the above-mentioned data processing method.

[0031] In another embodiment of the present invention, a terminal device is provided, including a processor, wherein the processor is used to execute each step in the above-mentioned data processing method.

[0032] Based on the above embodiment, at least one data source interface is first loaded, and then, the information to be processed is obtained based on the data source interface. Further, based on the classification label carried by the information to be processed, the information processing plug-in is loaded, and the processed information corresponding to the processed information is generated, and finally, the processed information is converted into a corresponding storage format and stored. The embodiment of the present application quickly accesses the data source interface, configures the reception, processing and storage of data, adapts to the needs of small and large data processing, and improves the computing and storage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1A schematic flow chart of a data processing method provided by embodiment 100 of the present application is shown;

[0035] Figure 2 A schematic diagram showing a specific process of a data processing method provided by embodiment 200 of the present application;

[0036] Figure 3 A schematic diagram showing the system interaction in which data processing is performed according to the embodiment 300 of the present application is performed;

[0037] Figure 4 A schematic diagram showing a data processing device further provided by the embodiment 400 of the present application;

[0038] Figure 5 A schematic diagram of a terminal device provided by embodiment 500 of the present application is shown. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the implementation of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Based on the problems in the prior art, the embodiments of the present application provide a method for data processing, which is mainly applicable to the field of computer technology. By constructing a data processing framework for the data processing system, connecting to the existing data source, and completing the reception, processing and storage of data with simple configuration, it can adapt to small and large data processing needs and simplify the development process to implement a method for data processing. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Figure 1FIG. 1 is a flow chart of a data processing method provided by Embodiment 100 of the present application. The detailed steps are as follows:

[0042] S11, loading at least one data source interface.

[0043] In this step, the data processing system provides a variety of common data source interfaces, such as JMS, Kafka, FileSystem, hdfsSystem, etc. Select at least one data source interface for data interaction.

[0044] S12, obtaining information to be processed based on the data source interface.

[0045] In this step, the configuration file of the data source interface is read, and the information to be processed is pulled from at least one data source interface through the read configuration file, and a classification tag is marked for the information to be processed. The classification tag is used to distinguish the business type of the message.

[0046] S13, based on the classification label carried by the information to be processed, loading the information processing plug-in, and generating processed information corresponding to the information to be processed after being processed.

[0047] In this step, the data processing system provides at least one information processing plug-in, and loads the corresponding information processing plug-in for the information to be processed through the classification label. Further, the information to be processed is parsed and serialized by the information processing plug-in to generate processed information corresponding to the information to be processed.

[0048] S14, converting the processed information into a corresponding storage format and storing it.

[0049] In this step, the data processing system converts the processed information into the storage format required by the corresponding storage. For example, for the Key-value storage format of the data storage layer Redis, the processed information needs to be converted into a storage format of (Key, Value) tuples, and finally written into the storage medium in batches.

[0050] As described above, based on the above embodiments, at least one data source interface is first loaded, and then, based on the data source interface, the information to be processed is obtained, and further, based on the classification label carried by the information to be processed, the information processing plug-in is loaded, and the processed information corresponding to the processed information is generated, and finally, the processed information is converted into a corresponding storage format and stored. The embodiment of the present application quickly accesses the data source interface, configures the reception, processing and storage of data, adapts to small and large data processing needs, and improves computing and storage efficiency.

[0051] like Figure 2As shown, it is a schematic diagram of a specific process of a data processing method provided by embodiment 200 of the present application. The detailed process of the specific process is as follows:

[0052] S201, loading at least one data source interface.

[0053] Here, the data processing system starts and loads the configuration file of the data source interface, loads at least one data source interface according to the configuration file of the data source interface, and initializes it.

[0054] S202, obtaining information to be processed through a data source interface.

[0055] S203, marking the information to be processed with a classification label.

[0056] Here, the data source interface periodically obtains the information to be processed and marks the classification label. The classification label can be based on business needs. For example, in the field of e-commerce, classification labels such as order type, shopping cart type, click type and browse type can be set for the information to be processed.

[0057] S204, based on the classification label carried by the information to be processed, load the information processing plug-in, and generate processed information corresponding to the information to be processed after processing.

[0058] Here, the data processing system is provided with an information processing plug-in corresponding to the type of information to be processed. Further, based on the classification label carried by the information to be processed, different information processing plug-ins are loaded to parse and deserialize the acquired information to be processed, and to generate processed information corresponding to the information to be processed.

[0059] S205: Filter information.

[0060] Here, the filter is loaded, and the processed information marked with the filter tag is filtered through the filter. Among them, the filter tag is added to the processed information that does not need to be further processed, and the filter is loaded using the serial peripheral interface (SerialPeripheralInterface, SPI) (if the filter is not configured, it is not filtered by default) to filter the processed information that does not meet the conditions.

[0061] S206, converting the processed information into a corresponding storage format and storing it.

[0062] Here, the data processing system loads the configuration file of the data storage layer and generates the data source storage layer based on the configuration file. The data storage layer provides communication interfaces and default implementations with storage media such as Redis, Hbase, ElasticSearch, mysql, JMS, FileSystem, and hdfs. Furthermore, the processed information is converted into the storage format required by the corresponding storage medium through SPI and stored.

[0063] S207, recording a first amount of to-be-processed information obtained within a preset time period.

[0064] Here, the data processing system collects a first quantity of information to be processed that has been acquired within a preset time period to calculate the upstream production speed.

[0065] S208, recording a second amount of processed information obtained within a preset time period.

[0066] Here, the data processing system collects a second amount of processed information that has been acquired within a preset time period to calculate the downstream consumption speed.

[0067] S209, comparing the numerical values ​​of the first quantity and the second quantity.

[0068] Here, if the first number is greater than the value of the second number, the data processing flow continues to be executed.

[0069] S210, when the second number is smaller than the first number and the difference between the second number and the first number is smaller than a preset threshold, adjusting the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads.

[0070] Here, when the second number is less than the first number and the difference between the second number and the first number is less than a preset threshold, it can be determined that the downstream consumption speed is significantly lower than the upstream production speed. At this time, the change rate can be determined according to the upstream production speed and the downstream consumption speed, and the acquisition speed of the information to be processed and the number of the first thread, and / or the acquisition speed of the information to be processed and the number of the second thread can be adjusted.

[0071] The present application implements a method for data processing based on the above steps. Start the data processing system, load the configuration file, generate the corresponding data source interface and data storage layer according to the configuration file, and after completing the initialization, the data source interface will be used to pull the information to be processed, and then put it directly into the data channel layer with classification labels (such as orders, shopping carts, etc.); the data processing layer also constantly checks whether there is information to be processed in the data channel layer, and if so, pulls it in batches, and then loads different information processing plug-ins according to the type of information to be processed to deserialize and generate processed information, and uses SPI to load filters (if no filter is configured, it is not filtered by default), filters the processed information, and then puts it into the data channel layer again; the data storage layer constantly checks whether there is processed information in the data channel layer, and if so, pulls it in batches, and then converts the processed information into the storage format required for the corresponding storage through the data adaptation layer loaded by SPI, and finally writes it in batches to the storage medium.

[0072] Embodiment 300 of the present application also provides a data processing system. Figure 3 As shown, it is a schematic diagram of the system interaction in which the data processing provided by the embodiment 300 of the present application is located. Among them, the system includes a data source layer 301, a data channel layer 302, a data processing layer 303, a data adaptation layer 304 and a data storage layer 305, wherein the data source layer 301 is used to provide at least one data source interface and transmit the information to be processed through the data channel layer 302; the data processing layer 303 is used to process the information to be processed obtained through the data channel layer 302 to generate processed information; the data adaptation layer 304 is used to convert the processed information into a corresponding storage format; the data storage layer 305 is used to store the processed information in a storage medium corresponding to the storage format. The data processing system further includes a system management layer 306, which is used to interact with the data source layer 301 and the data channel layer 302 to record the first amount of information to be processed obtained within a preset time period, and to interact with the data channel layer 302 to record the second amount of processed information obtained within a preset time period; compare the numerical values ​​of the first amount and the second amount, and when the second amount is less than the first amount and the difference between the second amount and the first amount is less than a preset threshold, adjust the acquisition speed of the information to be processed and the first thread number, and / or the acquisition speed of the information to be processed and the second thread number through the data channel layer 302. The data processing system further includes a data filtering layer 307, which is used to load a filter and filter the processed information marked with a filter tag after being processed by the data processing layer 303 through the filter.

[0073] Specifically, the data source layer (Source) 301: This layer is responsible for the interaction between the system management layer 306 and the data source layer 301, and provides interfaces and implementations with various data sources (such as: JMS, Kafka, FileSystem, hdfsSystem, etc.).

[0074] Data channel layer (Channel) 302: This layer is a data channel that connects various components and is used to cache production data. The default implementation is heap memory (it can also be extended to other storage implementations), and off-heap memory storage can be enabled. In addition, the message processing speed is also monitored in the Channel: upstream production speed, downstream consumption speed, and memory usage. This monitoring information is reported to the SystemManager for tuning decisions.

[0075] Data processing layer (Handler) 303: The function of this layer implements the use of information processing plug-ins in different business domains to parse and deserialize the information to be processed.

[0076] Data filtering layer (Filter) 307: This layer is responsible for filtering messages, such as filtering categories, adding to cart messages, etc.

[0077] Data adaptation layer (Adapter) 304: This layer is responsible for message conversion, converting the unified message model into the format required by the corresponding storage medium.

[0078] Data storage layer (Sink) 305: This layer is responsible for the interaction between the system and various data storage media, encapsulates the model adapted to the storage medium, and provides communication interfaces and default implementations with storage media such as Redis, Hbase, ElasticSearch, mysql, JMS, FileSystem, and hdfs.

[0079] System Manager 306: This layer is responsible for system memory management, dynamic tuning and performance monitoring.

[0080] Furthermore, the system design adopts rule-based adaptive tuning. The system management layer (SystemManager) collects the upstream production speed and downstream consumption speed at intervals, and adjusts the size of upstream and downstream batches and the number of threads according to the change rate. At the same time, the off-heap memory is enabled. Among them, the decision whether to enable the off-heap memory is mainly determined by two rules: the number of times the message is blocked and the number of full GC within a certain period of time. In addition, the speed of communication with the external system is monitored, the batch size is adjusted, and the speed and efficiency are balanced. In addition, it provides external system monitoring information display and supports user-defined optimization settings.

[0081] In order to balance the problem of resources and efficiency, the system as a whole adopts the event-driven Reactor mode, and adopts asynchronous multi-threading in each module. The size of the thread pool is dynamically adjusted by SystemManager. Since the collection of SystemManager information has a lag, in order to prevent the system from being overwhelmed by traffic during peak hours, the memory channel model of Channel is implemented in the form of blocking queues. When there is a situation where downstream consumption is slow and upstream production is too fast, upstream production will be blocked to ensure that Channel will not overflow memory and cause system crashes; at the same time, the number of blocking times and the blocked upstream modules are recorded and reported to SystemManager to facilitate system adaptive optimization and adjustment.

[0082] The embodiments of the present application use a universal templated development process to quickly access common data sources and implement data calculation and storage. Only simple configuration is required to complete data reception, processing and storage. It can adapt to small and large data requirements, simplify the development process, reduce the cost of tuning and learning, and improve efficiency. It also has adaptive performance tuning, dynamic memory management, graphical interface monitoring information display and parameter setting.

[0083] Based on the same inventive concept, the embodiment 400 of the present application further provides a data processing device, wherein: Figure 4 As shown, the device comprises:

[0084] A loading module 41, used to load at least one data source interface;

[0085] An acquisition module 42, used to acquire information to be processed based on a data source interface;

[0086] A generating module 43 is used to load the information processing plug-in based on the classification label carried by the information to be processed, and generate processed information corresponding to the information to be processed after the information to be processed is processed;

[0087] The storage module 44 is used to convert the processed information into a corresponding storage format and store it.

[0088] In this embodiment, the specific functions and interaction modes of the loading module 41, the obtaining module 42, the generating module 43 and the storing module 44 can be found in Figure 1 The description of the corresponding embodiments will not be repeated here.

[0089] Optionally, the device further comprises:

[0090] The acquisition module 45 is used to record a first amount of information to be processed obtained within a preset time period, and to record a second amount of processed information obtained within a preset time period;

[0091] The adjustment module 46 is used to compare the numerical values ​​of the first number and the second number, and when the second number is smaller than the first number and the difference between the second number and the first number is smaller than a preset threshold, adjust the acquisition speed of the information to be processed and the number of the first thread, and / or the acquisition speed of the information to be processed and the number of the second thread.

[0092] Optionally, the device further comprises:

[0093] The filtering module 47 is used to load a filter and filter the processed information marked with a filtering tag through the filter.

[0094] like Figure 5 As shown, another embodiment 500 of the present application further provides a terminal device, including a processor 501, wherein the processor 501 is used to execute the steps of the above-mentioned data processing method. Figure 5 It can also be seen that the terminal device provided in the above embodiment further includes a non-transitory computer-readable storage medium 502, on which a computer program is stored, and the computer program executes the steps of the above data processing method when it is run by the processor 501. In practical applications, the terminal device can be one or more computers, as long as it includes the above computer-readable medium and processor.

[0095] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, and FLASH, etc. When the computer program on the storage medium is run, each step in the above-mentioned data processing method can be executed. In practical applications, the computer-readable medium can be included in the device / apparatus / system described in the above embodiments, or it can exist independently without being assembled into the device / apparatus / system. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed, each step in the above-mentioned data processing method can be executed.

[0096] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0097] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order marked in different drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0099] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still change the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these changes, modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for data processing, characterized in that: include: Load at least one data source interface; Acquire information to be processed based on the data source interface; Based on the classification label carried by the information to be processed, load the information processing plug-in, and generate processed information corresponding to the information to be processed after processing; Converting the processed information into a corresponding storage format and storing it; Among them, obtaining the information to be processed based on the data source interface includes: reading the configuration file of the data source interface, pulling the information to be processed from at least one data source interface through the read configuration file, and marking the information to be processed with a classification tag; wherein the classification tag is used to distinguish the business type of the message.

2. The method according to claim 1, characterized in that: After the step of acquiring the information to be processed based on the data source interface, the method further comprises: Recording a first quantity of the to-be-processed information acquired within a preset time period, and recording a second quantity of the processed information acquired within the preset time period; Compare the numerical values ​​of the first number and the second number, and when the second number is smaller than the first number and the difference between the second number and the first number is smaller than a preset threshold, adjust the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads.

3. The method according to claim 1, characterized in that Between the step of generating processed information corresponding to the information to be processed after being processed and the step of converting the processed information into a corresponding storage format and storing it, the method further comprises: A filter is loaded, and the processed information marked with a filter tag is filtered through the filter.

4. A data processing device, characterized in that: The device comprises: A loading module, used to load at least one data source interface; An acquisition module, used for acquiring information to be processed based on the data source interface; A generating module, configured to load an information processing plug-in based on the classification label carried by the information to be processed, and generate processed information corresponding to the information to be processed after the information to be processed is processed; A storage module, used to convert the processed information into a corresponding storage format and store it; The acquisition module is used to read the configuration file of the data source interface, pull the information to be processed from at least one data source interface through the read configuration file, and mark the information to be processed with a classification tag; wherein the classification tag is used to distinguish the business type of the message.

5. The device according to claim 4, characterized in that The device further comprises: A collection module, used to record a first amount of the to-be-processed information obtained within a preset time period, and to record a second amount of the processed information obtained within the preset time period; An adjustment module is used to compare the numerical values ​​of the first number and the second number, and when the second number is smaller than the first number and the difference between the second number and the first number is smaller than a preset threshold, adjust the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads.

6. The device according to claim 4, characterized in that The device further comprises: The filtering module is used to load a filter and filter the processed information marked with a filtering tag through the filter.

7. A data processing system, characterized in that: The system comprises a data source layer, a data channel layer, a data processing layer, a data adaptation layer and a data storage layer, wherein the data source layer is used to provide at least one data source interface, the data source interface is used to obtain information to be processed, and transmit the information to be processed through the data channel layer; The data processing layer is used to process the information to be processed obtained through the data channel layer to generate processed information; The data adaptation layer is used to convert the processed information into a corresponding storage format; The data storage layer is used to store the processed information in a storage medium corresponding to the storage format; Among them, obtaining the information to be processed based on the data source interface includes: reading the configuration file of the data source interface, pulling the information to be processed from at least one data source interface through the read configuration file, and marking the information to be processed with a classification label; wherein the classification label is used to distinguish the business type of the message.

8. The system according to claim 7, characterized in that The data processing system further comprises a system management layer, wherein the system management layer is configured to: interacting with the data source layer and the data channel layer to record a first amount of the to-be-processed information acquired within a preset time period, and interacting with the data channel layer to record a second amount of the processed information acquired within the preset time period; Compare the numerical values ​​of the first number and the second number, and when the second number is less than the first number and the difference between the second number and the first number is less than a preset threshold, adjust the acquisition speed of the information to be processed and the number of first threads, and / or the acquisition speed of the information to be processed and the number of second threads through the data channel layer.

9. The system according to claim 7, characterized in that The data processing system further comprises a data filtering layer, and the data filtering layer is used to load a filter and filter the processed information marked with a filtering tag after being processed by the data processing layer through the filter.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to perform each step in the data processing method according to any one of claims 1 to 3.

Images