Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a data processing method and device, computer equipment and a storage medium. According to the embodiment of the invention, the to-be-processed data meeting the preset requirement in the initial data can be calculated and analyzed, and the to-be-processed data not meeting the preset requirement can also be stored in the first region, pre-processed together with the to-be-stored data and then input into the second region serving as the data output region; according to the technical scheme, the initial data are comprehensively processed and analyzed and are finally input into the second area to be output, the situation that in the prior art, only data obtained after real-time analysis and calculation can be output, and consequently data output is limited is avoided, and then the utilization rate of the initial data is increased.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, computer equipment and storage medium. Background Art

[0002] Currently, FMEA is an important tool in quality management and reliability engineering for identifying potential failure modes and their impacts. However, in the process of collecting and storing FMEA data, the conventional operation is that after the data is collected, it is processed and analyzed in real time by Flink, and then stored and output to the data management platform for users to query.

[0003] However, this approach only stores data after real-time processing and analysis, resulting in limited output data. Summary of the invention

[0004] In view of the above problems, the present application provides a data processing method, device, computer equipment and storage medium, which are used to solve the problem in the prior art that only data after real-time processing and analysis is stored, resulting in limited output data. It can calculate and analyze the data to be processed in the initial data that meets the preset requirements, and can also store the data to be processed that does not meet the preset requirements in the first area, and pre-process them together with the data to be stored and input them into the second area as the data output area, thereby realizing comprehensive processing and analysis of the initial data, and finally inputting them into the second area for output, avoiding the situation in the prior art that only the data after real-time analysis and calculation can be output, resulting in limited data output, thereby improving the utilization rate of the initial data.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, the method comprising: based on an acquisition rule of initial data, dividing the initial data into data to be stored and data to be processed, and storing the data to be stored in a first area; performing computational analysis on the data to be processed that meets preset requirements, storing first target data obtained by the computational analysis in a second area, and storing the data to be processed that does not meet preset requirements in the first area; wherein the second area is an area for receiving data after computational processing; preprocessing the data in the first area, and inputting the preprocessed data into the second area; wherein the data in the first area includes the data to be stored and the data to be processed that does not meet preset requirements, and the second area is a data output area.

[0006] In an optional manner, the step of dividing the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and storing the data to be stored in the first area, further includes: collecting multi-source heterogeneous data; and performing format unification processing on the multi-source heterogeneous data to obtain initial data.

[0007] In an optional manner, before the step of collecting multi-source heterogeneous data, it also includes: configuring a plug-in according to a data source; wherein the plug-in is used to collect data in a corresponding data source; configuring data collection rules according to the data, and obtaining a data collection tool according to the data collection rules and the plug-in; wherein the data collection tool is used to collect multi-source heterogeneous data.

[0008] In an optional manner, the step of collecting multi-source heterogeneous data further includes: using multiple parallel threads to collect heterogeneous data from different data sources; if an abnormality is detected in any thread, the data source corresponding to the abnormal thread is evenly distributed to other threads.

[0009] In an optional manner, the collection rules include real-time collection and offline collection; the collection rules based on initial data divide the initial data into the steps of data to be stored and data to be processed, further including: using the initial data whose collection rules are real-time collection as data to be processed; using the initial data whose collection rules are offline collection as data to be stored; wherein the offline collection is to collect all data before the current time node.

[0010] In an optional manner, the step of preprocessing the data in the first area further includes: performing data cleaning and data conversion on the data in the first area to obtain transition data; performing aggregation analysis on the transition data to obtain second target data; wherein the second target data is the preprocessed data.

[0011] In an optional manner, after the step of preprocessing the data in the first area and inputting the preprocessed data into the second area, the method further includes: based on a preset query interface, transmitting the data in the second area to a remote platform to realize remote data query.

[0012] According to another aspect of an embodiment of the present application, a data processing device is provided, the device comprising: a data classification module, for dividing the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and storing the data to be stored in a first area; a calculation and analysis module, for performing calculation and analysis on the data to be processed that meets the preset requirements, storing the first target data obtained by the calculation and analysis in a second area, and storing the data to be processed that does not meet the preset requirements in the first area; wherein the second area is an area for receiving data after calculation and processing; a preprocessing module, for preprocessing the data in the first area, and inputting the preprocessed data into the second area; wherein the data in the first area includes the data to be stored and the data to be processed that does not meet the preset requirements, and the second area is a data output area.

[0013] According to another aspect of an embodiment of the present application, a computer device is provided, comprising: a controller; and a memory for storing one or more programs, wherein when the one or more programs are executed by the controller, the controller implements a data processing method as described in any one of the above claims.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a computer device / equipment to execute a data processing method as described in any one of the above claims.

[0015] The embodiment of the present application divides the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and stores the data to be stored in the first area; at the same time, the data to be processed that meets the preset requirements is calculated and analyzed, and the first target data obtained by the calculation and analysis is stored in the second area, and the data to be processed that does not meet the preset requirements is stored in the first area; finally, the data stored in the first area is preprocessed, and the preprocessed data is also input into the second area as the data output area. In this way, the data to be processed that meets the preset requirements in the initial data can be calculated and analyzed, and the data to be processed that does not meet the preset requirements can also be stored in the first area, and then preprocessed together with the data to be stored and input into the second area as the data output area, so as to achieve comprehensive processing and analysis of the initial data, and finally all are input into the second area for output, avoiding the situation in the prior art that only the data after real-time analysis and calculation can be output, resulting in limited data output, thereby improving the utilization rate of the initial data.

[0016] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0018] Figure 1 A schematic diagram showing a flow chart of an embodiment of a data processing method provided by the present application;

[0019] Figure 2 A flow chart showing another embodiment of a data processing method provided by the present application is shown;

[0020] Figure 3 A flow chart showing another embodiment of a data processing method provided by the present application is shown;

[0021] Figure 4 A flow chart showing another embodiment of a data processing method provided by the present application is shown;

[0022] Figure 5 A flow chart showing another embodiment of a data processing method provided by the present application is shown;

[0023] Figure 6 A flow chart showing another embodiment of a data processing method provided by the present application is shown;

[0024] Figure 7 A schematic diagram of a data code for storing data in a first area in an embodiment provided by the present application is shown;

[0025] Figure 8 A schematic diagram of a data code for storing data in the second area in an embodiment provided by the present application is shown;

[0026] Fig. 9 A schematic diagram showing the structure of an embodiment of a data processing device provided by the present application is shown;

[0027] Fig.10 A schematic structural diagram of an embodiment of a computer device provided by the present application is shown. DETAILED DESCRIPTION

[0028] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0029] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0030] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0031] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0032] FMEA is a systematic analysis method used to identify and solve potential failure modes and their effects in product design and manufacturing processes. Currently, the collection and storage of FMEA data usually rely on a variety of tools and platforms, such as spreadsheets, database management systems, and dedicated FMEA software. However, these methods have problems such as data silos, high integration difficulty, and poor real-time performance. (FMEA: Failure Mode and Effect Analysis).

[0033] The above problems are caused by the following reasons:

[0034] 1. Data sources are diverse and scattered, and data formats are not unified, making data collection difficult.

[0035] 2. Existing data conversion tools are inefficient and cannot meet the needs of offline and real-time data processing at the same time.

[0036] 3. The data storage system performs poorly when processing large-scale data queries and cannot provide efficient query and analysis capabilities.

[0037] In order to solve the above problems, specific embodiments are described below.

[0038] Figure 1 A flowchart of an embodiment of a data processing method of the present application is shown, and the method is executed by a computer device. Figure 1 As shown, the method comprises the following steps:

[0039] Step S110: Based on the collection rule of the initial data, the initial data is divided into data to be stored and data to be processed, and the data to be stored is stored in the first area.

[0040] The collection rules include real-time collection and offline collection. Real-time collection is to collect data from the data source in real time, while offline collection is to collect all data in the data source in the previous period of time in one batch. The first area is used to store data that has not been calculated.

[0041] Step S120: performing calculation analysis on the data to be processed that meets the preset requirements, storing the first target data obtained by the calculation analysis in the second area, and storing the data to be processed that does not meet the preset requirements in the first area.

[0042] The second area is an area for receiving data after calculation and processing.

[0043] Specifically, the preset requirements here are the real-time computing requirements customized by the user in the flink calculation. The calculation and analysis are performed in the customized flink, and the first target data obtained by the calculation and analysis is stored in the second area. Since some of the data to be processed still do not need real-time calculation and analysis, they are directly stored in the first area.

[0044] Step S130: pre-processing the data in the first area, and inputting the pre-processed data into the second area.

[0045] The data in the first area includes data to be stored and data to be processed that does not meet preset requirements, and the second area is a data output area.

[0046] Specifically, the second area is connected to the remote platform / terminal via a preset query interface, so that the data output by the second area can be queried. Preprocessing includes data cleaning, data conversion, and aggregate analysis.

[0047] Beneficial effect: The embodiment of the present application divides the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and stores the data to be stored in the first area; at the same time, the data to be processed that meets the preset requirements is calculated and analyzed, and the first target data obtained by the calculation and analysis is stored in the second area, and the data to be processed that does not meet the preset requirements is stored in the first area; finally, the data stored in the first area is preprocessed, and the preprocessed data is also input into the second area as the data output area. Thus, the data to be processed that meets the preset requirements in the initial data can be calculated and analyzed, and the data to be processed that does not meet the preset requirements can also be stored in the first area, and then preprocessed together with the data to be stored and input into the second area as the data output area, realizing comprehensive processing and analysis of the initial data, and finally inputting them into the second area for output, avoiding the situation in the prior art that only the data after real-time analysis and calculation can be output, resulting in limited data output, thereby improving the utilization rate of the initial data.

[0048] This embodiment classifies the initial data, and then selects appropriate data to be processed for analysis and calculation according to the preset requirements, and stores all the data that have not been analyzed and calculated in the first area for preprocessing, and finally obtains the processed data in the second area. The data to be processed that meets the preset requirements is processed in real time, while the data to be processed that does not meet the preset requirements and the data to be stored that have not been analyzed and calculated are processed offline. The combination of the two achieves the effect of offline + real-time data processing and improves the data conversion rate.

[0049] In some embodiments, Figure 2 As shown, before step S110, the following steps are also included:

[0050] Step S210: Collect multi-source heterogeneous data.

[0051] Among them, multi-source heterogeneous data refers to data with different structures from different data sources.

[0052] Specifically, data sources include PMS (production management system), QMS (quality management system), PLM (product life cycle system), standards and regulations, and other systems.

[0053] Step S220: Perform format unification processing on multi-source heterogeneous data to obtain initial data.

[0054] Specifically, the collected data is converted, cleaned, aggregated, and other operations are performed to ensure the uniformity and quality of the data format.

[0055] Beneficial effect: This embodiment collects multi-source heterogeneous data and performs format unification processing on the collected multi-source heterogeneous data to obtain initial data, thereby making subsequent initial data processing more convenient due to the unified format.

[0056] In some embodiments, Figure 3 As shown, before step S210, the following steps are also included:

[0057] Step S310: configuring the plug-in according to the data source.

[0058] Among them, the plug-in is used to collect data in the corresponding data source;

[0059] Specifically, configure data source plug-ins, such as MySQL (relational database), Kafka (message queue), etc.

[0060] ①MySQL data source plug-in configuration: configure the host address, port number, database name and authentication information (user name and password)

[0061] ②Kafka data source plug-in configuration: set the boot server address, topic name and consumer group ID.

[0062] Step S320: configuring data collection rules according to the data, and obtaining a data collection tool according to the data collection rules and the plug-in.

[0063] Among them, the data collection tool is used to collect multi-source heterogeneous data.

[0064] Specifically, configure data collection rules, including data filtering, field mapping, etc. (1) Data filtering rules: define filtering conditions, such as filtering out records that meet the requirements based on severity and occurrence.

[0065] (2) Field mapping rules: Perform field-level mapping to map the fields in the original data to the corresponding fields in the target data model to ensure the consistency and availability of the data structure.

[0066] Data collection tools can be formed by building various collection rules and plug-ins on the SeaTunnel platform. SeaTunnel is a simple, easy-to-use, high-performance data processing product that can handle massive amounts of data. SeaTunnel supports plug-ins for multiple data sources. By configuring different plug-ins, it can achieve unified collection of structured and unstructured data.

[0067] Beneficial effects: By configuring plug-ins and data collection rules, a dedicated data collection tool is built for data collection. It can quickly realize unified collection and structural conversion of various structured data from various data sources. This solves the problem of data collection difficulties caused by diverse and scattered data sources and inconsistent data formats, and improves the overall data collection and conversion efficiency.

[0068] In some embodiments, Figure 4 As shown, step S210 further includes:

[0069] Step S211: using multiple parallel threads to collect heterogeneous data from different data sources.

[0070] Specifically, data is collected in parallel by multiple threads, and the number of data sources corresponding to each thread is similar.

[0071] Step S212: If it is detected that any thread is abnormal, the data source corresponding to the abnormal thread is evenly distributed to other threads.

[0072] Specifically, when a thread exception is detected, the data source corresponding to the abnormal thread is evenly distributed to other normal threads.

[0073] Beneficial effect: This embodiment ensures the number of collections through multi-threaded parallel collection, and when a thread exception is detected, it can also evenly distribute the data source corresponding to the abnormal thread to other threads, thereby ensuring the performance of the overall thread collection while improving feasibility.

[0074] In some embodiments, Figure 5 As shown, the collection rules include real-time collection and offline collection; the collection rules based on the initial data in step S110, which divides the initial data into the steps of data to be stored and data to be processed, further include:

[0075] Step S111: taking the initial data collected in real time as data to be processed.

[0076] Specifically, the data to be processed is the data from the data source collected in real time, and the initial data is obtained after unified processing. Real-time is a relatively offline concept, and real-time collection is collected all the time. As long as new data appears, it will be collected. It is mentioned in the above embodiment that the data to be processed is essentially further screened and processed according to the preset requirements, and those that meet the preset requirements are calculated and analyzed, and those that do not meet the requirements are stored. Therefore, the preset requirement is to perform real-time calculations on flink (a framework and distributed processing engine for state calculations on unbounded and bounded data streams), so the data collected in real time must be the first choice as the data to be processed.

[0077] Step S112: taking the initial data collected offline according to the collection rule as data to be stored.

[0078] Among them, offline collection is to collect all data before the current time node.

[0079] Specifically, there is a current time node when offline collection is triggered, and offline collection collects data for a period of time. Offline collection is to ensure the integrity of the data and the data that can be analyzed in units of a period of time, so there is no need for real-time analysis and calculation, so the first choice is the data to be stored.

[0080] Beneficial effects: According to the collection rules, the initial data is quickly divided into data to be processed and data to be stored, which corresponds to the dual processing methods of real-time and offline. The classification link is reduced during subsequent processing, which improves the efficiency of subsequent data processing.

[0081] In some embodiments, Figure 6 As shown, the step S130 of preprocessing the data in the first region further includes:

[0082] Step S131: clean and convert the data in the first region to obtain transition data.

[0083] Specifically, data in the first region is cleaned and converted to obtain transition data.

[0084] Step S132: Perform aggregation analysis on the transition data to obtain second target data.

[0085] The second target data is preprocessed data.

[0086] Specifically, the aggregation analysis is performed according to the user's customized requirements, and obtaining the second target data also indicates that the offline data processing is completed.

[0087] Beneficial effect: By preprocessing the data in the first area, offline processing of the data in the first area is achieved.

[0088] In some embodiments, after the step of preprocessing the data in the first region and inputting the preprocessed data into the second region in step S130, the method further includes:

[0089] Based on a preset query interface, the data in the second area is transmitted to the remote platform to realize remote data query.

[0090] Specifically, the preset query interface includes a RESTful API or a SQL interface, and the remote platform can query and analyze the data in the second area through the preset query interface.

[0091] Beneficial effects: This embodiment allows the data in the second area to be transmitted to the remote platform through a preset query interface to realize remote data query, which solves the technical problem that the existing data storage system has poor performance when processing large-scale data queries and cannot provide efficient query analysis capabilities.

[0092] In some embodiments, taking Mysql data of a PMS business system, 2) log file data and sensor data of a QMS business system as examples, detailed instructions are performed according to the steps of the above embodiments.

[0093] First, data collection is carried out. Taking the SeaTunnel platform as an example, SeaTunnel is used to efficiently collect data offline and in real time from different data sources (such as databases, log files, sensor data, etc.). SeaTunnel converts, cleans, and aggregates the collected data to ensure the uniformity and quality of the data format.

[0094] SeaTunnel supports plug-ins for multiple data sources. By configuring different plug-ins, it can achieve unified collection of structured and unstructured data. Taking FMEA data as an example, the collection process is as follows:

[0095] (1) Configure data source plug-ins, such as MySQL, Kafka, etc.

[0096] ①MySQL data source plug-in configuration: configure the host address, port number, database name, and authentication information (user name and password).

[0097] ②Kafka data source plug-in configuration: set the boot server address, topic name and consumer group ID.

[0098] (2) Configure data collection rules, including data filtering, field mapping, etc.

[0099] ① Data filtering rules: define filtering conditions, such as filtering out records that meet the requirements based on severity and occurrence.

[0100] ②Field mapping rules: Perform field-level mapping to map the fields in the original data to the corresponding fields in the target data model to ensure the consistency and availability of the data structure.

[0101] The sample data collected is as follows:

[0102] (1) Mysql data of the PMS business system, as shown in Table 1 below:

[0103] Table 1 Mysql data example of PMS business system

[0104]

[0105]

[0106] (2) Log file data of the QMS business system, as shown in Table 2 below:

[0107] Table 2 Example of log file data of QMS business system

[0108] Timestamp Component Event Details 2023-06-01 12:00:00 Pump Maintenance Routine check 2023-06-02 14:30:00 Valve Failure Stuck due to rust Timestamp Component Event Details

[0109] (4) Sensor data, as shown in Table 3 below:

[0110] Table 3 Sensor data example

[0111] SensorID Component Measurement Value Unit Timestamp S001 Pump Pressure 3.5 bar 2023-06-01 12:00:00 S002 Valve Flow Rate 1.2 <![CDATA[m 3 / h]]> 2023-06-02 14:30:00

[0112] Next comes data processing. SeaTunnel uses its efficient processing engine to process data in multiple steps:

[0113] (1) Data cleaning: filter out invalid data and fill in missing values.

[0114] For example, delete the records whose Severity or RPN is empty.

[0115] Taking the Mysql data of the PMS business system in Table 1 as an example, the data sample after cleaning is shown in Table 4 below:

[0116] Table 4 Cleaning data sample

[0117]

[0118] (2) Data conversion: Convert data in different formats into a unified format, perform field mapping and type conversion.

[0119] For example, the data types of the "Severity", "Occurrence", and "Detection" fields are converted from strings to integers (it is not obvious here, but the field types are actually converted).

[0120] Taking the Mysql data of the PMS business system in Table 1 as an example, after data cleaning and data conversion, the data sample obtained is shown in Table 5 below:

[0121] Table 5. Sample data after conversion

[0122]

[0123] After the above processing, the data obtained is the initial data, and the initial data is distinguished and divided into data to be processed and data to be stored.

[0124] The first area, namely the ODS layer (Operational Data Store): is used to store the original data collected from the data source. It is basically consistent with the original data, but has been processed in a unified format. There is no difference in the example.

[0125] Sample data: Same as the original data sample.

[0126] ①Mysql data, as shown in Table 6:

[0127] Table 6 Example of MySQL data in the first area

[0128]

[0129] ②Log file data, as shown in Table 7:

[0130] Table 7 Example of log file data in the first area

[0131]

[0132]

[0133] ③Sensor data, as shown in Table 8 below:

[0134] Table 8 Example of sensor data in the first area

[0135] SensorID Component Measurement Value Unit Timestamp S001 Pump Pressure 3.5 bar 2023-06-01 12:00:00 S002 Valve Flow Rate 1.2 <![CDATA[m 3 / h]]> 2023-06-02 14:30:00

[0136] DWD layer (Data Warehouse Detail): stores the detailed data after cleaning and conversion, i.e. the transitional data obtained after executing step S131 in the above embodiment.

[0137] Sample data: Same as the cleaned and transformed data sample.

[0138] Table 9 Transition data example

[0139]

[0140] DWS layer (Data Warehouse Summary): stores data after aggregation analysis, i.e., the second target data example obtained after step S132 in the above embodiment.

[0141] Sample data: Same as the aggregated data sample.

[0142] Table 10 Second target data example

[0143] Failure Mode Average RPN Leakage 108 Stuck 80

[0144] The second target data is also stored in the second area.

[0145] As for the real-time processing part (i.e. the part of data to be processed), we first integrate the Kafka stream data source through SeaTunnel to retain the original data format. The real-time data sample data code is as follows Figure 7 Then enter the second area, the real-time processing computing layer (ADS layer): the real-time processed data is cleaned, converted and aggregated, and then directly stored in the corresponding real-time data table in the second area. The sample data code after real-time processing is as follows Figure 8 shown.

[0146] The execution process of this embodiment is as follows:

[0147] 1. Data is collected from various data sources (such as MySQL, MongoDB, Kafka, etc.) through SeaTunnel;

[0148] 2. SeaTunnel is responsible for data conversion and cleaning. Part of the processed data can be directly sent to the first area, and part of the data can flow into Kafka;

[0149] 3. If real-time analysis and calculation are required for the data flowing into Kafka, it will flow into Flink, and after real-time calculation, the real-time calculation results will be written to the second area. Otherwise, a SeaTunnel task can be configured to write the Kafka data into the first area.

[0150] 4. The data written into the first area can be layered into a unified data warehouse model;

[0151] 5. Finally, the downstream FMEA application platform accesses Doris through the RESTful API or SQL interface for data query and analysis.

[0152] Beneficial effect: The embodiment of the present application divides the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and stores the data to be stored in the first area; at the same time, the data to be processed that meets the preset requirements is calculated and analyzed, and the first target data obtained by the calculation and analysis is stored in the second area, and the data to be processed that does not meet the preset requirements is stored in the first area; finally, the data stored in the first area is preprocessed, and the preprocessed data is also input into the second area as the data output area. Thus, the data to be processed that meets the preset requirements in the initial data can be calculated and analyzed, and the data to be processed that does not meet the preset requirements can also be stored in the first area, and then preprocessed together with the data to be stored and input into the second area as the data output area, realizing comprehensive processing and analysis of the initial data, and finally inputting them into the second area for output, avoiding the situation in the prior art that only the data after real-time analysis and calculation can be output, resulting in limited data output, thereby improving the utilization rate of the initial data.

[0153] Fig. 9 A schematic diagram of the structure of an embodiment of a data processing device of the present application is shown. Fig. 9 As shown, the device 400 includes: a data classification module 410, a calculation and analysis module 420 and a preprocessing module 430.

[0154] A data classification module 410 is used to classify the initial data into data to be stored and data to be processed based on the collection rule of the initial data, and store the data to be stored in the first area;

[0155] The calculation and analysis module 420 is used to perform calculation and analysis on the data to be processed that meets the preset requirements, store the first target data obtained by the calculation and analysis in the second area, and store the data to be processed that does not meet the preset requirements in the first area; wherein the second area is an area for receiving the data after calculation and processing;

[0156] The preprocessing module 430 is used to preprocess the data in the first area and input the preprocessed data into the second area; wherein the data in the first area includes data to be stored and data to be processed that does not meet preset requirements, and the second area is a data output area.

[0157] Beneficial effect: The embodiment of the present application divides the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and stores the data to be stored in the first area; at the same time, the data to be processed that meets the preset requirements is calculated and analyzed, and the first target data obtained by the calculation and analysis is stored in the second area, and the data to be processed that does not meet the preset requirements is stored in the first area; finally, the data stored in the first area is preprocessed, and the preprocessed data is also input into the second area as the data output area. Thus, the data to be processed that meets the preset requirements in the initial data can be calculated and analyzed, and the data to be processed that does not meet the preset requirements can also be stored in the first area, and then preprocessed together with the data to be stored and input into the second area as the data output area, realizing comprehensive processing and analysis of the initial data, and finally inputting them into the second area for output, avoiding the situation in the prior art that only the data after real-time analysis and calculation can be output, resulting in limited data output, thereby improving the utilization rate of the initial data.

[0158] It should be noted that the data processing device provided in the above embodiment and the data processing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0159] Fig.10 A structural schematic diagram of an embodiment of a computer device of the present application is shown, which shows a structural schematic diagram of a computer system of a computer device suitable for implementing an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the computer device.

[0160] See also Fig.10 As shown, the computer device includes: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the above-mentioned data processing method is executed.

[0161] Please continue reading Fig.10As shown, the computer system 500 of the computer device includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503, such as executing the method in the above embodiment. In RAM503, various programs and data required for system operation are also stored. CPU 501, ROM 502 and RAM 503 are connected to each other through bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0162] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0163] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the system of the present application are executed.

[0164] Another aspect of the present application provides a computer-readable storage medium, in which at least one executable instruction is stored. When the executable instruction is executed on a computer device / equipment, the computer device / equipment executes a data processing method as described above.

[0165] Beneficial effect: The embodiment of the present application divides the initial data into data to be stored and data to be processed based on the collection rules of the initial data, and stores the data to be stored in the first area; at the same time, the data to be processed that meets the preset requirements is calculated and analyzed, and the first target data obtained by the calculation and analysis is stored in the second area, and the data to be processed that does not meet the preset requirements is stored in the first area; finally, the data stored in the first area is preprocessed, and the preprocessed data is also input into the second area as the data output area. Thus, the data to be processed that meets the preset requirements in the initial data can be calculated and analyzed, and the data to be processed that does not meet the preset requirements can also be stored in the first area, and then preprocessed together with the data to be stored and input into the second area as the data output area, realizing comprehensive processing and analysis of the initial data, and finally inputting them into the second area for output, avoiding the situation in the prior art that only the data after real-time analysis and calculation can be output, resulting in limited data output, thereby improving the utilization rate of the initial data.

[0166] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0167] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0169] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0170] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0171] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A data processing method, It is characterized in that The method comprises: Based on the collection rule of the initial data, the initial data is divided into data to be stored and data to be processed, and the data to be stored is stored in the first area; Performing calculation and analysis on the data to be processed that meets the preset requirements, storing the first target data obtained by the calculation and analysis in the second area, and storing the data to be processed that does not meet the preset requirements in the first area; wherein the second area is an area for receiving the data after calculation and processing; The data in the first area is preprocessed, and the preprocessed data is input into the second area; wherein the data in the first area includes the data to be stored and the data to be processed that does not meet the preset requirements, and the second area is a data output area.

2. The method according to claim 1, It is characterized in that Before the step of dividing the initial data into data to be stored and data to be processed based on the collection rule of the initial data, and storing the data to be stored in the first area, the method further includes: Collect multi-source heterogeneous data; The multi-source heterogeneous data are processed into a unified format to obtain initial data.

3. The method according to claim 2, It is characterized in that Before the step of collecting multi-source heterogeneous data, the method further includes: Configure a plug-in according to a data source; wherein the plug-in is used to collect data in a corresponding data source; A data collection rule is configured according to the data, and a data collection tool is obtained according to the data collection rule and the plug-in; wherein the data collection tool is used to collect multi-source heterogeneous data.

4. The method according to claim 2, It is characterized in that The step of collecting multi-source heterogeneous data further includes: Use multiple parallel threads to collect heterogeneous data from different data sources; If an abnormality is detected in any thread, the data source corresponding to the abnormal thread is evenly distributed to other threads.

5. The method according to claim 1, It is characterized in that The collection rules include real-time collection and offline collection; The step of dividing the initial data into data to be stored and data to be processed based on the collection rule of the initial data further includes: The initial data collected in real time according to the collection rule is used as the data to be processed; The initial data collected under the collection rule of offline is used as the data to be stored; wherein, the offline collection is to collect all data before the current time node.

6. The method according to claim 1, It is characterized in that The step of preprocessing the data in the first region further comprises: Performing data cleaning and data conversion on the data in the first area to obtain transition data; Aggregate and analyze the transition data to obtain second target data; wherein the second target data is preprocessed data.

7. The method according to claim 1, It is characterized in that After the step of preprocessing the data in the first area and inputting the preprocessed data into the second area, the method further includes: Based on a preset query interface, the data in the second area is transmitted to a remote platform to implement remote data query.

8. A data processing device, It is characterized in that The device comprises: A data classification module, configured to classify the initial data into data to be stored and data to be processed based on a collection rule of the initial data, and store the data to be stored in the first area; A calculation and analysis module, used for performing calculation and analysis on the data to be processed that meets the preset requirements, storing the first target data obtained by the calculation and analysis in the second area, and storing the data to be processed that does not meet the preset requirements in the first area; wherein the second area is an area for receiving the data after calculation and processing; A preprocessing module is used to preprocess the data in the first area and input the preprocessed data into the second area; wherein the data in the first area includes the data to be stored and the data to be processed that does not meet the preset requirements, and the second area is a data output area.

9. A computer device, It is characterized in that include: Controller; A memory for storing one or more programs. When the one or more programs are executed by the controller, the controller implements a data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device / equipment, the computer device / equipment executes the operation of a data processing method as described in any one of claims 1 to 7.

Images