Data processing method, device and server

Abstract

The present invention provides a data processing method, device and server, firstly determining delayed data from the acquired data based on the time information of the acquired data; then processing the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data; further fusing the intermediate result corresponding to the delayed data and the processing result of normal data corresponding to the delayed data, thereby obtaining a final processing result corresponding to the delayed data and the normal data. The present invention obtains an intermediate result by processing the delayed data, and then fusing the intermediate result and the processing result obtained based on the normal data to obtain a final processing result, thereby improving data processing efficiency and saving computing resources.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device and server. Background Art

[0002] In related technologies, due to abnormal collection of edge nodes or abnormal jitter of communication networks, some data may be delayed in being sent to the data center. The data center usually processes the data previously sent normally by the edge nodes and the delayed data as a whole, and calculates the corresponding business indicators based on the preset business needs. This method will repeatedly calculate a large amount of normally sent business data, which wastes computing resources and is costly. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a data processing method, device and server to avoid repeated calculation of normal data, improve data processing efficiency and reduce processing costs.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, comprising: determining delayed data from the acquired data based on time information of the acquired data; wherein the delayed data includes: data that arrives at the current device after exceeding a specified time period after the data is generated; processing the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data; fusing the intermediate result corresponding to the delayed data and the processing result of normal data corresponding to the delayed data to obtain a final processing result corresponding to the delayed data and the normal data; wherein the normal data includes: data that arrives at the current device within a specified time period after the data is generated.

[0005] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation scheme of the first aspect, wherein the above-mentioned time information includes the time when the data is generated and the time when the data arrives at the current device; based on the time information of the acquired data, the step of determining delayed data from the acquired data includes: judging whether the time difference between the time when the acquired data arrives at the current device and the time when the data is generated is greater than a specified time period; if it is greater than the specified time period, determining that the acquired data is delayed data.

[0006] In combination with the first possible implementation of the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein the time information of the acquired data is determined by: receiving a data log sent by a preset edge node device; recording the arrival time of the data log at the current device; parsing the data log to obtain the generation time of the data in the data log; and determining the arrival time of the data log as the time when the data in the data log arrives at the current device.

[0007] In combination with the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the above-mentioned time information includes the time when the data is generated and the time when the data arrives at the current device; the processing result of normal data corresponding to the delayed data is obtained by: judging whether the time difference between the time when the acquired data arrives at the current device and the time when the data is generated is less than or equal to the specified time period; if it is less than or equal to the specified time period, determining that the acquired data is normal data; based on a preset data processing method, processing the normal data to obtain the processing result corresponding to the normal data.

[0008] In combination with the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the above-mentioned time information includes the generation time of the data; before the step of fusing the intermediate results corresponding to the delayed data and the processing results of the normal data corresponding to the delayed data to obtain the final processing results corresponding to the delayed data and the normal data, the method also includes: determining the normal data with the same data type as the delayed data and in the same time period as the generation time of the delayed data as the normal data corresponding to the delayed data.

[0009] In combination with the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein, before the step of fusing the intermediate results corresponding to the delayed data and the processing results of the normal data corresponding to the delayed data to obtain the final processing results corresponding to the delayed data and the normal data, the method further includes: if new delayed data is received, updating the intermediate results corresponding to the delayed data based on the new delayed data.

[0010] In combination with the fifth possible implementation of the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein the step of updating the intermediate result corresponding to the delay data based on the new delay data includes: updating the new delay data into the delay data to obtain updated delay data; based on a preset data processing method, processing the updated delay data to obtain the intermediate result corresponding to the updated delay data.

[0011] In combination with the first aspect, an embodiment of the present invention provides a seventh possible implementation method of the first aspect, wherein the above-mentioned data processing methods include multiple methods; each data processing method corresponds to a business requirement; based on a preset data processing method, the delayed data is processed to obtain an intermediate result corresponding to the delayed data, comprising: according to different business requirements, the delayed data is processed based on the data processing method corresponding to the business requirement to obtain an intermediate result corresponding to the delayed data.

[0012] In combination with the first aspect, an embodiment of the present invention provides an eighth possible implementation of the first aspect, wherein the above-mentioned acquired data includes multiple information; the data processing method includes summing specified information; based on the preset data processing method, the delayed data is processed to obtain an intermediate result corresponding to the delayed data, including: extracting specified information from multiple information in the delayed data; summing the extracted specified information to obtain a summed result; and determining the summed result as the intermediate result corresponding to the delayed data.

[0013] In a second aspect, an embodiment of the present invention further provides a data processing device, comprising: a delayed data determination module, used to determine delayed data from the acquired data based on the time information of the acquired data; wherein the delayed data includes: data that arrives at the current device after exceeding a specified time period after the data is generated; a delayed data processing module, used to process the delayed data based on a preset data processing method, and obtain an intermediate result corresponding to the delayed data; a data fusion module, used to fuse the intermediate result corresponding to the delayed data and the processing result of normal data corresponding to the delayed data, and obtain a final processing result corresponding to the delayed data and the normal data; wherein the normal data includes: data that arrives at the current device within a specified time period after the data is generated.

[0014] In a third aspect, an embodiment of the present invention further provides a server, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above data processing method.

[0015] In a fourth aspect, an embodiment of the present invention further provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned data processing method.

[0016] The above-mentioned data processing method, device and server first determine the delayed data from the acquired data based on the time information of the acquired data; then process the delayed data based on the preset data processing method to obtain the intermediate result corresponding to the delayed data; further fuse the intermediate result corresponding to the delayed data and the processing result of the normal data corresponding to the delayed data, so as to obtain the final processing result corresponding to the delayed data and the normal data. This method obtains the intermediate result by processing the delayed data, and then fuses the intermediate result and the processing result obtained based on the normal data to obtain the final processing result, thereby improving the data processing efficiency and saving computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 An application scenario of a data processing method provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of a data processing method provided by an embodiment of the present invention;

[0020] Figure 3 A flowchart of another data processing method provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] In related technologies, after receiving data sent by edge nodes, the data center will calculate the business indicators corresponding to the data based on the preset business requirements. For example, when a user logs in to a set website through a personal computer at a certain time, the personal computer sends the access record to the data center, and the data center counts the number of visits to the website in a certain period of time. For ease of understanding, Figure 1 The application scenario is shown, which includes a data center and multiple edge nodes; the data center is connected to multiple edge nodes to form a content delivery network (Content Delivery Network, referred to as "CDN"). Among them, the data center can also be called a central server, and the edge node can also be called an edge node device; the edge node device can be a personal computer, industrial equipment, etc.

[0025] However, due to reasons such as abnormal collection of edge nodes or abnormal jitter of the communication network, a small amount of data may be delayed in being sent to the data center. For example, the access record generated by a user visiting the website at 10 o'clock in the morning may be sent to the data center together with the access record generated at 12 o'clock due to the above reasons. In response to this situation, after obtaining the delayed data, the data center usually processes the data previously sent normally by the edge node and the delayed data as a whole data to calculate the corresponding business indicators. This method will repeatedly calculate a large amount of normally sent data, waste computing resources, and have a high cost.

[0026] Based on this, a data processing method, device and server provided in an embodiment of the present invention can be used in the data processing process of various business systems, such as sales business systems, website business systems, etc.

[0027] To facilitate understanding of this embodiment, a data processing method disclosed in an embodiment of the present invention is first introduced in detail.

[0028] See first Figure 2 A flow chart of a data processing method shown; the method comprises the following steps:

[0029] Step S200, based on the time information of the acquired data, determining delayed data from the acquired data; wherein the delayed data includes: data that arrives at the current device after the data is generated and exceeds a specified time period.

[0030] The above data can be sent from the edge node to the data center, and the current device can refer to the data center. The time information of the data can include the time when the data was generated and the time when the data center received the data. Usually, in order to enable the data center to understand the working status or business situation of the edge node in real time, the edge node will send data to the data center at a set frequency, or the edge node will send data to the data center at an agreed time.

[0031] The above-mentioned specified time period can be a time period with the endpoint being the time when the edge node sends data twice adjacently, or a time period with the time difference when the data center receives data twice adjacently. It can also be limited to the length of the specified time period without limiting the endpoint of the specified time period; for example, the length of the specified time period can be the same as the time difference when the edge node sends data twice adjacently. Due to the high transmission speed of network data, the time when the data center receives the data can be approximated as the time when the edge node sends the data. Data whose time difference between the time when the data arrives at the data center and the time when it is generated is greater than the above-mentioned specified time period can be regarded as delayed data.

[0032] Under normal circumstances, after data is generated in an edge node, the data will be sent to the data center within a specified time period after it is generated. For example, assuming that after the edge node sends data to the data center at 10 o'clock, it will send data to the data center again at 10:30; then the data generated in the edge node between 10 o'clock and 10:30 should be sent to the data center at 10:30, and the specified time period for the data received by the data center this time should be from 10 o'clock to 10:30. However, due to abnormal data transmission or network problems, the data received by the data center may include data generated at 9:30, which arrives at the data center beyond the specified time period, and the data can be determined as delayed data.

[0033] In some cases, a maximum delay time threshold is also set, that is, when data arrives at the current device after exceeding the specified time period, but the time difference between its arrival time and the time of generation has exceeded the maximum delay time threshold, the data can be discarded without related processing. For example, you want to count the visit records of a website from 10 am to 12 pm on a certain day, but receive several visit records of the website in this time period two weeks later. If the maximum delay time threshold is one week, the time difference between the arrival time and the time of generation of these data has exceeded the maximum delay time threshold, and these data can be discarded.

[0034] Step S202: Process the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data.

[0035] The above data processing method is usually pre-set by relevant personnel in the data center, and is related to the type of data received and the information of the edge node that you want to know. For example, when the edge node is a personal computer, you want to know the traffic usage of personal computers in a certain area within a certain time period in real time, such as total traffic, average traffic of personal computers, etc.; at this time, the data received by the data center is the traffic data of multiple personal computers in the area within the time period. For example, when the content that the data center wants to know is the sales of a certain mobile phone, the data it receives can be the mobile phone sales records of various retail outlets, etc.

[0036] In the above case, the data processing method can be a summation process, firstly summing up the traffic in the traffic data of the time period that is received and sent normally to obtain a preliminary total traffic result; when delayed data is received, the traffic in the delayed data can also be summed up, and the summed result can be used as the intermediate result corresponding to the delayed data. The data processing method can also be an average value, when several delayed data are received, the traffic in the delayed data can be averaged, and the average value result and the number of delayed data can be used as the intermediate result corresponding to the delayed data. There can be multiple data processing methods, and the same delayed data can be processed multiple times based on different data processing methods to obtain multiple intermediate results corresponding to the delayed data corresponding to the data processing method.

[0037] When the delayed data is received again, the delayed data received this time can be processed together with the delayed data received previously according to a preset processing method to generate an intermediate result corresponding to all the delayed data. The delayed data received this time can also be processed according to a preset processing method to obtain an intermediate result corresponding to the delayed data this time, and then the intermediate result corresponding to the delayed data this time and the intermediate result corresponding to the delayed data obtained by the previous processing are merged to obtain an intermediate result corresponding to all the delayed data. Both of the above methods are feasible. Considering the simplicity of the data processing flow, the first processing method is usually used when the delayed data is received again.

[0038] Step S204, fusing the intermediate results corresponding to the delayed data and the processing results of the normal data corresponding to the delayed data to obtain the final processing results corresponding to the delayed data and the normal data; wherein the normal data includes: data that arrives at the current device within a specified time period after the data is generated.

[0039] The above-mentioned normal data usually refers to data of the same data type as the delayed data and is sent to the current device in a timely manner, that is, data that arrives at the current device within a specified time period. The same data type can refer to data for the same business needs. For example, when the delayed data is the traffic data of the edge node device, the normal data is also the traffic data of the edge node device, and the generation time of the delayed data and the normal data are both within the time range corresponding to the traffic usage that the data center wants to know (that is, the same business need).

[0040] In order to simplify the fusion processing process, the processing result of the normal data is usually obtained by processing the normal data in a preset data processing method. The fusion processing method corresponds to the preset data processing method. For example, when the preset data processing method is addition, the processing result corresponding to the normal data is the sum of the normal data, and the intermediate result corresponding to the delayed data is the sum of the delayed data. Then, the fusion process is the sum of the processing result corresponding to the normal data and the intermediate result of the delayed data, and the final processing result corresponding to the delayed data and the normal data is obtained.

[0041] The above-mentioned data processing method first determines the delayed data from the acquired data based on the time information of the acquired data; then processes the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data; further fuses the intermediate result corresponding to the delayed data and the processing result of the normal data corresponding to the delayed data, thereby obtaining the final processing result corresponding to the delayed data and the normal data. This method obtains the intermediate result by processing the delayed data, and then fuses the intermediate result and the processing result obtained based on the normal data to obtain the final processing result, thereby improving the data processing efficiency and saving computing resources.

[0042] The embodiment of the present invention also provides another data processing method, which is implemented on the basis of the method described in the above embodiment; the method focuses on describing the specific implementation process of determining the time information of the acquired data (see steps S300-S306); when the time information includes the time when the data is generated and the time when the data arrives at the current device, based on the time information of the acquired data, determining the normal data and the delayed data from the acquired data and the specific implementation process of processing the two (see steps S308-S316), and determining the specific implementation process of the normal data corresponding to the delayed data (see step S318); Figure 3 As shown, the method comprises the following steps:

[0043] Step S300, receiving a data log sent by a preset edge node device.

[0044] The above data log may be a log file. A log file usually includes multiple logs. Each piece of data is usually saved as a log in the form of a string. When the edge node device sends data to the data center, it usually sends the data in the form of a log file.

[0045] Step S302, record the arrival time of the data log to the current device. The arrival time can be recorded in the form of a timestamp; a timestamp is usually a sequence of characters that uniquely identifies a certain moment in time to unify the time of the edge node device and the data center.

[0046] Step S304, parse the data log to obtain the generation time of the data in the data log. When a piece of data is generated, its generation time is also a part of the data. Other information and generation time of the data will generate a log based on a preset algorithm. The log is usually represented by a string of characters. The log parsing process is equivalent to the reverse process of the log generation process. By parsing the log, the generation time and other information of the data saved in the log can be obtained. For example, when the data is flow data, the usage time and usage amount of the flow can be obtained, and information such as flow direction can also be included. Among them, the generation information of the data can also be saved in the log in the form of a timestamp.

[0047] Step S306: determine the arrival time of the data log as the time when the data in the data log arrives at the current device. Since the data has been stored in the data log in the form of a log, the arrival time of the data log at the current device is the same as the time when the data arrives at the current device.

[0048] Step S308, determine whether the time difference between the time when the acquired data arrives at the current device and the time when it is generated is greater than the specified time period; if less than or equal to the specified time period, execute step S310; if greater than the specified time period, execute step S314.

[0049] In specific implementation, when the specified time period includes a time node, the time length of the specified time period can be obtained based on the time node. The calculated time difference between the time when the data arrives at the current device and the time when it is generated is compared with the time length of the specified time period. When the specified time period does not include a time node but only includes a time length, the calculated time difference between the time when the data arrives at the current device and the time when it is generated can be directly compared with the specified time period to determine whether the data arrives at the current device in a timely manner.

[0050] Step S310, determining whether the acquired data is normal data. Specifically, when the acquired data arrives at the current device within a specified time period, the data is normal data.

[0051] Step S312, based on the preset data processing method, the normal data is processed to obtain the processing result corresponding to the normal data, and step S300 is executed. Although the normal data may not contain all the data that needs to be processed, most of the data will usually be sent to the current device normally, and in order to process the data in a timely manner, the received normal data can be processed based on the preset data processing method to obtain a processing result as a preliminary result of the business demand corresponding to the data processing method. The processing result is then stored in the preset normal data processing result storage location. If there is a need, the data can also be displayed to relevant personnel as a preliminary result of the corresponding business demand.

[0052] Step S314: determine whether the acquired data is delayed data. Specifically, when the acquired data arrives at the current device later than the specified time period, the data is delayed data.

[0053] Step S316: Process the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data.

[0054] In specific implementation, the above data processing methods may include multiple ones; each data processing method corresponds to a business requirement; for example, for mobile phone sales data, the business requirement may be to count the sales of a certain brand of mobile phones, or to count the sales of all brands of mobile phones, or to count the sales of a certain brand of mobile phones in a certain area, etc. In specific implementation, the delayed data may be processed according to different business requirements and based on the data processing methods corresponding to the business requirements to obtain the intermediate results corresponding to the delayed data.

[0055] The above acquired data usually includes multiple information, such as mobile phone sales data may include brand, sales volume, sales region, sales time, etc.; when the business requirement is to count the sales volume of a certain brand of mobile phones, the corresponding data processing method may be to sum the specified information; in this case, the above step S316 may be implemented in the following manner:

[0056] (1) extracting designated information from the multiple information in the delayed data, wherein, corresponding to the business requirement, the designated information is the sales volume of the mobile phone of the brand in each data.

[0057] (2) The extracted designated information is summed to obtain a sum result.

[0058] (3) The summation result is determined as the intermediate result corresponding to the delayed data.

[0059] After receiving the delayed data and processing the delayed data in accordance with the preset data processing method using the above method, the delayed data is usually saved to a preset delayed data storage location, and the intermediate result corresponding to the delayed data is saved to a preset intermediate result storage location. If new delayed data is received, the intermediate result corresponding to the delayed data can be updated based on the new delayed data. At this time, the new delayed data can be updated to the delayed data to obtain the updated delayed data.

[0060] Specifically, the delay data of the same data type as the new delay data and generated in the same time period can be read from the preset delay data storage location; then the read delay data and the new delay data are determined as updated delay data; and then based on the preset data processing method, the updated delay data is processed to obtain the intermediate result corresponding to the updated delay data. The intermediate result can be obtained by processing the delay data of the same data type and generated in the same time period received multiple times.

[0061] Step S318: determine the normal data that has the same data type as the delayed data and is in the same time period as the delayed data as the normal data corresponding to the delayed data.

[0062] When the current device receives normal data, it usually processes the normal data according to a preset data processing method, and saves the normal data and the processing results in a preset storage location. When delayed data is received, it can search for normal data with the same data type and the same generation time period as the delayed data in the preset storage location according to the data type of the delayed data and the generation time of the delayed data, as the normal data corresponding to the delayed data.

[0063] Step S320 , fusing the intermediate results corresponding to the delayed data and the processing results of the normal data corresponding to the delayed data to obtain the final processing results corresponding to the delayed data and the normal data.

[0064] The above-mentioned final processing result is the final processing result corresponding to the current delayed data and normal data. If new delayed data is received and the intermediate result is updated based on the new delayed data, the above-mentioned step S320 is continued to be adopted to perform fusion processing based on the updated intermediate result and the processing result corresponding to the normal data to obtain the updated final data result.

[0065] In the above method, the received data log is parsed to obtain the generation time of the data, and when the time difference between the arrival time and the generation time of the data is greater than the specified time period, the data is determined to be delayed data; then the delayed data is processed based on a preset data processing method to obtain an intermediate result, and then the processing result of the normal data corresponding to the delayed data is merged with the intermediate result to obtain the final result, thereby avoiding repeated calculation of normal data and improving calculation efficiency.

[0066] The embodiment of the present invention also provides another data processing method. Figure 2 The method shown is implemented on the basis of

[0067] Data delay is usually caused by abnormal data collection of the data acquisition device (equivalent to the "above-mentioned edge node device") and abnormal jitter of the transmission network. The main manifestation of data delay is the large deviation between the data generation time and the data processing time, which leads to the omission of delayed data when processing the computing task, resulting in low accuracy of the final result.

[0068] In related technologies, in order to solve the problem of low accuracy of the final result, the analysis and processing of delayed data can be completed by repeatedly calculating the full amount of data; wherein, the full amount of data includes the data sent normally and the data sent with delay, so the data sent normally usually needs to be repeatedly calculated. The method of repeatedly calculating the full amount of data wastes computing resources, is costly, and can cause system instability. When delayed data is received multiple times, the disadvantages of this method are more obvious.

[0069] The method provided in the embodiment of the present invention identifies and classifies delayed data, stores the delayed data separately from normal data, and processes them through a computing task. The processing results obtained from the normal data and the intermediate results obtained from the delayed data are also stored separately. When entering the final storage system, the intermediate results of the delayed data are merged with the processing results of the normal data to obtain the final processing results. This method ultimately reduces the computing cost and improves the reliability of the system.

[0070] This method can also be called a data fusion method for delayed data. It mainly processes delayed data separately through multiple heterogeneous executions of computing tasks (heterogeneous execution: means multiple runs to read different original data, mainly delayed data here); this method is applied to a central server connected to an edge node device. The specific steps of this method are as follows:

[0071] (1) By parsing the log (equivalent to the above-mentioned data log), normal data and delayed data are written separately into different data storages; specifically, normal data and delayed data are distinguished by log time (i.e., the time when the data is generated) and the benchmark time of the task operation (the time when the data processing starts. Since the data is processed in real time, this time can also be considered as the time when the data arrives at the central server). Delayed data refers to data whose time of arrival at the central server and generation time are greater than a preset time threshold.

[0072] (2) The computing task is divided into two parts. The first part performs coarse-grained aggregation operations to reduce the amount of data (equivalent to processing normal data based on a preset data processing method to obtain the processing results of normal data, and processing delayed data based on a preset data processing method to obtain the intermediate results of delayed data). The second part mainly builds a data mart based on different needs.

[0073] A data mart, also called a data market, can be regarded as a database that stores business indicators corresponding to multiple business needs, where the business indicators here can be the final processing results of the normal data and delayed data mentioned above.

[0074] In the first part, the task reads normal data when it runs for the first time (normal data is obtained first) and writes the result data to the normal result storage. When it is not the first time to run, the delayed data is read and the result data is written to the delayed result storage. Then, in the second part of the calculation task, the data in the normal result storage and the data in the delayed result storage are read out at the same time for processing, and the final result is output to the corresponding storage of the data mart layer. While reducing the computing cost, the fusion of normal data and delayed data is realized to meet various data requirements.

[0075] The above data processing method distinguishes normal data from delayed data by the arrival time of the data, calculates the normal data once, and updates and calculates the delayed data multiple times, thereby reducing the data error caused by data delay, and then merges the delayed data with the normal data and writes it into the storage system to complete the processing of the delayed data. This method improves the efficiency of data processing and saves computing resources.

[0076] Corresponding to the above data processing method embodiment, the embodiment of the present invention provides a data processing device, such as Figure 4 As shown, the device comprises:

[0077] The delayed data determination module 400 is used to determine the delayed data from the acquired data based on the time information of the acquired data; wherein the delayed data includes: data that arrives at the current device after the data is generated and exceeds the specified time period;

[0078] The delayed data processing module 402 is used to process the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data;

[0079] The data fusion module 404 is used to fuse the intermediate results corresponding to the delayed data and the processing results of the normal data corresponding to the delayed data to obtain the final processing results corresponding to the delayed data and the normal data; wherein the normal data includes: data that arrives at the current device within a specified time period after the data is generated.

[0080] The above-mentioned data processing device first determines the delayed data from the acquired data based on the time information of the acquired data; then processes the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data; further fuses the intermediate result corresponding to the delayed data and the processing result of the normal data corresponding to the delayed data, thereby obtaining the final processing result corresponding to the delayed data and the normal data. This method obtains the intermediate result by processing the delayed data, and then fuses the intermediate result and the processing result obtained based on the normal data to obtain the final processing result, thereby improving the data processing efficiency and saving computing resources.

[0081] Furthermore, the above-mentioned time information includes the time when the data is generated and the time when the data arrives at the current device; the above-mentioned delayed data determination module is also used to: determine whether the time difference between the time when the acquired data arrives at the current device and the time when it is generated is greater than the specified time period; if it is greater than the specified time period, determine that the acquired data is delayed data.

[0082] Furthermore, the above-mentioned device also includes a time acquisition module, which is used to: receive the data log sent by the preset edge node device; record the arrival time of the data log to the current device; parse the data log to obtain the generation time of the data in the data log; and determine the arrival time of the data log as the time when the data in the data log arrives at the current device.

[0083] Furthermore, the above-mentioned time information includes the time when the data is generated and the time when the data arrives at the current device; the above-mentioned device also includes a processing result acquisition module, which is used to: determine whether the time difference between the time when the acquired data arrives at the current device and the time when the data is generated is less than or equal to the specified time period; if it is less than or equal to the specified time period, determine that the acquired data is normal data; based on a preset data processing method, process the normal data to obtain the processing result corresponding to the normal data.

[0084] Furthermore, the above-mentioned time information includes the generation time of the data; the above-mentioned device also includes a normal data determination module, which is used to: determine the normal data with the same data type as the delayed data and in the same time period as the generation time of the delayed data as the normal data corresponding to the delayed data.

[0085] Furthermore, the above-mentioned device also includes an intermediate result updating module, which is used to: if new delayed data is received, update the intermediate result corresponding to the delayed data based on the new delayed data.

[0086] Furthermore, the intermediate result updating module is also used to: update the new delay data into the delay data to obtain updated delay data; based on a preset data processing method, process the updated delay data to obtain the intermediate result corresponding to the updated delay data.

[0087] Furthermore, the above-mentioned data processing methods include multiple methods; each data processing method corresponds to a business requirement; the above-mentioned delayed data processing module is also used to: according to different business requirements, process the delayed data based on the data processing method corresponding to the business requirement to obtain the intermediate result corresponding to the delayed data.

[0088] Furthermore, the above-mentioned acquired data includes multiple information; the data processing method includes summing up the specified information; the above-mentioned delayed data processing module is also used to: extract the specified information from the multiple information in the delayed data; sum the extracted specified information to obtain the summation result; and determine the summation result as the intermediate result corresponding to the delayed data.

[0089] The data processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned data processing method embodiment. For the sake of brief description, for matters not mentioned in the data processing device embodiment, reference may be made to the corresponding contents in the aforementioned data processing method embodiment.

[0090] The embodiment of the present invention also provides a server, see Figure 5 As shown, the server includes a processor 130 and a memory 131 . The memory 131 stores machine executable instructions that can be executed by the processor 130 . The processor 130 executes the machine executable instructions to implement the above data processing method.

[0091] Further, Figure 5 The server shown also includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0092] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0093] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0094] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned data processing method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0095] The data processing method, device and server computer program product provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0097] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that: include: Based on the time information of the acquired data, the delayed data and the normal data are determined from the acquired data; wherein the time information includes the generation time of the data and the time when the data arrives at the current device; the delayed data includes: data when the time difference between the time when the data arrives at the current device and the generation time is greater than a specified time period; the normal data includes: data when the time difference between the time when the data arrives at the current device and the generation time is less than or equal to the specified time period; Based on a preset data processing method, the delayed data and the normal data are processed to obtain an intermediate result corresponding to the delayed data and a processing result corresponding to the normal data; Determine normal data that has the same data type as the delayed data and is in the same time period as the generation time of the delayed data as normal data corresponding to the delayed data; If new delay data is received, the new delay data is updated into the delay data to obtain updated delay data; based on a preset data processing method, the updated delay data is processed to obtain an intermediate result corresponding to the updated delay data; The intermediate result corresponding to the delayed data and the processing result of the normal data corresponding to the delayed data are fused to obtain the final processing results corresponding to the delayed data and the normal data.

2. The method according to claim 1, characterized in that The step of determining delayed data from the acquired data based on time information of the acquired data comprises: Determine whether the time difference between the time when the acquired data arrives at the current device and the time when the data is generated is greater than the specified time period; If it is greater than the specified time period, it is determined that the acquired data is delayed data.

3. The method according to claim 2, characterized in that The time information of the acquired data is determined in the following manner: Receive data logs sent by preset edge node devices; Record the arrival time of the data log at the current device; Parsing the data log to obtain the generation time of the data in the data log; The arrival time of the data log is determined as the time when the data in the data log arrives at the current device.

4. The method according to claim 1, characterized in that: The processing result of the normal data is obtained in the following way: Determine whether the time difference between the time when the acquired data arrives at the current device and the time when the data is generated is less than or equal to the specified time period; If it is less than or equal to the specified time period, determining that the acquired data is the normal data; Based on a preset data processing method, the normal data is processed to obtain a processing result corresponding to the normal data.

5. The method according to claim 1, characterized in that The data processing methods include multiple methods; each data processing method corresponds to a business requirement; The step of processing the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data includes: According to different business requirements, the delayed data is processed based on the data processing method corresponding to the business requirement to obtain an intermediate result corresponding to the delayed data.

6. The method according to claim 1, characterized in that The acquired data includes a plurality of information; the data processing method includes summing the specified information; The step of processing the delayed data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data includes: extracting specified information from a plurality of information in the delay data; Performing sum processing on the extracted designated information to obtain a sum result; The summation result is determined as an intermediate result corresponding to the delayed data.

7. A data processing device, characterized in that: include: A data determination module, configured to determine delayed data and normal data from the acquired data based on time information of the acquired data; wherein the time information includes the generation time of the data and the time when the data arrives at the current device; the delayed data includes: data whose time difference between the time when the data arrives at the current device and the generation time is greater than a specified time period; the normal data includes: data whose time difference between the time when the data arrives at the current device and the generation time is less than or equal to the specified time period; A data processing module, used to process the delayed data and the normal data based on a preset data processing method to obtain an intermediate result corresponding to the delayed data and a processing result corresponding to the normal data; A data fusion module is used to determine normal data that has the same data type as the delayed data and is in the same time period as the generation time of the delayed data as normal data corresponding to the delayed data; if new delayed data is received, the new delayed data is updated to the delayed data to obtain updated delayed data; based on a preset data processing method, the updated delayed data is processed to obtain an intermediate result corresponding to the updated delayed data; the intermediate result corresponding to the delayed data and the processing result of the normal data corresponding to the delayed data are fused to obtain the final processing result corresponding to the delayed data and the normal data; wherein the normal data includes: data that arrives at the current device within the specified time period after the data is generated.

8. A server, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the data processing method according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the data processing method according to any one of claims 1 to 6.

Images