Data processing method and device, computer equipment, storage medium and computer program product
By introducing a streaming computing framework into the power business system and utilizing Apache Kafka and Apache Flink for real-time data processing and abnormal data correction, the problem of insufficient real-time data analysis in the power business system has been solved, and efficient data analysis and monitoring have been achieved.
Patent Information
- Application Number
- CN202411107137.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-19
AI Technical Summary
The real-time performance of data analysis in existing power business systems is insufficient, resulting in low real-time performance of data processing platforms.
A streaming computing framework is used to analyze data from multiple power data sources in the power business system. Real-time data processing is performed using tools such as Apache Kafka and Apache Flink. Combined with data preprocessing and anomaly correction, efficient data analysis results are obtained.
It improves the real-time performance and accuracy of data analysis in the power business system, ensures data quality, and generates detailed data monitoring reports.
Smart Images

Figure CN118982290A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] In multiple business domain projects of the power business system, it is usually necessary to perform data analysis on the business domain data to ensure the data quality of the business domain projects.
[0003] In current technology, big data processing platforms are usually used to integrate computing resources and storage resources to conduct centralized data analysis on massive business domain data of power business systems. The real-time performance needs to be improved and there is a problem of low real-time performance. Summary of the invention
[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, storage medium and computer program product that can improve the real-time performance of data analysis in response to the above technical problems.
[0005] In a first aspect, the present application provides a data processing method. The method comprises:
[0006] Acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time;
[0007] Using a streaming computing framework, according to the current business domain data and the analysis indicators of the power object to be analyzed, a first data analysis result corresponding to each of the analysis indicators is obtained;
[0008] Based on the first data analysis result, a second data analysis result of the current business domain data of the power object to be analyzed is obtained.
[0009] In one of the embodiments, the obtaining of current business domain data of the power object to be analyzed from multiple power data sources in the power business system includes: determining the data source type of each of the power data sources; determining a data acquisition strategy matching each of the power data sources according to the data source type; and obtaining the current business domain data of the power object to be analyzed according to the data acquisition strategy.
[0010] In one of the embodiments, acquiring the current business domain data of the power object to be analyzed according to the data acquisition strategy includes: acquiring the original business domain data of the power object to be analyzed from multiple power data sources according to the data acquisition strategy; and performing data preprocessing on the original business domain data to obtain the current business domain data.
[0011] In one of the embodiments, the streaming computing framework is used to obtain first data analysis results corresponding to each analysis indicator based on the current business domain data and the analysis indicators of the power object to be analyzed, including: determining a data analysis strategy corresponding to each analysis indicator; and performing data analysis on the current business domain data according to the data analysis strategy to obtain the first data analysis results corresponding to each analysis indicator.
[0012] In one of the embodiments, the current business domain data is analyzed according to the data analysis strategy to obtain the first data analysis result corresponding to each of the analysis indicators, including: using the streaming computing framework to establish a task process for each of the analysis indicators; inputting the current business domain data into the task process, and according to the corresponding data analysis strategy, performing data analysis on the current business domain data to obtain the first data analysis result corresponding to each of the analysis indicators.
[0013] In one of the embodiments, after obtaining the second data analysis result of the current business domain data of the power object to be analyzed based on the first data analysis result, it also includes: determining abnormal data in the current business domain data according to the second data analysis result; correcting the abnormal data to obtain optimized business domain data corresponding to the current business domain data; and using the data monitoring model corresponding to the power object to be analyzed to obtain a data monitoring report of the power object to be analyzed according to the optimized business domain data.
[0014] In a second aspect, the present application provides a data processing device. The device comprises:
[0015] A data acquisition module, used to acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time;
[0016] A data analysis module, used to obtain a first data analysis result corresponding to each analysis indicator according to the current business domain data and each analysis indicator of the power object to be analyzed by using a streaming computing framework;
[0017] The summary module is used to obtain a second data analysis result of the current business domain data of the power object to be analyzed based on the first data analysis result.
[0018] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0019] Acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time;
[0020] Using a streaming computing framework, according to the current business domain data and the analysis indicators of the power object to be analyzed, a first data analysis result corresponding to each of the analysis indicators is obtained;
[0021] Based on the first data analysis result, a second data analysis result of the current business domain data of the power object to be analyzed is obtained.
[0022] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0023] Acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time;
[0024] Using a streaming computing framework, according to the current business domain data and the analysis indicators of the power object to be analyzed, a first data analysis result corresponding to each of the analysis indicators is obtained;
[0025] Based on the first data analysis result, a second data analysis result of the current business domain data of the power object to be analyzed is obtained.
[0026] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0027] Acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time;
[0028] Using a streaming computing framework, according to the current business domain data and the analysis indicators of the power object to be analyzed, a first data analysis result corresponding to each of the analysis indicators is obtained;
[0029] Based on the first data analysis result, a second data analysis result of the current business domain data of the power object to be analyzed is obtained.
[0030] In the above-mentioned data processing method, device, computer equipment, storage medium and computer program product, the current business domain data of the power object to be analyzed is obtained from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time; using the streaming computing framework, according to the current business domain data and each analysis indicator of the power object to be analyzed, the first data analysis result corresponding to each analysis indicator is obtained; based on the first data analysis result, the second data analysis result of the current business domain data of the power object to be analyzed is obtained. In the method provided in the embodiment of the present application, a streaming computing framework is introduced to perform real-time calculations on the current business domain data, thereby improving the real-time performance of the data analysis of the current business domain data of the power object to be analyzed. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0032] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0033] Figure 2 A schematic diagram of a process for obtaining current business domain data provided in an embodiment of the present application;
[0034] Figure 3 A schematic diagram of a data preprocessing process provided in an embodiment of the present application;
[0035] Figure 4 A schematic diagram of a process for obtaining a first data analysis result provided in an embodiment of the present application;
[0036] Figure 5 A structural block diagram of a data processing device provided in an embodiment of the present application;
[0037] Figure 6 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0039] In an exemplary embodiment, Figure 1As shown, a data processing method is provided. This embodiment takes the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0040] Step 102: Acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system.
[0041] Among them, the multiple power data sources can be a database system associated with the power business system, for example, a power knowledge graph, and the power business system can have at least one business domain application or business domain system, for example, one-stop energy consumption theme analysis, one-stop energy industrial product service capability analysis, and energy industry value chain material transportation capacity analysis and other Internet business domain related applications or systems. Among them, the one-stop energy consumption theme analysis can be used to analyze the energy consumption of the manufacturing industry in the province covered by the power grid in new energy services, so as to analyze the new energy service capabilities and aggregation. The one-stop energy industrial product service capability analysis can be used to analyze the demand satisfaction and customer characteristics of the procurement of electrical industrial products through empowerment, and provide auxiliary decision-making for the potential increase in social electricity demand. The energy industry value chain material transportation capacity analysis can be used to analyze the integration of internal and external transportation resources through the analysis of the contribution of logistics transportation capacity, enable the allocation and supply of power grid materials, and provide auxiliary decision-making for the capacity and focus of further power grid infrastructure material transportation capacity. At least one business domain application or business domain system of the power business system can be used as the power object to be analyzed, and the power object to be analyzed can be any one of the at least one business domain application or business domain system. The current business domain data is the business domain data of the power object to be analyzed at the current time. For example, the current business domain data can be the current business domain data of the business domain system used to analyze the energy consumption of the manufacturing industry in the province covered by the power grid in terms of new energy services at the current time, that is, a one-stop energy consumption theme analysis. Obtaining the current business domain data of the power object to be analyzed can be considered as obtaining the associated business domain data of the power object to be analyzed in real time.
[0042] Step 104 , using a streaming computing framework, according to the current business domain data and the analysis indicators of the power object to be analyzed, obtain a first data analysis result corresponding to each analysis indicator.
[0043] Among them, in this step, the stream computing framework can be used to perform real-time data analysis on the current business domain data of the power object to be analyzed, and obtain the data quality analysis results of the business domain data of the power object to be analyzed at the current time. The stream computing framework can be Apache Kafka and Apache Flink, etc. Apache Flink is a stream data processing engine that supports complex real-time data processing and analysis tasks, has the characteristics of low latency and high throughput, can seamlessly combine stream processing and batch processing, and supports real-time stream computing and offline batch processing at the same time, providing a unified processing engine. Apache Kafka is a high-throughput, low-latency distributed message queue system for publishing and subscribing to real-time data; it can also persistently store large amounts of data, and support fast reading and writing of data to ensure data reliability. Kafka can also be used as a buffer area for data processing processes, allowing decoupling between producers and consumers to achieve asynchronous data stream processing. In one possible implementation, at least one analysis indicator may be pre-set for the power object to be analyzed, and the at least one analysis indicator may include but is not limited to completeness, accuracy, consistency, repeatability, data type and format, data source verification, data history and change tracking, data security and compliance, etc., wherein the integrity refers to the integrity of data rows and columns, and the integrity of data rows and columns may be checked to ensure that no data is lost; the accuracy refers to the precision and correctness of the data, and the precision and correctness of the data may be verified, and the data may be compared with the source system data to ensure that they match, or rules and verification may be used to detect data errors; the consistency refers to the consistency of data between different data sources and data tables, and the consistency of data between different data sources and data tables may be checked. It can include consistency of data values, units and naming, verifying the consistency of the same data elements in different data tables, and ensuring that they have the same value and unit; repeatability refers to the repeatability of the data processing process, which can ensure that the data processing process is repeatable without interference from external factors; data type and format refers to the data type of the data matches the expected data type and the format of the data matches the expected format, and the type and format checking standards can be used to ensure that the data matches the expected data type and format; data source verification refers to verifying the availability and validity of the data source, checking whether the data source connection is normal, and verifying the metadata and data dictionary of the data source; data history and change tracking refers to tracking historical changes of data and auditing of data changes, recording each change to the data, including when, how and why the data was changed; data security and compliance refers to the implementation of data encryption, access control and data masking to protect sensitive data.In the process of obtaining the first data analysis result corresponding to each analysis indicator, a task process can be established for each analysis indicator of the at least one analysis indicator based on the streaming computing framework. For the task process of each analysis indicator, the current business domain data can be input into the task process of the analysis indicator, and the streaming computing framework is used to perform data analysis on the current business domain data to obtain, for each analysis indicator, the first data analysis result of the current business domain data. The first data analysis result is used to indicate whether the current business domain data meets the standard of the corresponding analysis indicator. In a possible implementation method, an indicator threshold value can be pre-set for each analysis indicator, such as an integrity threshold value and an accuracy threshold value. The analysis indicator integrity is used as an example for explanation. Take the integrity analysis result of the current business domain data in terms of integrity, which can also be called data integrity. If the data integrity is greater than the integrity threshold, it indicates that the data integrity of the current business domain data meets the standard of the integrity index, and then it can be concluded that the first data analysis result of the analysis index for the integrity of the current business domain data meets the standard of the integrity index; if the data integrity is less than or equal to the integrity threshold, it indicates that the data integrity of the current business domain data does not meet the standard of the integrity index, and then it can be concluded that the first data analysis result of the analysis index for the integrity of the current business domain data does not meet the standard of the integrity index, and the first data analysis result can also include an abnormal data identifier of abnormal data in the current business domain data.
[0044] Step 106: Based on the first data analysis result, a second data analysis result of the current business domain data of the power object to be analyzed is obtained.
[0045] The first data analysis result of the current business domain data for the at least one analysis indicator is summarized to obtain the second data analysis result corresponding to the current business domain data.
[0046] In the method of this embodiment, the current business domain data of the power object to be analyzed is obtained from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time; using the streaming computing framework, according to the current business domain data and each analysis indicator of the power object to be analyzed, the first data analysis result corresponding to each analysis indicator is obtained; based on the first data analysis result, the second data analysis result of the current business domain data of the power object to be analyzed is obtained. In the method provided by the embodiment of the present application, a streaming computing framework is introduced to perform real-time calculations on the current business domain data, thereby improving the real-time performance of the data analysis of the current business domain data of the power object to be analyzed.
[0047] In an exemplary embodiment, Figure 2As shown, step 102 may include steps 202 to 206. Among them:
[0048] Step 202: determine the data source type of each power data source.
[0049] Step 204: Determine a data acquisition strategy that matches each power data source according to the data source type.
[0050] Step 206: Acquire the current business domain data of the power object to be analyzed according to the data acquisition strategy.
[0051] Among them, there can be multiple data acquisition methods for different data source types, such as API calls, data crawling, and log files, and the data formats and data structures of different data source types may also be different. Therefore, different data acquisition strategies can be set for different data source types.
[0052] In the method of this embodiment, the difference in data source types of different power data sources can be taken into consideration, and different data acquisition strategies can be set for different power data sources, which can improve the integrity and accuracy of data acquisition.
[0053] In an exemplary embodiment, Figure 3 As shown, step 206 may include steps 302 to 304. Among them:
[0054] Step 302: According to the data acquisition strategy, original business domain data of the power object to be analyzed is acquired from multiple power data sources.
[0055] Step 304: pre-process the original business domain data to obtain current business domain data.
[0056] Among them, there may be missing values, duplicate data and outliers in the business domain data. In order to ensure the quality and consistency of the business domain data, the collected original business domain data can be cleaned and preprocessed. In addition, considering that the data format and data structure of different data source types may also be different, the collected original business domain data can also be converted and mapped.
[0057] In the method of this embodiment, the collected original business domain data can be processed to unify the format and structure, thereby improving the accuracy and efficiency of data analysis.
[0058] In an exemplary embodiment, Figure 4 As shown, step 104 may include steps 402 to 404. Among them:
[0059] Step 402: determine the data analysis strategy corresponding to each analysis indicator.
[0060] Step 404: perform data analysis on the current business domain data according to the data analysis strategy to obtain first data analysis results corresponding to each analysis indicator.
[0061] In an exemplary embodiment, step 404 may include:
[0062] Using the streaming computing framework, a task process for each analysis indicator is established; the current business domain data is input into the task process, and data analysis is performed on the current business domain data according to the corresponding data analysis strategy to obtain the first data analysis result corresponding to each analysis indicator.
[0063] Among them, for different analysis indicators, the corresponding data analysis methods will be different. Therefore, different data analysis strategies can be set for different analysis indicators. Specifically, a task process can be established for each of the at least one analysis indicator based on the streaming computing framework. For each task process of the analysis indicator, the current business domain data can be input into the task process of the analysis indicator, and the streaming computing framework is used to perform data analysis on the current business domain data to obtain a first data analysis result of the current business domain data for each analysis indicator. The first data analysis result is used to indicate whether the current business domain data meets the standard of the corresponding analysis indicator. In a possible implementation method, an indicator threshold can be pre-set for each analysis indicator, such as an integrity threshold and an accuracy threshold. Taking integrity as an example for explanation, when obtaining the current business domain data in The integrity analysis result in terms of integrity, which can also be called data integrity, if the data integrity is greater than the integrity threshold, it indicates that the data integrity of the current business domain data meets the standard of the integrity index, and it can be concluded that the first data analysis result of the analysis index for the integrity of the current business domain data meets the standard of the integrity index; if the data integrity is less than or equal to the integrity threshold, it indicates that the data integrity of the current business domain data does not meet the standard of the integrity index, and it can be concluded that the first data analysis result of the analysis index for the integrity of the current business domain data does not meet the standard of the integrity index, and the first data analysis result can also include an abnormal data identifier of abnormal data in the current business domain data. In a possible implementation, in the process of using the streaming computing framework to perform data analysis on the current business domain data, a preset sliding window can be used to perform real-time data analysis on the current business domain data. The sliding distance of the sliding window can be set to ds, the window size of the sliding window can be set to ms, and the sliding window can be divided into m / d×α window units, where α is an adjustment factor; the number of occurrences of abnormal data of the current business domain data is counted according to multiple window units of the sliding window, and the number of occurrences of abnormal data counted by all window units is merged, so that the number of occurrences of business domain data anomalies in the most recent ms can be obtained by outputting every ds.
[0064] In the method of this embodiment, a streaming computing framework is introduced to perform real-time computing on the current business domain data, thereby improving the real-time performance of data analysis on the current business domain data of the power object to be analyzed.
[0065] In an exemplary embodiment, the embodiment of the present application further includes the following steps, which are used to describe the data processing of the current business domain data after obtaining the second data analysis result, and the steps specifically include:
[0066] According to the second data analysis result, abnormal data in the current business domain data is determined; the abnormal data is corrected to obtain optimized business domain data corresponding to the current business domain data; using the data monitoring model corresponding to the power object to be analyzed, according to the optimized business domain data, a data monitoring report of the power object to be analyzed is obtained.
[0067] After obtaining the second data analysis result of the current business domain data, the abnormal data in the current business domain data can be eliminated and corrected according to the second data analysis result to obtain the optimized business domain data corresponding to the current business domain data. Then, the optimized business domain data of the power object to be analyzed can be stored, and further, the data monitoring model corresponding to the power object to be analyzed can be used to obtain the data monitoring report of the power object to be analyzed according to the optimized business domain data. Specifically, the data monitoring model can include an input layer, a multi-scale convolutional network layer, a recurrent neural network layer, an attention layer and an output layer, wherein the input layer is used to take the optimized business domain data as input, pre-process the optimized business domain data, and obtain the value of the optimized business domain data. The multi-scale convolutional network is a variant of the convolutional neural network, which is designed to process feature information of different scales to improve the performance and generalization ability of the model. The scale of the multi-scale convolutional network is achieved by adjusting the size of the convolution kernel in the convolution layer, and the size of the convolution kernel is related to the data analysis result. The recurrent neural network layer uses GRU. GRU (Gated Recurrent Unit) is a common variant of recurrent neural network (RNN) used to process sequence data, such as time series prediction tasks. GRU introduces a gating mechanism to solve the problems of gradient disappearance and gradient explosion in traditional RNN, which helps to capture long-term dependencies. The attention layer is used to assign weights to the feature vectors output by the recurrent neural network layer and adjust the weights of different feature vectors. The output layer is used to output monitoring and early warning reports.
[0068] In the method of this embodiment, a data monitoring model is introduced to perform further data analysis and data monitoring on the optimized business domain data, thereby improving the accuracy of data monitoring and further improving data quality.
[0069] In an exemplary embodiment, after obtaining the first data analysis result, the second data analysis result and the data monitoring report of the current business domain data of the power object to be analyzed, the first data analysis result, the second data analysis result and the data monitoring report of the current business domain data of the power object to be analyzed can be displayed based on a visualization interface, and the visualization interface can be run on different devices and browsers, and has cross-platform compatibility and responsive design. The functions of the visualization interface are as follows: 1. User-friendliness: Whether the interface is easy to use and the user can easily find the required information and functions. Whether the navigation is intuitive and the user can quickly browse different parts and functions. 2. Responsiveness: Whether the interface is consistent on different devices and screen sizes and can adapt to different screens. Whether the interface responds quickly to user interactions to avoid long loading or freezing. 3. Consistency: Whether the consistency of interface elements, such as fonts, colors, button styles and icons, conforms to brand identity and design guidelines. Consistent navigation and layout to ensure that users have similar experiences on different pages. 4. Accessibility: Whether the interface meets accessibility standards to ensure that users can use it regardless of whether they have disabilities. Provide screen reader support, keyboard navigation and other auxiliary functions. 5. Performance: Whether the loading time and performance of the interface meet the user's expectations, avoiding long waits or page crashes. Whether the response time is within a reasonable range to ensure a smooth user experience. 6. Security: The security measures of the interface, such as data encryption, authentication, and authorization, to protect the security of users and data. Prevent common network attacks, such as cross-site scripting (XSS) and SQL injection. 7. Data accuracy: Whether the data displayed on the interface is accurate and consistent with the data in the back-end system. Implementation of data validation and verification rules to prevent invalid or inaccurate data entry. 8. Functional completeness: Whether the functions of the interface meet the needs and expectations of users, including various interactions and operations. Whether the interface supports different browsers and operating systems. 9. Error handling: Whether the interface provides clear error messages and help documents to help users solve problems. The fault tolerance of the interface, whether it can handle different types of errors and abnormal situations. 10. User feedback: Whether the interface collects user feedback and error reports for improvement and repair. Whether user support and contact information are accessible and effective.
[0070] In the method of this embodiment, a visual interface is introduced to improve the convenience of obtaining data reports.
[0071] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0072] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.
[0073] In one embodiment, Figure 5 As shown, a data processing device is provided, including: a data acquisition module 502, a data analysis module 504 and a summary module 506, wherein:
[0074] The data acquisition module 502 is used to acquire the current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time;
[0075] The data analysis module 504 is used to obtain a first data analysis result corresponding to each analysis indicator according to the current business domain data and each analysis indicator of the power object to be analyzed by using a streaming computing framework;
[0076] The summarizing module 506 is configured to obtain a second data analysis result of the current business domain data of the power object to be analyzed based on the first data analysis result.
[0077] In one embodiment, the data acquisition module 502 is further used to: determine the data source type of each of the power data sources; determine a data acquisition strategy that matches each of the power data sources based on the data source type; and acquire the current business domain data of the power object to be analyzed based on the data acquisition strategy.
[0078] In one embodiment, the data acquisition module 502 is further used to: acquire original business domain data of the power object to be analyzed from multiple power data sources according to the data acquisition strategy; and perform data preprocessing on the original business domain data to obtain the current business domain data.
[0079] In one embodiment, the data analysis module 504 is further used to: determine the data analysis strategy corresponding to each of the analysis indicators; perform data analysis on the current business domain data according to the data analysis strategy to obtain the first data analysis result corresponding to each of the analysis indicators.
[0080] In one embodiment, the data analysis module 504 is also used to: use the streaming computing framework to establish a task process for each of the analysis indicators; input the current business domain data into the task process, and perform data analysis on the current business domain data according to the corresponding data analysis strategy to obtain the first data analysis results corresponding to each of the analysis indicators.
[0081] In one embodiment, the summary module 506 is also used to: determine the abnormal data in the current business domain data based on the second data analysis result; correct the abnormal data to obtain the optimized business domain data corresponding to the current business domain data; use the data monitoring model corresponding to the power object to be analyzed, and obtain the data monitoring report of the power object to be analyzed based on the optimized business domain data.
[0082] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0083] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to data processing. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.
[0084] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0085] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0086] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0087] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0088] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0090] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0091] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time; Using a streaming computing framework, according to the current business domain data and the analysis indicators of the power object to be analyzed, a first data analysis result corresponding to each of the analysis indicators is obtained; Based on the first data analysis result, a second data analysis result of the current business domain data of the power object to be analyzed is obtained.
2. The method according to claim 1, characterized in that The obtaining of current business domain data of the power object to be analyzed from multiple power data sources in the power business system includes: Determining a data source type of each of the power data sources; Determining, according to the data source type, a data acquisition strategy that matches each of the power data sources; According to the data acquisition strategy, the current business domain data of the power object to be analyzed is acquired.
3. The method according to claim 2, characterized in that The step of acquiring the current business domain data of the power object to be analyzed according to the data acquisition strategy includes: According to the data acquisition strategy, original business domain data of the power object to be analyzed is acquired from the plurality of power data sources; The original business domain data is preprocessed to obtain the current business domain data.
4. The method according to claim 1, characterized in that: The method of using the streaming computing framework to obtain the first data analysis results corresponding to the analysis indicators according to the current business domain data and the analysis indicators of the power object to be analyzed includes: Determine the data analysis strategy corresponding to each of the analysis indicators; According to the data analysis strategy, data analysis is performed on the current business domain data to obtain the first data analysis results corresponding to each of the analysis indicators.
5. The method according to claim 4, characterized in that The step of performing data analysis on the current business domain data according to the data analysis strategy to obtain the first data analysis result corresponding to each of the analysis indicators includes: Using the streaming computing framework, establishing a task process for each of the analysis indicators; The current business domain data is input into the task process, and data analysis is performed on the current business domain data according to the corresponding data analysis strategy to obtain the first data analysis results corresponding to each of the analysis indicators.
6. The method according to claim 1, characterized in that After obtaining the second data analysis result of the current business domain data of the power object to be analyzed based on the first data analysis result, the method further includes: Determining abnormal data in the current business domain data according to the second data analysis result; Correcting the abnormal data to obtain optimized business domain data corresponding to the current business domain data; A data monitoring report of the power object to be analyzed is obtained by utilizing the data monitoring model corresponding to the power object to be analyzed and according to the optimized business domain data.
7. A data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire current business domain data of the power object to be analyzed from multiple power data sources in the power business system; the current business domain data is the business domain data of the power object to be analyzed at the current time; A data analysis module, used to obtain a first data analysis result corresponding to each analysis indicator according to the current business domain data and each analysis indicator of the power object to be analyzed by using a streaming computing framework; The summary module is used to obtain a second data analysis result of the current business domain data of the power object to be analyzed based on the first data analysis result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
