Multi-service data integration processing method and device based on new energy, and server

Abstract

The invention provides a multi-service data integration processing method and device based on new energy, and a server, which can be used in the technical field of new energy. Based on the method, before specific implementation, a multi-service data integration processing system which is adaptive to a new energy multi-service data processing scene and at least comprises a big data storage and calculation engine module, a data development tool and a data management tool is constructed. In specific implementation, a multi-service data integration processing system is utilized to be in butt joint with a plurality of service data source ends under the new energy background, multiple different types of service data related to new energy are acquired and integrated, and comprehensive target joint service data is obtained; and creating and using a target business processing flow and a target data warehouse by using a multi-business data integration processing system, integrating different types of business data, predicting a business processing strategy in a target time period through joint operation, and contacting the business processing strategy to an associated target business implementation end, so that the business implementation end can be accurately helped to make a decision.

Description

Technical Field

[0001] This specification belongs to the field of new energy technology, and in particular to a multi-business data integration processing method, device and server based on new energy. Background Art

[0002] With the development of the new energy industry, large new energy companies usually need to connect and process a large amount of multi-source heterogeneous business data from different data sources involving new energy.

[0003] However, when processing the above-mentioned massive business data based on existing technologies, there is often chaos in data operation management, and it is impossible to fully and comprehensively integrate and utilize the above-mentioned business data, and it is impossible to effectively and accurately mine the data value implicit in the above-mentioned business data, which in turn affects the subsequent decision-making and execution of new energy companies at multiple business levels such as production, marketing, and operation and maintenance.

[0004] Currently, no effective solution has been proposed to the above problems. Summary of the invention

[0005] This specification provides a multi-business data integration and processing method, device and server based on new energy, which can be well adapted to the new energy multi-business data processing scenario, and can comprehensively and effectively integrate and utilize various types of business data involving new energy provided by multiple business data sources, accurately help the business implementation end to make decisions, so that the business implementation end can more efficiently and accurately adjust and execute related businesses to obtain better business results.

[0006] This specification provides a multi-business data integration processing method based on new energy, which is applied to a node server in a multi-business data integration processing system, wherein the multi-business data integration processing system also includes: a big data storage and computing engine module, a data development tool, and a data management tool. The multi-business data integration processing system is connected to a new energy plant station end through a first connection and is connected to a power grid market end through a second connection. The method includes:

[0007] Receive and respond to target joint business data processing requests, and call data development tools to generate corresponding target business processing flows;

[0008] According to the target business processing flow, the data management tool is called to obtain the power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and the power grid market data related to the power grid market through the second connection; and the power generation capacity data, plant equipment operation data and power grid market data are combined to obtain the target joint business data;

[0009] According to the target business processing flow, the big data storage and computing engine module is called to create a corresponding target data warehouse; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and the target joint business data is stored in the original data layer;

[0010] According to the target business processing flow, call the data management tool to read the target joint business data through the original data layer; pre-process the target joint business data to obtain the pre-processed target joint business data; and store the pre-processed target joint business data in the detailed data layer;

[0011] According to the target business processing flow, the big data storage and computing engine module is called to read the pre-processed target joint business data through the detailed data layer, combined with the public attribute data in the public dimension layer, and through joint calculation, the business processing strategy for the target time period is predicted as the target processing result; and the target processing result is stored in the summary data layer;

[0012] Based on the target processing result, determining the associated target business implementation end; storing the target processing result in the application data layer; and establishing a third connection between the target business implementation end and the application data layer;

[0013] Sending target prompt information about the target processing result to the target business implementation end; wherein the target prompt information carries at least an interface identifier of the data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the relevant business of executing the target time period according to the target processing result.

[0014] In one embodiment, after calling the data management tool to obtain power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and obtaining power grid market data related to the power grid market through the second connection, the method further includes:

[0015] Calling the data management tool to detect whether there is sensitive data in the power generation capacity data, plant equipment operation data, and power grid market data according to the preset sensitive information data table; and if it is determined that sensitive data exists, encrypting the sensitive data according to the preset homomorphic encryption rules to obtain the ciphertext data of the sensitive data;

[0016] Call the data management tool to replace the sensitive data in power generation capacity data, plant equipment operation data, and power grid market data with the ciphertext data of the sensitive data; and destroy the sensitive data.

[0017] In one embodiment, calling a data management tool to obtain power generation capacity data related to a new energy plant through a first connection includes:

[0018] Detect and identify the data type of power generation capacity data;

[0019] For streaming data in power generation capacity data, call the real-time data integration development function component in the data development tool, and build the corresponding first connection in combination with Flink CDC and Kafka; call the data management tool to obtain and cache the streaming data in power generation capacity data through the first connection;

[0020] For batch data in power generation capacity data, call the offline data integration development function component in the data development tool to build a corresponding first connection; call the data management tool to obtain the batch data in the power generation capacity data through the first connection.

[0021] In one embodiment, preprocessing the target joint service data includes:

[0022] Call the data management tool to standardize the field type, field name, field length, data type, and data format in the target joint business data according to the preset standardization rules, and obtain the standardized target joint business data;

[0023] According to the target business processing flow, call the data management tool to build data tables and data models related to the target joint business data processing.

[0024] In one embodiment, the data model includes: a power generation capacity prediction model, a distribution network power prediction model, and a production and marketing decision model.

[0025] In one embodiment, according to the target business processing flow, the big data storage and computing engine module is called to read the target joint business data through the detailed data layer and predict the business processing strategy for the target time period as the target processing result through joint operation based on the pre-processed target joint business data and the public attribute data in the public dimension layer, including:

[0026] According to the target business processing flow, the big data storage and computing engine module is called to read the target joint business data through the detailed data layer and run the power generation capacity prediction model and the distribution network power prediction model respectively based on the pre-processed target joint business data and the public attribute data in the public dimension layer to obtain the power generation capacity prediction results and the distribution network demand power prediction results for the target time period;

[0027] According to the target business processing flow, the big data storage and computing engine module is called to combine the power generation capacity forecast result of the target time period and the distribution network demand power forecast result of the target time period to obtain the target integrated data group for the target time period;

[0028] According to the target business processing flow, the big data storage and computing engine module is called to utilize the target integrated data group, combined with the public attribute data in the public dimension layer, to run the production and marketing decision model, and predict the production and marketing joint strategy for the target time period as the target processing result.

[0029] In one embodiment, the method further comprises:

[0030] Calling the data management tool to determine the preset data quality detection rules corresponding to the relevant data involved in the target business processing flow from the preset data quality detection rule set, and combining them to obtain the target data quality detection rules;

[0031] Call the data management tool to perform quality inspection on the relevant data appearing in the multi-business data integration processing system according to the target data quality inspection rules;

[0032] According to the quality inspection results, the relevant data that failed the inspection will be marked as abnormal data; and the abnormal data will be traced and investigated according to the target quality rules and target business processing flow to determine the cause and / or location of the abnormality.

[0033] This specification also provides a multi-business data integration processing device based on new energy, which is applied to a node server in a multi-business data integration processing system, wherein the multi-business data integration processing system also includes: a big data storage and computing engine module, a data development tool, and a data management tool. The multi-business data integration processing system is connected to the new energy plant terminal through a first connection and to the power grid market terminal through a second connection. The device includes:

[0034] The first calling module is used to receive and respond to the target joint business data processing request, and call the data development tool to generate the corresponding target business processing flow;

[0035] The second calling module is used to call the data management tool to obtain the power generation capacity data and plant equipment operation data related to the new energy plant through the first connection according to the target business processing flow, and to obtain the power grid market data related to the power grid market through the second connection; and combine the power generation capacity data, plant equipment operation data and power grid market data to obtain the target joint business data;

[0036] The third calling module is used to call the big data storage and computing engine module to create a corresponding target data warehouse according to the target business processing flow; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and store the target joint business data in the original data layer;

[0037] The fourth calling module is used to call the data management tool to read the target joint business data through the original data layer according to the target business processing flow; pre-process the target joint business data to obtain the pre-processed target joint business data; and store the pre-processed target joint business data into the detailed data layer;

[0038] The fifth calling module is used to call the big data storage and calculation engine module to read the pre-processed target joint business data through the detailed data layer according to the target business processing flow, and combine the public attribute data in the public dimension layer to predict the business processing strategy for the target time period as the target processing result through joint calculation; and store the target processing result in the summary data layer;

[0039] A determination module, for determining an associated target business implementation end based on the target processing result; storing the target processing result in the application data layer; and establishing a third connection between the target business implementation end and the application data layer;

[0040] A sending module is used to send target prompt information about the target processing result to the target business implementation end; wherein the target prompt information carries at least an interface identifier of a data interface associated with a third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the relevant business of executing the target time period according to the target processing result.

[0041] The present specification also provides a server, including a processor and a memory for storing processor executable instructions, wherein the processor implements the relevant steps of the multi-business data integration processing method based on new energy when executing the instructions.

[0042] The present specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the relevant steps of the multi-business data integration processing method based on new energy are implemented.

[0043] Based on the new energy-based multi-business data integration and processing method, device and server provided in this specification, before specific implementation, a multi-business data integration and processing system can be constructed that is adapted to the new energy multi-business data processing scenario and includes at least a big data storage and computing engine module, data development tools, and data management tools. During specific implementation, a target joint business data processing request can be received and responded to. First, a data development tool is called to generate a corresponding target business processing flow. Then, according to the target business processing flow, a data management tool is called to obtain various different types of business data related to new energy provided by multiple business data source ends through corresponding connections for combination to obtain relatively complete and comprehensive target joint business data. Then, according to the target business processing flow, a big data storage and computing engine module is called to create a target data warehouse including an original data layer, a detailed data layer, a public dimension layer, a summary data layer, and an application data layer. The target joint business data is first stored in the original data layer. Then, according to the target business processing flow, the big data storage and computing engine module and the data management tool can be called, combined with the target data warehouse, to integrate different types of business data through joint operations, and predict the business processing strategy for the target time period as the target processing result. Then, a target business implementation end associated with the target processing result is determined, and the target processing result is reached to the target business implementation end, so that the target business implementation end can adjust the execution of the relevant business of the target time period in a targeted manner according to the business processing strategy. This makes it better adapted to new energy multi-business data processing scenarios, and can comprehensively and effectively integrate and utilize various types of business data related to new energy provided by multiple business data sources, mine and utilize the data value implicit in a large amount of business data, and accurately help the business implementation end make decisions, so that the business implementation end can more efficiently and accurately adjust and execute various related businesses such as production, marketing, operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flowchart of a multi-business data integration processing method based on new energy provided by an embodiment of this specification;

[0046] Figure 2 It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0047] Figure 3It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0048] Figure 4 It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0049] Figure 5 It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0050] Figure 6 It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0051] Figure 7 It is a schematic diagram of the structure of a server provided by an embodiment of this specification;

[0052] Figure 8 This is a schematic diagram of the structure of a multi-service data integration processing device based on new energy provided by an embodiment of this specification;

[0053] Fig. 9 It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0054] Fig.10 It is a schematic diagram of an embodiment of a multi-service data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example;

[0055] Fig.11 It is a schematic diagram of an embodiment of a multi-business data integration processing method based on new energy provided by an embodiment of this specification, in a scenario example. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0057] It should be noted that the user-related information and data involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by relevant parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users or relevant parties to choose to authorize or refuse.

[0058] It should also be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0059] See also Figure 1 As shown, the embodiment of this specification provides a multi-service data integration processing method based on new energy. Among them, the method is specifically applied to the node server side in the multi-service data integration processing system. Figure 2 As shown, the multi-business data integration processing system also includes: a big data storage and computing engine module, a data development tool, and a data management tool. The multi-business data integration processing system is connected to the new energy plant terminal through a first connection and to the power grid market terminal through a second connection. In specific implementation, the method may include the following contents:

[0060] S101: receiving and responding to a target joint business data processing request, and calling a data development tool to generate a corresponding target business processing flow;

[0061] S102: according to the target business processing flow, calling the data management tool to obtain power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and to obtain power grid market data related to the power grid market through the second connection; and combining the power generation capacity data, plant equipment operation data and power grid market data to obtain the target joint business data;

[0062] S103: According to the target business processing flow, the big data storage and computing engine module is called to create a corresponding target data warehouse; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and the target joint business data is stored in the original data layer;

[0063] S104: according to the target business processing flow, calling the data management tool to read the target joint business data through the original data layer; preprocessing the target joint business data to obtain the preprocessed target joint business data; and storing the preprocessed target joint business data into the detailed data layer;

[0064] S105: According to the target business processing flow, the big data storage and calculation engine module is called to read the target joint business data through the detailed data layer and predict the business processing strategy for the target time period as the target processing result through joint operation based on the pre-processed target joint business data and the public attribute data in the public dimension layer; and the target processing result is stored in the summary data layer;

[0065] S106: Based on the target processing result, determine the associated target service implementation end; store the target processing result in the application data layer; and establish a third connection between the target service implementation end and the application data layer;

[0066] S107: Sending target prompt information about the target processing result to the target business implementation end; wherein, the target prompt information carries at least an interface identifier of the data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the relevant business to be executed in the target time period according to the target processing result.

[0067] Specifically, the multi-business data integration and processing system can be connected to a plurality of different business data source terminals of the new energy industry to access and obtain a variety of different types of business data related to new energy.

[0068] Among them, the above-mentioned business data source ends may at least include: new energy plants, power grid market ends, etc.

[0069] Furthermore, the above-mentioned new energy plant and station end may specifically include: wind power station, solar power station, hydropower station, etc. The above-mentioned power grid market end may specifically include: distribution network, power market, carbon emission market, etc.

[0070] Accordingly, the business data related to new energy obtained based on the new energy plant side as the data source may at least include: power generation capacity data, plant equipment operation type data, etc. The business data related to new energy obtained based on the power grid market side as the data source may at least include: power grid market data, etc.

[0071] Furthermore, the above-mentioned power generation capacity data may specifically include one or more of the following: power generation data of a specified time period, power generation data of a historical reference time period, wind speed, temperature, humidity, rainfall, etc.

[0072] The above-mentioned plant equipment operation data may specifically include one or more of the following: plant type, equipment number, equipment operation status parameters, equipment cumulative operation time, fault detection occurrence time, maximum maintenance start interval, mean fault repair time, the most recent fault maintenance record, etc.

[0073] The above-mentioned power grid market data includes one or more of the following: current electricity prices, electricity prices at the same time in history, effective power supply of the current power grid, electricity prices at the same time in history, material costs, labor costs, energy storage costs, etc.

[0074] Of course, it should be noted that the business data listed above is only a schematic illustration. In specific implementation, according to specific circumstances and processing requirements, the accessed and used business data may also include other types of business data. This specification does not limit this.

[0075] Specifically, the multi-business data integration processing system may also be connected to a plurality of different business implementation terminals.

[0076] The business implementation end may specifically include at least one of the following: intelligent operation and maintenance system, power marketing system, intelligent production control system, etc. In addition, the business implementation end may also include: monitoring center, operation and maintenance personnel, decision-making department, etc.

[0077] The target time period may specifically be a current time period, such as today, or a future time period, such as the next month.

[0078] The above-mentioned business processing strategy can be specifically understood as a recommendation strategy for guiding the associated business implementation end to adjust and execute the related business in the target time period. The above-mentioned business processing strategy can be an independent strategy for a single business implementation end, or a linkage strategy for multiple associated business implementation ends at the same time.

[0079] In specific implementation, the above-mentioned multi-business data integration and processing system can be used to comprehensively and completely access and integrate various types of multi-source heterogeneous business data provided by different business data sources; and the above-mentioned business data integration and processing system can be used to fully utilize the integrated various types of business data, through joint operations, effectively mine and utilize the data value of business data, accurately predict the appropriate business processing strategy, and provide it to the related target business implementation end as a decision-making reference, so that the target business implementation end can more efficiently and accurately adjust the execution of subsequent related businesses and obtain better business results.

[0080] Specifically, for example, in a production and operation and maintenance scenario, the multi-business data integration and processing system can be used to integrate and utilize power generation capacity data, plant equipment operation data, and power grid market data to determine a production and operation joint strategy for the intelligent operation and maintenance system and the intelligent production control system, and provide them as target processing results to the intelligent operation and maintenance system and the intelligent production control system (as the associated target business implementation end), so that the intelligent operation and maintenance system and the intelligent production control system can respectively adjust the subsequent production and power generation business and operation and maintenance inspection business in a targeted manner according to the above-mentioned production and operation joint strategy, so as to ensure the stable and safe operation of related equipment while ensuring the power generation capacity and taking into account the operation and maintenance costs.

[0081] For another example, in a production and marketing scenario, the multi-business data integration and processing system can be used to integrate and utilize power generation capacity data and power grid market data to determine the production and marketing joint strategies for the intelligent production control system and the power marketing system, and provide them as target processing results to the intelligent operation and maintenance system and the intelligent production control system (as the associated target business implementation end), so that the intelligent production control system and the power marketing system can respectively adjust the subsequent production and power generation business and the power marketing business in a targeted manner according to the above-mentioned production and marketing joint strategies, so as to ensure that the power generation capacity meets the requirements of the power grid while obtaining better marketing effects and relatively high profit data.

[0082] Of course, it should be noted that the above-mentioned production and operation scenarios and production and marketing scenarios are only schematic illustrations. In specific implementation, according to specific circumstances and processing requirements, it can also be applied to other relatively complex new energy multi-business data processing scenarios. This manual does not limit this.

[0083] Based on the above embodiments, by constructing and using a multi-business data integration and processing system, it can be better adapted to the new energy multi-business data processing scenario, and can comprehensively and effectively integrate and utilize various types of business data involving new energy provided by multiple business data sources, accurately assisting the business implementation end in making decisions, and enabling the business implementation end to more efficiently and accurately adjust and execute related businesses.

[0084] In some embodiments, the multi-service data integration processing system may further include a node server responsible for executing specific data processing.

[0085] Among them, the above-mentioned node server may specifically include a server responsible for data processing in the background that is applied to one side of a multi-business data integration processing system and can realize functions such as data transmission and data processing. Specifically, the node server may be, for example, an electronic device with data calculation, storage and network interaction functions. Alternatively, the node server may also be a software program running in the electronic device to provide support for data processing, storage and network interaction. In this embodiment, the number of servers included in the node server is not specifically limited. The node server may specifically be one server, or several servers, or a server cluster formed by several servers.

[0086] During specific implementation, the manager as a user can use the user terminal to initiate a target joint business data processing request to the node server, triggering the node server to use the relevant components in the multi-business data integration processing system to complete the corresponding data processing, and obtain the required target processing results for specific business scenarios, which are then provided to the target business implementation end to be managed.

[0087] The user terminal may include a front end applied to the user side and capable of realizing functions such as data collection and data transmission. Specifically, the user terminal may be, for example, an electronic device such as a desktop computer, a tablet computer, a laptop computer, a smart phone, etc. Alternatively, the user terminal may also be a software application that can run in the above electronic devices.

[0088] In some embodiments, see Figure 2 As shown, the above-mentioned big data storage and computing engine module may specifically include: a distributed file system (e.g., a Hadoop distributed file system), a database (e.g., an MPP database), an offline computing component (e.g., a Hive offline computing component), a real-time computing component (e.g., a Flink real-time computing component), an in-memory computing component (e.g., a Spark in-memory computing component), etc.

[0089] The above-mentioned data development tools may at least include: data integration development tools, data processing flow development tools (for example, data ETL development tools), etc.

[0090] The above-mentioned data integration development tool may further include: real-time data integration development functional components, offline data integration development functional components, etc.

[0091] The above-mentioned ETL (Extract-Transform-Load) can be specifically understood as a process for describing the extraction (extract), transformation (transform), and loading (load) of data from the source end to the destination end.

[0092] The above-mentioned data management tools may at least include: data standard management tools, data quality management tools, etc.

[0093] The data quality management tool may also be configured with a preset data quality detection rule set, which may include preset data quality detection rules corresponding to multiple different types of data.

[0094] In specific implementation, the above-mentioned target joint business data processing request may specifically carry target requirements and configuration parameters. Accordingly, the node server may respond to the target joint business data processing request, call the data processing flow development tool in the data development tool, and automatically develop and configure the corresponding target business processing flow, such as the ETL processing flow, according to the target requirements and configuration parameters.

[0095] In specific implementation, the node server can call the data management tool according to the target business processing flow to collect and obtain the corresponding power generation capacity data, plant equipment operation data, and power grid market data as initial data through the first connection between the new energy plant and the second connection between the power grid market. The first connection is adapted to the new energy plant and the second connection is adapted to the power grid market.

[0096] At the same time, the node server can also call the big data storage and computing engine module to use the distributed file system and database to create the corresponding target data warehouse according to the target business processing flow.

[0097] Among them, see Figure 3 As shown, the target data warehouse may specifically include the following five layers: original data layer, detailed data layer, common dimension layer, summary data layer and application data layer.

[0098] Among them, the above-mentioned original data layer can be an ODS layer (Operational Data Store), which can be specifically used to store multiple business data (for example, which can be recorded as original data) directly accessed through multiple business data source ends; and, in conjunction with the target business processing flow, provide a temporary buffer for the target data warehouse.

[0099] The detailed data layer may be a DWD layer (Data Warehouse Detail), which may be specifically used to store business data that has been pre-processed, such as standardization processing, data cleaning processing, etc., based on fine-grained storage.

[0100] The above-mentioned public dimension layer may be a DIM layer (Dimension, universal dimension layer), which may be used to store public attribute data shared between different businesses, such as basic information of new energy plants, shared dictionary tables, basic information of plant equipment, etc. In specific implementation, the public dimension layer may be associated with the other four data layers to complete the relevant business data processing.

[0101] The summary data layer may be a DWS layer (Data WareHouse Summary), which may be specifically used to store processing results obtained by corresponding operations (including model operations, indicator operations, statistical operations, etc.) using the data in the detailed data layer.

[0102] The application data layer may be an ADS layer (Application Data Store), which may be specifically used to store result data that is secondary processed on the processing results in the summary data layer and adapted to the associated business implementation end.

[0103] Specifically, the above-mentioned application data layer can be constructed using relational OLTP data, OLAP data and MPP database; and when the application data layer has high concurrency requirements for data sharing needs at the business implementation end, it can also be used in conjunction with a Redis cluster or an ES cluster to improve the efficiency of data call concurrency.

[0104] The target data warehouse using the above-mentioned hierarchical structure can, on the one hand, make the data storage at different stages relatively more reasonable, and facilitate relatively more efficient centralized reading and use; on the other hand, it can also separately connect the application layer in the target data warehouse to the business implementation end, so as to only share relevant result data with the business implementation end, thereby effectively avoiding the leakage of other data (for example, original data, etc.) when sharing result data with the business implementation end, and better protecting the security of relevant data.

[0105] In some embodiments, the data management tool may further include a data encryption tool. The data encryption tool may be configured with a preset homomorphic encryption rule and a preset sensitive information data table. The preset homomorphic encryption rule may specifically be a double trapdoor homomorphic encryption algorithm. Based on the preset homomorphic encryption rule, "data content is invisible but available" can be achieved.

[0106] After calling the data management tool to obtain power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and obtaining power grid market data related to the power grid market through the second connection, the method may further include the following contents when it is specifically implemented:

[0107] S1: Calling the data management tool to detect whether there is sensitive data in the power generation capacity data, plant equipment operation data, and power grid market data according to the preset sensitive information data table; and if it is determined that sensitive data exists, encrypting the sensitive data according to the preset homomorphic encryption rules to obtain the ciphertext data of the sensitive data;

[0108] S2: Calling the data management tool to replace the sensitive data in the power generation capacity data, plant equipment operation data, and power grid market data with the ciphertext data of the sensitive data; and destroying the sensitive data.

[0109] The preset sensitive information data table may specifically include field names of sensitive data that are relatively sensitive in different types of business data and need to be kept confidential.

[0110] Before specific implementation, business data processing records from different business data sources and different business implementation ends can be collected; based on the above business data processing records, information data with encryption frequency higher than the preset frequency threshold is counted and screened out as candidate sensitive data; at the same time, confidentiality requirement texts fed back by different business data sources and different business implementation ends, as well as confidentiality regulations texts published to the public are collected, and further sorted out on the basis of the above candidate sensitive data to determine sensitive data that meets the requirements; then the field names of the above sensitive data are obtained and used to construct a preset sensitive information data table.

[0111] During specific implementation, the node server can first call the data management tool to detect the field names of each specific data in the power generation capacity data, plant equipment operation data, and power grid market data according to the preset sensitive information data to find out the sensitive data that needs to be encrypted; then encrypt the above sensitive data according to the preset homomorphic encryption rules to obtain the corresponding ciphertext data of the sensitive data; and use the ciphertext data of the sensitive data to replace the sensitive data in the original data, and destroy the original plaintext sensitive data.

[0112] In this way, for the non-sensitive data in the original data, the original data in plain text can continue to be used for specific operations and other related data processing; for the sensitive data in the original data, its corresponding ciphertext data can be used to participate in specific operations and other related data processing in the ciphertext state. This can effectively prevent the data information of some sensitive data in the original data from being leaked, and better protect the data security of related business data.

[0113] In some embodiments, see Figure 4 As shown, the above-mentioned calling data management tool obtains the power generation capacity data related to the new energy plant through the first connection. When implemented specifically, the following contents may be included:

[0114] S1: Detect and identify the data type of power generation capacity data;

[0115] S2: For streaming data in power generation capacity data, call the real-time data integration development function component in the data development tool, combine Flink CDC and Kafka to build the corresponding first connection; call the data management tool to obtain and cache the streaming data in power generation capacity data through the first connection;

[0116] S3: For the batch data in the power generation capacity data, call the offline data integration development function component in the data development tool to build a corresponding first connection; call the data management tool to obtain the batch data in the power generation capacity data through the first connection.

[0117] The above-mentioned Flink CDC (Change Data Capture) can specifically refer to Flink CDCConnector, which is a connector for capturing and processing database changes. Flink CDC can be used to capture data changes from relational databases, such as insert, update, and delete operations, and synchronize these changes to Flink stream processing applications.

[0118] The above-mentioned Kafka may specifically refer to a high-throughput distributed publish-subscribe messaging system, which may be used to process all action stream data of consumers on a website.

[0119] In specific implementation, for streaming data, the node server can specifically call the real-time data integration development function component in the data development tool, and build the corresponding first connection in combination with Flink CDC and Kafka.

[0120] In this way, we can use the advantages of Flink CDC to collect relevant streaming data in real time online to avoid omissions in data collection. At the same time, we can also use the advantages of Kafka to cache real-time data streams that cannot be processed and stored in time to prevent data flow blockage from causing excessive system load and affecting the stability of the overall system operation.

[0121] For batch data, the node server can specifically call the offline data integration development function components in the data management tool, cooperate with the link between the business data source and the database table, and follow the corresponding library-table mapping relationship and scheduling strategy, so as to efficiently and stably collect and store the full data or incremental data of different business data sources on a regular basis, avoiding collection errors.

[0122] It should be noted that for plant equipment operation data and power grid market data, reference can be made to the above embodiments for power generation capacity data, different data types of streaming data and batch data can be distinguished, and matching collection and processing methods can be used to perform targeted collection and processing respectively. This specification does not elaborate on this.

[0123] In some embodiments, the above-mentioned preprocessing of the target joint service data may include:

[0124] S1: Calling the data management tool to standardize the field type, field name, field length, data type, and data format in the target joint business data according to the preset standardization rules, and obtaining the standardized target joint business data;

[0125] S2: According to the target business processing flow, call the data management tool to build data tables and data models related to the target joint business data processing.

[0126] The above data table can be specifically understood as a data structure for participating in subsequent specific data processing such as joint operation, etc. The above data model can be specifically understood as a business model for subsequent specific data processing.

[0127] When performing specific preprocessing, the target joint business data can be preprocessed in various ways, including data standardization, data cleaning, data filtering, etc., by calling the data standard management tool in the data management tool.

[0128] Based on the above embodiment, the target joint business data can be pre-processed so that different types of business data from different business data sources can be relatively unified and compatible, and suitable for subsequent relatively more complex data processing.

[0129] In some embodiments, the data model may specifically include: a power generation capacity prediction model, a distribution network power prediction model, a production and marketing decision model, etc.

[0130] Of course, it should be noted that the business models listed above are only schematic illustrations. In specific implementation, according to specific business scenarios and processing requirements, other types of business models such as equipment failure prediction models, power market change prediction models, and data statistical analysis models may also be included. This specification does not limit this.

[0131] In specific implementation, the node server can train the required data model through transfer learning based on the target business requirements and configuration parameters in the target joint business data processing request, combined with the template model pre-trained using the basic sample data set.

[0132] Specifically, for example, take building a power generation capacity prediction model as an example.

[0133] Before specific implementation, the power generation capacity data and plant equipment operation data in the historically accumulated basic sample data set with good generalization can be used as the first joint sample data to train the initial model, obtain the corresponding template model, and store it in the preset template model library.

[0134] In specific implementation, the template model can be obtained from a preset template model library according to the target joint business data processing request; at the same time, according to the target joint business data processing request; and according to the target business requirements and configuration parameters in the target joint business data processing request, the model parameters of the template model are adjusted, for example, the output layer and input layer of the template model are modified to adapt to the target business requirements, the currently acquired power generation capacity data, and the plant equipment operation data, so as to obtain the modified template model; then the power generation capacity data and the plant equipment operation data of the historical time period (for example, the last six months) are obtained from the detailed data layer, and the data that meets the requirements are filtered out and combined as the second joint sample data; the above second joint sample data is used to train the modified template model, so that the power generation capacity prediction model that meets the requirements can be obtained more efficiently and accurately.

[0135] In some embodiments, see Figure 5 As shown, according to the target business processing flow, the big data storage and computing engine module is called to read through the detailed data layer and predict the business processing strategy for the target time period as the target processing result through joint operation based on the pre-processed target joint business data and the public attribute data in the public dimension layer. When implemented specifically, the following contents may be included:

[0136] S1: According to the target business processing flow, the big data storage and computing engine module is called to read the target joint business data through the detailed data layer and run the power generation capacity prediction model and the distribution network power prediction model respectively according to the pre-processed target joint business data and the public attribute data in the public dimension layer to obtain the power generation capacity prediction results and the distribution network demand power prediction results of the target time period;

[0137] S2: according to the target business processing flow, calling the big data storage and computing engine module to combine the power generation capacity forecast result of the target time period and the power demand forecast result of the distribution network in the target time period, and obtaining the target integrated data group for the target time period;

[0138] S3: According to the target business processing flow, call the big data storage and computing engine module to use the target integrated data group, combine the public attribute data in the public dimension layer, run the production and marketing decision model, and predict the production and marketing joint strategy for the target time period as the target processing result.

[0139] In specific implementation, the preprocessed power generation capacity data and preprocessed plant equipment operation data required in the preprocessed target joint business data can be extracted and imported into the first data table; at the same time, the required public attribute information is screened out from the public dimension layer and imported into the first data table according to the corresponding association relationship, so that the preprocessed power generation capacity data and preprocessed plant equipment operation data in the first data table are respectively integrated and associated with the corresponding first public attribute data to obtain the integrated and associated first data table; the integrated and associated first data table is input into the power generation capacity prediction model, and the power generation capacity prediction model is run to obtain the power generation capacity prediction result for the corresponding target time period.

[0140] In a similar manner, the pre-processed target joint business data can be used in combination with the required second public attribute data to construct an integrated and associated second data table; then the distribution network power forecasting model is used to process the integrated and associated second data table to obtain the distribution network demand power forecast result for the corresponding target time period.

[0141] In specific implementation, the power generation capacity forecast results and the distribution network demand forecast results corresponding to each time point in the target time period can be determined according to the power generation capacity forecast results of the target time period and the distribution network demand forecast results of the target time period, and the power generation capacity forecast data and the distribution network demand forecast data corresponding to each time point in the target time period can be combined to obtain multiple data pairs; each data pair corresponds to a time point in the target time period; and then the multiple data pairs are combined in sequence according to the chronological order of the time points to obtain a target integrated data group for the target time period.

[0142] Furthermore, the public dimension layer can be queried to obtain the required third public attribute data; and the third public attribute data can be fused with the target integrated data group to obtain fused data; and the fused data can be input into the production and marketing decision model, and the production and marketing decision model can be run to obtain the production and marketing joint strategy for the target time period.

[0143] Based on the above embodiment, by calling the big data storage and computing engine module, the target joint business data can be fully utilized to automatically predict the production and marketing joint strategy with high reference value for the target time period as the target processing result efficiently and accurately.

[0144] In some embodiments, after obtaining the target processing result, the business implementation end associated with the target processing result can be determined according to the specific content of the target processing result as the target business implementation end; then the target processing result is obtained from the public dimension layer and processed according to the basic information of the target business implementation end to obtain a target processing result that is compatible with the target business implementation end.

[0145] During specific implementation, the target processing result may be stored in the application data layer, and a third connection may be established between the application layer and the target service implementation end according to the address information of the target service implementation end.

[0146] Furthermore, the interface identifier of the data interface of the third connection can also be obtained; at the same time, a connection credential for the third connection is generated; and then the target prompt information carrying the interface identifier and the connection credential is sent to the target service implementation end.

[0147] Correspondingly, the target service implementation end can respond to the target prompt information, find and use the corresponding data interface to access the third connection according to the interface identifier, and then use the connection credential to download the target processing result from the application data layer through the third connection. In this way, the target processing result can be delivered to the corresponding target service implementation end more safely and accurately.

[0148] After obtaining the target processing result, the target business implementation end can adjust the relevant businesses in the target time period in a targeted manner according to the corresponding business processing strategy.

[0149] Specifically, for example, in production and marketing scenarios, the intelligent production control system can adjust the power generation power at each time point in the target time period in a targeted manner according to the production and marketing strategy; at the same time, determine how much additional electricity to feed into the grid at a specific time point in the target time period, how much additional electricity to store in the compressed air energy storage system, or how much electricity to release from the compressed air energy storage system to supplement the grid.

[0150] In addition, the power marketing system can adjust the power quotes at various time points in the target time period in a targeted manner according to the production and marketing strategy; at the same time, it can flexibly adjust the amount of electricity fed into the power grid at various time points.

[0151] This will enable new energy companies to obtain better market returns while meeting the necessary grid power generation requirements.

[0152] In some embodiments, see Figure 6 As shown, when the method is implemented specifically, it may also include the following contents:

[0153] S1: calling a data management tool to determine, from a preset data quality detection rule set, preset data quality detection rules corresponding to relevant data involved in a target business process flow, and combining them to obtain target data quality detection rules;

[0154] S2: Call the data management tool to perform quality inspection on the relevant data appearing in the multi-business data integration processing system according to the target data quality inspection rules;

[0155] S3: According to the quality inspection results, the relevant data that failed the inspection are marked as abnormal data; and the abnormal data is traced and investigated according to the target quality rules and target business processing flow to determine the cause and / or location of the abnormality.

[0156] In specific implementation, the abnormal data can be reviewed in fine-grained according to the target quality detection rules to determine the abnormal problem manifestation of the abnormal data, for example, the field name does not meet the requirements; or the number of characters in the data value exceeds the threshold, etc.; then, based on the abnormal problem manifestation, the target business processing flow is sorted out to screen out the processing links that may be related to the abnormal problem manifestation as suspicious links; then, the processing rules and module components used in the suspicious links are checked separately to ultimately determine the abnormal cause and / or abnormal location that caused the above-mentioned abnormal problem manifestation.

[0157] Then, according to the cause of the abnormality, a matching target abnormality repair solution can be determined from the preset abnormality repair solution set; then, according to the target abnormality repair solution, the relevant responsible terminal is contacted in time to carry out targeted repair processing on the above abnormal location. After completing the above repair processing, the relevant business data processing is re-performed.

[0158] During specific implementation, the data quality management tool in the data management tool can be called to use the corresponding data quality detection rules to conduct online quality monitoring of the data that appears and is used during the operation of the multi-business data integration and processing system, promptly discover abnormal data with quality problems, and perform targeted processing to ensure that the data used in the entire processing process is accurate and reliable, thereby obtaining the target processing results with better accuracy and higher reference value.

[0159] Based on the above embodiments, comprehensive and effective quality control can be performed on the entire operation process of the multi-business data integration processing system to reduce data errors in the data processing process and obtain target processing results with better effects.

[0160] From the above, it can be seen that based on the multi-business data integration and processing method based on new energy provided in the embodiment of this specification, before specific implementation, a multi-business data integration and processing system can be constructed that is adapted to the new energy multi-business data processing scenario and includes at least a big data storage and computing engine module, data development tools, and data management tools. During specific implementation, a target joint business data processing request can be received and responded to. First, a data development tool is called to generate a corresponding target business processing flow. Then, according to the target business processing flow, a data management tool is called to obtain various types of business data related to new energy provided by multiple business data sources through corresponding connections for combination to obtain complete and comprehensive target joint business data. Then, according to the target business processing flow, a big data storage and computing engine module is called to create a target data warehouse including an original data layer, a detailed data layer, a public dimension layer, a summary data layer, and an application data layer. The target joint business data is first stored in the original data layer. Then, according to the target business processing flow, the big data storage and computing engine module and the data management tool can be called, combined with the target data warehouse, to integrate different types of business data through joint operations, and predict the business processing strategy for the target time period as the target processing result. Then, a target business implementation end associated with the target processing result is determined, and the target processing result is reached to the target business implementation end, so that the target business implementation end can adjust the execution of the relevant business of the target time period in a targeted manner according to the business processing strategy. This makes it better adapted to new energy multi-business data processing scenarios, and can comprehensively and effectively integrate and utilize various types of business data involving new energy provided by multiple business data sources, accurately helping the business implementation side to make decisions, so that the business implementation side can adjust and execute related businesses more efficiently and accurately.

[0161] This specification embodiment provides a server, referring to Figure 7 The server includes a network communication port 701, a processor 702 and a memory 703, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0162] The network communication port 701 may be specifically used to receive a target joint service data processing request.

[0163] The processor 702 can be specifically used to respond to the target joint business data processing request, call the data development tool to generate the corresponding target business processing flow; according to the target business processing flow, call the data management tool to obtain the power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and obtain the power grid market data related to the power grid market through the second connection; and combine the power generation capacity data, plant equipment operation data and power grid market data to obtain the target joint business data; according to the target business processing flow, call the big data storage and computing engine module to create the corresponding target data warehouse; wherein the target data warehouse at least includes: the original data layer, the detailed data layer, the common dimension layer, the summary data layer and the application data layer; and store the target joint business data in the original data layer; according to the target business processing flow, call the data management tool to read the target joint business data through the original data layer; pre-process the target joint business data to obtain the pre-processed target joint business data Combined business data; and the pre-processed target joint business data is stored in the detailed data layer; according to the target business processing flow, the big data storage and calculation engine module is called to read through the detailed data layer and, based on the pre-processed target joint business data, combined with the public attribute data in the public dimension layer, through joint operation, predict the business processing strategy for the target time period as the target processing result; and the target processing result is stored in the summary data layer; based on the target processing result, the associated target business implementation end is determined; the target processing result is stored in the application data layer; and a third connection is established between the target business implementation end and the application data layer; target prompt information about the target processing result is sent to the target business implementation end; wherein, the target prompt information carries at least an interface identifier of a data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the execution of related businesses in the target time period according to the target processing result.

[0164] The memory 703 may be specifically used to store corresponding instruction programs, and relevant data such as target business processing flow.

[0165] Based on the above method, the relevant structural performance of the server can be effectively utilized, the data processing speed of the electronic equipment can be improved, and the multi-business data integration processing based on new energy can be efficiently realized.

[0166] In this embodiment, the network communication port 701 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0167] In this embodiment, the processor 702 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.

[0168] In this embodiment, the memory 703 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function that has no physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0169] The embodiment of the present specification also provides a computer-readable storage medium based on the above-mentioned multi-business data integration processing method based on new energy, and the computer-readable storage medium stores computer program instructions, which, when executed, implement: receiving and responding to a target joint business data processing request, calling a data development tool to generate a corresponding target business processing flow; according to the target business processing flow, calling a data management tool to obtain power generation capacity data and plant equipment operation data related to a new energy plant through a first connection, and obtaining power grid market data related to a power grid market through a second connection; and combining the power generation capacity data, plant equipment operation data and power grid market data to obtain target joint business data; according to the target business processing flow, calling a big data storage and computing engine module to create a corresponding target data warehouse; wherein the target data warehouse includes at least: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and storing the target joint business data in the original data layer; according to the target business processing flow, calling the data management tool to obtain the target joint business data through the original data layer Read the target joint business data; pre-process the target joint business data to obtain the pre-processed target joint business data; and store the pre-processed target joint business data in the detailed data layer; according to the target business processing flow, call the big data storage and calculation engine module to read through the detailed data layer and predict the business processing strategy for the target time period as the target processing result based on the pre-processed target joint business data combined with the public attribute data in the public dimension layer through joint operation; and store the target processing result in the summary data layer; based on the target processing result, determine the associated target business implementation end; store the target processing result in the application data layer; and establish a third connection between the target business implementation end and the application data layer; send target prompt information about the target processing result to the target business implementation end; wherein the target prompt information at least carries the interface identifier of the data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the execution of related businesses in the target time period according to the target processing result.

[0170] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0171] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other implementations and will not be described in detail here.

[0172] The embodiment of the present specification also provides a computer program product, which at least includes a computer program, and when the computer program is executed by a processor, it implements the following method steps: receiving and responding to a target joint business data processing request, calling a data development tool to generate a corresponding target business processing flow; according to the target business processing flow, calling a data management tool to obtain power generation capacity data and plant equipment operation data related to a new energy plant through a first connection, and obtaining power grid market data related to a power grid market through a second connection; and combining the power generation capacity data, plant equipment operation data and power grid market data to obtain target joint business data; according to the target business processing flow, calling a big data storage and computing engine module to create a corresponding target data warehouse; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and storing the target joint business data in the original data layer; according to the target business processing flow, calling a data management tool to read the target joint business data through the original data layer; The business data is preprocessed to obtain the preprocessed target joint business data; and the preprocessed target joint business data is stored in the detailed data layer; according to the target business processing flow, the big data storage and calculation engine module is called to read through the detailed data layer and predict the business processing strategy for the target time period as the target processing result based on the preprocessed target joint business data and the public attribute data in the public dimension layer through joint operation; and the target processing result is stored in the summary data layer; based on the target processing result, the associated target business implementation end is determined; the target processing result is stored in the application data layer; and a third connection is established between the target business implementation end and the application data layer; target prompt information about the target processing result is sent to the target business implementation end; wherein the target prompt information carries at least an interface identifier of a data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the execution of related businesses in the target time period according to the target processing result.

[0173] See also Figure 8As shown, the embodiment of this specification also provides a multi-business data integration processing device based on new energy, which can be specifically applied to a node server in a multi-business data integration processing system, wherein the multi-business data integration processing system also includes: a big data storage and computing engine module, a data development tool, and a data management tool. The multi-business data integration processing system is connected to the new energy plant end through a first connection and to the power grid market end through a second connection. The device can specifically include the following structural modules:

[0174] The first calling module 801 may be specifically used to receive and respond to a target joint business data processing request, and call a data development tool to generate a corresponding target business processing flow;

[0175] The second calling module 802 may be specifically used to call the data management tool to obtain power generation capacity data and plant equipment operation data related to the new energy plant through the first connection according to the target business processing flow, and to obtain power grid market data related to the power grid market through the second connection; and to combine the power generation capacity data, plant equipment operation data and power grid market data to obtain the target joint business data;

[0176] The third calling module 803 may be specifically used to call the big data storage and computing engine module to create a corresponding target data warehouse according to the target business processing flow; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and store the target joint business data in the original data layer;

[0177] The fourth calling module 804 may be specifically used to call the data management tool to read the target joint business data through the original data layer according to the target business processing flow; pre-process the target joint business data to obtain the pre-processed target joint business data; and store the pre-processed target joint business data into the detailed data layer;

[0178] The fifth calling module 805 may be specifically used to call the big data storage and computing engine module to read the target joint business data through the detailed data layer and predict the business processing strategy for the target time period as the target processing result through joint operation according to the target business processing flow, combined with the public attribute data in the public dimension layer; and store the target processing result in the summary data layer;

[0179] The determination module 806 may be specifically used to determine the associated target service implementation end based on the target processing result; store the target processing result in the application data layer; and establish a third connection between the target service implementation end and the application data layer;

[0180] The sending module 807 can be specifically used to send target prompt information about the target processing result to the target business implementation end; wherein the target prompt information carries at least an interface identifier of the data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the relevant business of executing the target time period according to the target processing result.

[0181] In one embodiment, after calling the data management tool to obtain power generation capacity data and plant equipment operation data related to the new energy plant through a first connection, and obtaining power grid market data related to the power grid market through a second connection, the device can also be used when it is implemented: calling the data management tool to detect whether there is sensitive data in the power generation capacity data, plant equipment operation data, and power grid market data according to a preset sensitive information data table; and when it is determined that sensitive data exists, encrypting the sensitive data according to preset homomorphic encryption rules to obtain ciphertext data of the sensitive data; calling the data management tool to replace the sensitive data in the power generation capacity data, plant equipment operation data, and power grid market data with the ciphertext data of the sensitive data; and destroying the sensitive data.

[0182] In some embodiments, when the above-mentioned second calling module 802 is implemented, the data management tool can be called in the following manner to obtain the power generation capacity data related to the new energy plant through the first connection: detect and identify the data type of the power generation capacity data; for the streaming data in the power generation capacity data, call the real-time data integration development function component in the data development tool, and build the corresponding first connection in combination with Flink CDC and Kafka; call the data management tool to obtain and cache the streaming data in the power generation capacity data through the first connection; for the batch data in the power generation capacity data, call the offline data integration development function component in the data development tool to build the corresponding first connection; call the data management tool to obtain the batch data in the power generation capacity data through the first connection.

[0183] In some embodiments, when the above-mentioned fourth calling module 804 is implemented, the target joint business data can be pre-processed in the following manner: call the data management tool according to the preset standardization rules, and use the data standard management tool to standardize the field type, field name, field length, data type, and data format in the target joint business data to obtain the standardized target joint business data; according to the target business processing flow, call the data management tool to construct data tables and data models related to the target joint business data processing.

[0184] In some embodiments, the data model may specifically include: a power generation capacity prediction model, a distribution network power prediction model, a production and marketing decision model, etc.

[0185] In some embodiments, when the fifth calling module 805 is specifically implemented, the following method can be used to call the big data storage and computing engine module according to the target business processing flow, read through the detailed data layer, and according to the pre-processed target joint business data, combined with the public attribute data in the public dimension layer, through joint operation, predict the business processing strategy for the target time period as the target processing result: according to the target business processing flow, call the big data storage and computing engine module to read through the detailed data layer and according to the pre-processed target joint business data, combined with the public attribute data in the public dimension layer, respectively run the power generation capacity prediction model and the distribution network electricity prediction model to obtain the power generation capacity prediction result of the target time period and the distribution network demand electricity prediction result of the target time period; according to the target business processing flow, call the big data storage and computing engine module to combine the power generation capacity prediction result of the target time period and the distribution network demand electricity prediction result of the target time period to obtain the target integrated data group for the target time period; according to the target business processing flow, call the big data storage and computing engine module to use the target integrated data group, combined with the public attribute data in the public dimension layer, to run the production and marketing decision model to predict the production and marketing joint strategy for the target time period as the target processing result.

[0186] In some embodiments, when the device is implemented, it can also be used to: call the data management tool to determine the preset data quality detection rules corresponding to the relevant data involved in the target business processing flow from the preset data quality detection rule set, and combine them to obtain the target data quality detection rules; call the data management tool to perform quality detection on the relevant data appearing in the multi-business data integration processing system according to the target data quality detection rules using the data quality management tool; based on the quality detection results, mark the relevant data that fails the detection as abnormal data; and trace the abnormal data according to the target quality rules and the target business processing flow to determine the cause and / or location of the abnormality.

[0187] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] It can be seen from the above that the multi-business data integration and processing device based on new energy provided in the embodiment of this specification can be well adapted to the new energy multi-business data processing scenario, and can comprehensively and effectively integrate and utilize various types of business data involving new energy provided by multiple business data sources, accurately help the business implementation end to make decisions, and enable the business implementation end to adjust and execute related businesses more efficiently and accurately.

[0189] In a specific scenario example, the multi-business data integration processing method based on new energy provided in this specification can be applied to the interaction and integrated analysis of new energy production and marketing data.

[0190] In this scenario example, currently large-scale new energy power generation enterprises are using or planning to use digital technology to support production and marketing business, such as building or planning power marketing systems, intelligent operation and maintenance systems, production control systems and other digital platforms for auxiliary management and decision-making. These systems have played a certain role in innovating the production and marketing business management model. At the same time, the production and marketing systems have accumulated a large amount of dispersed data in their respective operations. If the production and marketing systems are independent, it is easy to cause the data island effect, resulting in the value of massive multi-source heterogeneous data cannot be fully mined, and even data duplication and calculation may occur, resulting in unnecessary waste of resources. With the development of the multi-category market environment, the traditional power marketing model has gradually failed to adapt to the new environment. Therefore, the new energy power field is exploring a new management model for the integration of power production and marketing. The technical basis for the formation of the new model is to integrate and analyze the data of the production and marketing systems. At present, the typical method for data integration between production and marketing systems is to open interface call permissions or business database table read permissions between each system.

[0191] However, if we consider the continued growth of future business scenarios and data scale, there are great limitations in terms of business database performance bottlenecks, network security, and system and data management. Therefore, the difficulty in the circulation of production and marketing data and insufficient governance are still the main shortcomings that plague the formation and implementation of a new integrated management model for power production and marketing.

[0192] Based on the above considerations, this scenario example proposes a new energy production and marketing data interaction and integrated analysis method for the data needs and data flow process in the wind farm production and marketing integrated business scenario in new energy. This method breaks through the new energy production and marketing data barriers, realizes the collection and acquisition of required data, unified management and analysis and calculation, and distribution of result data, and improves the efficiency of data interaction, flow, and integrated analysis.

[0193] The overall architecture of the solution consists of two parts, namely data governance basic support tools and multi-source heterogeneous data governance. Data governance basic support tools (for example, a multi-business data integration processing system that includes at least data management tools, data development tools, and big data storage and computing engine modules) are used for data storage and computing, data management, and data development processes; multi-source heterogeneous data governance is based on the data governance basic support tools to establish a production and marketing data warehouse (for example, a target data warehouse). The process of establishing a data warehouse involves data warehouse design, data standardization, data quality monitoring, data indicator calculation, and result data distribution. Based on the above process, the original data required for the input parameters of the production and marketing integration model is standardized and data quality monitored, and data processing and distribution are performed according to the content and format of the model's input parameters. Finally, the output result data of the model is distributed to the intelligent production control system. For specific implementation, refer to Fig. 9 , Fig.10 As shown, the following steps may be included.

[0194] S1, build the basic support tool module for data governance.

[0195] Preferably, the data governance basic support tool module in step S1 integrates the big data storage and computing engine submodule (for example, the big data storage and computing engine module), the data management submodule (data management tool), and the data development submodule (data development tool). Fig.10 As shown, the storage and computing engine submodule includes big data storage, big data computing and cluster management; the data management submodule includes data standard management, data quality management, and data model management; the data development submodule includes data source management, data integration development, data ETL development, and task operation and maintenance monitoring.

[0196] Preferably, the specific construction method of step S1 may include the following contents:

[0197] S11. Build storage and computing engines, relying on cloud computing platforms, to build distributed storage modules and computing clusters with high reliability, scalability and low cost. The storage module is equipped with a hybrid Hadoop distributed file system and a relational distributed OLAP database based on the MPP large-scale parallel processing architecture to cope with the high-speed access scenario of massive data.

[0198] The computing module uses the data warehouse tool Hive to process batch data offline, and is paired with Spark based on memory computing to improve the efficiency of massive data processing. At the same time, Flink and Kafka are used to process streaming data in real time. In order to ensure the stable operation of the system, comprehensive cluster management functions are also adopted, including cluster operation overview, cluster host management, cluster component management, and cluster alarm services.

[0199] S12. Build data standard management tools to support the entire process of data standardization, provide data standard formulation and data standard publishing functions, ensure consistency in subsequent data modeling, data development and data application, perform data standardization management from the data source, and improve the efficiency of the data flow process.

[0200] Data standard formulation supports online definition of field attributes, enumerated data dictionaries, units of measurement, and naming roots, and provides version management, templated batch import, and batch export functions; data standard publishing supports submitting publishing applications, reviewing publishing applications, and canceling publishing.

[0201] S13. Build data quality management tools to support quality monitoring of batch data and streaming data flows. Conduct data quality monitoring and generate quality reports by configuring monitoring rules, developing data quality monitoring tasks, and executing data quality monitoring tasks to support subsequent tracking, analysis, and guide data providers to rectify data quality issues, ultimately achieving closed-loop management of data quality.

[0202] Preferably, offline data quality monitoring configures offline data quality monitoring tasks for batch data and provides a query function for data quality monitoring results. The task configuration requires selecting the added data source, adding table-level and field-level monitoring rules, configuring task scheduling strategies, and setting alarm methods; the quality monitoring results dynamically display the feedback information of each task single scheduling instance, including execution status, whether an alarm is triggered, whether the alarm is sent successfully, etc.

[0203] Real-time data quality monitoring provides data quality monitoring tasks for streaming data configuration and provides queries on data quality monitoring results. The task configuration requires selecting the added real-time streaming data source, adding table-level and field-level monitoring rules, configuring task execution strategies, and setting alarm methods. The quality monitoring results dynamically display feedback information for each single execution instance of the task, including execution status, whether an alarm is triggered, whether the alarm is sent successfully, etc.

[0204] S14. Build a data source management tool that supports adding, updating, and deleting data sources, so that you can select the data source you need to connect to during data task development. The supported data source types include but are not limited to MySQL, Hive, HDFS, Kafka, and Postgresql.

[0205] S15. Build data integration development tools, including real-time data integration task development and offline data integration task development, based on the characteristics of data generation.

[0206] Preferably, real-time data integration task development is usually used to collect streaming data such as measurement points with high data generation frequency and power forecasting. It is based on Flink CDC technology and establishes the field mapping relationship between the source table and the target table through a visual page configuration method to realize real-time collection of streaming data.

[0207] Offline data integration task development is usually used to collect batch data that does not require high real-time performance. This function supports both full and incremental modes. The full mode is to synchronize all data in the source table to the target table at one time according to the configured field mapping relationship during each scheduling process, while the incremental mode is to collect only the data that has changed in the source table since the last scheduling during each scheduling process, and the data that has not changed will not be collected repeatedly.

[0208] S16. Build data ETL development tools, mainly supporting tasks such as data preprocessing and cleaning, indicator calculation, data distribution and data synchronization. According to the data usage scenario, real-time task development and offline task development functions are provided.

[0209] Preferably, real-time task development is based on the Flink real-time computing engine, and offline task development is based on offline computing engines such as Hive and Spark. Both methods support syntax verification, online debugging, viewing run logs and execution results, lineage analysis and preview, and reference to custom functions, and provide flexible task scheduling functions for offline development.

[0210] S17. Build a task monitoring and operation and maintenance tool to monitor the current running status of the data integration tasks and data ETL tasks that have been started and run, and support intelligent monitoring and alarming of real-time and offline tasks. If the monitoring rules are triggered, an automatic alarm will be issued to the designated recipient and the alarm record will be saved.

[0211] S2, build a new energy production and marketing data warehouse (eg, target data warehouse).

[0212] The production and marketing data warehouse adopts a layered architecture, which is divided into: original data layer, detailed data layer, common dimension layer, summary data layer and data application layer, corresponding to ODS, DWD, DIM, DWS and ADS respectively.

[0213] ODS: The data structure is consistent with the data source, permanently storing the incremental or full amount of collected data from the source of the aggregation application system and the centralized control center. It is also the data buffer for ETL process control of the data warehouse, completing the data type conversion or incremental merging processing of the original data.

[0214] DWD: The data structure adopts a standardized data model to integrate and clean the original data of the ODS layer to store the most fine-grained and complete detailed data.

[0215] DIM: Stores the full attribute values ​​of public dimensions such as plant basic information and dictionary tables. It can be associated with the fact tables in the ODS, DWD, DWS, and ADS layers. It is equivalent to extracting attributes that frequently appear in the fact table to form a dictionary table and storing it in a single table for management.

[0216] DWS: Performs indicator analysis and calculation based on the detail data table in the DWD layer and the dimension data table in the DIM layer, stores the summary indicator data after analysis and calculation, and provides flexible and scalable atomic basic indicators for the ADS layer.

[0217] ADS: Based on the summary indicator data in the DWS layer, secondary processing is performed to generate shared data in the format required for application system analysis and use. Data storage mainly uses relational OLTP, OLAP and MPP database types. When the concurrency requirements of the application system data sharing requirements are high, Redis cluster or ES cluster is used to improve the concurrency efficiency of data calls.

[0218] The hierarchical architecture of the data warehouse is the basis of data flow in data governance. The data governance links of the main data flow process of this method include original data access, data standardization, data quality monitoring, data indicator calculation and result data distribution.

[0219] S3, collects raw data based on the input parameters of production and marketing integration.

[0220] The types of raw data of the input parameters of production and marketing integration include annual maintenance plan data, unplanned maintenance data, power forecast data, electricity price forecast data, and cost estimation data. The details of each type of data collection are as follows. The raw data comes from the relevant application systems of the plant, centralized control center or headquarters, including streaming data and batch data. For streaming data, due to the high frequency of data generation and the large scale of data generated per unit time, the real-time data integration development function built in step S1 is used to collect real-time data from the data source end using Flink CDC, and Kafka is used to cache unprocessed data streams to prevent the problem of excessive system load caused by data stream blockage. Since the collected raw data streams need to be analyzed offline in the future, the data streams need to be stored in the distributed file system built in step S11. For batch data, the offline data integration development function built in step S1 is used to add database links at the data source end and the target end, select database table objects at both ends, match the library table field mapping, and configure the scheduling strategy. The final data is still stored in the distributed file system built in S11.

[0221] Preferably, the details of collecting each type of data in step S3 are as follows.

[0222] (1) Annual maintenance plan data

[0223] Collection content: Annual maintenance plan data of each plant and station after it is put into operation. The data items mainly include maintenance plan year, regional company, administrative power plant, standard power station, power generation type, installed capacity, maintenance serial number, maintenance project name, planned start time, planned completion time, construction period, planned cost, and whether to carry out.

[0224] Collection method: Batch data offline incremental collection is adopted. The scheduling period of collection tasks is once a day. Insertion or update operation is performed based on whether the primary key data exists. The data source is the centralized control center.

[0225] (2) Unplanned fault repair data

[0226] Collection content: Unplanned maintenance data of each power plant after commissioning. The data items mainly include regional companies, administrative power plants, standard power plants, power generation types, installed capacity, fault types, wind turbine numbers, fault detection occurrence time, maximum maintenance start interval, and average fault repair time.

[0227] Collection method: Batch data offline incremental collection is adopted. The collection task is scheduled once a day. Insertion or update operations are performed based on whether the primary key data exists. The data source is the intelligent operation and maintenance system.

[0228] (3) Power prediction data

[0229] Collection content: Power forecast data of each plant after commissioning. The data items mainly include regional companies, administrative power plants, standard power stations, power generation types, installed capacity, wind turbine number, forecast time, and forecast power.

[0230] Collection method: adopt streaming data real-time incremental collection method, the collection frequency of the collection task is real-time collection, and the insertion or update operation is performed based on whether the primary key data exists. The data source is the centralized control center.

[0231] (4) Electricity price forecast data

[0232] Collection content: Real-time and day-ahead electricity price forecast data after each plant is put into operation. The data items mainly include forecast type, regional company, administrative power plant, standard power station, forecast time, and forecast electricity price.

[0233] Collection method: adopt streaming data real-time incremental collection method, the collection frequency of the collection task is real-time collection, and the insertion or update operation is performed based on whether the primary key data exists. The data source is the power marketing system.

[0234] (5) Cost estimation data

[0235] Collection content: planned and unplanned maintenance cost data of each plant after commissioning. The data items mainly include estimated year, regional company, administrative power plant, standard power station, wind turbine number, material cost, labor cost, and safety cost.

[0236] Collection method: Batch data offline incremental collection is adopted. The scheduling period of collection tasks is once a day. Insertion or update operation is performed based on whether the primary key data exists. The data source is the centralized control center.

[0237] S4, build a data standardization model for the collected raw data.

[0238] In order to ensure the convenience and standardization of the use of data for the input parameters of the production and marketing integration model, it is necessary to standardize the five types of original data connected to the ODS layer of the data warehouse, that is, to build data standards for annual maintenance plan data, unplanned fault maintenance data, power forecast data, electricity price forecast data, and cost estimate data. By building data standards, the business meaning, field structure and value range of each data item are uniformly defined and constrained. The five types of data element formats and related dictionary table definitions of annual maintenance plan data, unplanned fault maintenance data, power forecast data, electricity price forecast data, and cost estimate data can be found in Table 1, Table 2, Table 3, Table 4, Table 5, and Table 6 respectively.

[0239] Table 1 Annual maintenance plan data

[0240] Field Name (Chinese) Field name (English) Field Type Length / precision Is this field required? Is it a primary key? Remark Maintenance plan year jxj String 4 yes yes The format is YYYY Administrative Power Plant xD String 50 yes yes Standard power station bzd String 50 yes yes Maintenance serial number jxxh String 50 yes yes Maintenance project name jxxmmc String 50 yes yes Planned start time jhkgsj date / yes no The format is YYYY-MM-DD Planned completion time j date / yes no The format is YYYY-MM-DD Duration gq Integer 20 yes no Unit: Day Program Fees Jhf Floating point 20,2 yes no Whether to carry out sf String 1 yes no

[0241] Table 2 Unplanned fault maintenance data

[0242]

[0243]

[0244] Table 3 Power prediction data

[0245]

[0246] Table 4 Electricity price forecast data

[0247]

[0248] Table 5 Cost estimation data

[0249] Field Name (Chinese) Field name (English) Field Type Length / precision Is this field required? Is it a primary key? Remark Estimated Year ygnd String 4 yes yes The format is YYYY Administrative Power Plant xD String 50 yes yes Standard power station bzd String 50 yes yes Fan No. f String 50 yes yes Material Cost clb Floating point 20,2 yes no Labor costs rgb Floating point 20,2 yes no Security costs aq Floating point 20,2 yes no

[0250] Table 6 Basic information data of plant

[0251] Field Name (Chinese) Field name (English) Field Type Length / precision Is this field required? Is it a primary key? Remark Regional Companies qygs String 50 yes no Reference Area Company Dictionary Table Administrative Power Plant xD String 50 yes yes Standard power station bzd String 50 yes yes Power generation type fdx String 2 yes no Reference generation type dictionary table Installed capacity zr Floating point 20,4 yes no Unit: 10,000 kW

[0252] The above defines the business fields in the data element format. In actual data modeling, the following system fields should be added to each physical data table.

[0253] cd_id: The unique identification field for each form entry. The value is a random sequence generated by the uuid function.

[0254] cd_time: Incremental timestamp field used for data collection. The value is the current time when the data arrives at the front-end switching node at the data source end. When data is updated at the data source end, this field is dynamically updated to the current time.

[0255] cd_source: data source identification field, the value is the centralized control center or application system, such as Hunan centralized control center and intelligent operation and maintenance system.

[0256] cd_batch: data aggregation batch number field, mainly used for data reconciliation with the data source after receiving the data.

[0257] cd_operation: data operation identification field, the values ​​are i, u, d, which represent add, update, and delete operations respectively.

[0258] Data aggregation is carried out in the form of "full data for the first time, incremental update". Each table uses cd_id and cd_source as the system primary key of data records, cd_time as the incremental timestamp field for data pre-exchange, and each table adds the updatetime field to record data update time. If you need to update or delete data, you only need to modify the field corresponding to the business primary key to be updated and update cd_id, cd_time and cd_operation at the same time. At the same time, update_time is synchronously updated to the current time after receiving the updated data.

[0259] The data physical table model is constructed based on the aggregation library and basic library defined in the DWD layer of the data warehouse based on the data element format. The DDL table creation statement template is as follows.

[0260] CREATE TABLE `{English table name}`(

[0261] `cd_id`varchar(100)NOT NULL COMMENT'record id',

[0262] {Business Field 1}

[0263] {Business Field 2} ...

[0265] {Business field n}

[0266] `cd_source`varchar(6)NOT NULL COMMENT'Source',

[0267] `cd_time` datetime(0) NOT NULL COMMENT 'insertion time',

[0268] `cd_batch`varchar(50)NOT NULL COMMENT'batch number',

[0269] `cd_operation`varchar(1)NOT NULL COMMENT'Data operation type',

[0270] `update_time` datetime(0) NOT NULL COMMENT 'update time',

[0271] PRIMARY KEY(`cd_id`,`cd_source`)USING BTREE

[0272] )ENGINE=InnoDB CHARACTER SET=utf8 COMMENT='{Chinese table name}' ROW_FORMAT=Dynamic.

[0273] The data dictionary can be found in Table 7, Table 8, and Table 9.

[0274] Table 7 Electricity price forecast type dictionary

[0275]

[0276] Table 8 Dictionary table of power generation types

[0277]

[0278] Table 9 Fault type dictionary

[0279]

[0280]

[0281] S5, build a data quality verification model based on the data standardization model.

[0282] The data quality verification model is a set of data quality verification rules based on the data standardization model and the business rules of production and marketing integration, and a data quality verification-rectification closed-loop process is designed to monitor the timeliness, accuracy, completeness, consistency and uniqueness of the data after DWD layer integration and cleaning in real time, so as to ensure that the calculation results of the production and marketing integration model are more accurate. The data quality verification rules for annual maintenance plan data, unplanned fault maintenance data, power forecast data, electricity price forecast data, and cost estimation data can be found in Tables 10, 11, 12, 13, 14, and 15.

[0283] Table 10 Rules for annual maintenance plan data quality verification

[0284]

[0285] Table 11 Rules for quality inspection of unplanned fault maintenance data

[0286]

[0287] Table 12 Power prediction data quality verification rules

[0288]

[0289] Table 13 Rules for checking the quality of electricity price forecast data

[0290]

[0291]

[0292] Table 14 Cost estimate data quality verification rules

[0293]

[0294] Table 15 Quality verification rules for basic information data of power plants and stations

[0295]

[0296] See also Fig.11 As shown, in the data quality verification-rectification closed-loop process, the DWD layer aggregation library stores the pre-processed raw data, and performs quality verification on each field of each table in the aggregation library in turn in terms of data format and data content according to the data quality verification rules and records the verification results. If a field in a single data record does not meet the verification rules, it is determined to be erroneous data and the system primary key, data source, batch number and error cause of the erroneous data record are pushed to the error data record table in the error library and further sent to the corresponding control center and application system according to the data source field in the error data record table. Otherwise, it is determined to be correct data and pushed to the data physical table in the DWD layer basic library with the same table structure as the aggregation library. The above-mentioned error data record table structure can be seen in Table 16.

[0297] Table 16 Error data record table structure

[0298] Field Name (Chinese) Field name (English) Field Type Length / precision Is this field required? Is it a primary key? Remark Table name table_name String 50 yes yes Reference Area Company Dictionary Table Record unique identifier cd_id String 50 yes yes System primary key 1 Data Source cd_source String 50 yes yes System master key 2 Batch number cd_batch String 50 yes no Example: 2024092400001 Error details error_detail String 50 yes no {field1:{error1;}}

[0299] S6, calculate indicators based on the data requirements of production and marketing integration.

[0300] Preferably, step S6 uses the data ETL development tool constructed in step S16 to perform data cleaning, preprocessing and indicator calculation on the original data according to the data warehouse hierarchical order in step S2. The final indicator calculation result should integrate the data that are related but not in the same table as much as possible, and use the primary key field association method to ensure the uniqueness of the association result.

[0301] Preferably, the data indicator calculation is to implement the fusion, statistics and other analytical calculation processes on the detailed data in the DWD layer basic library according to the data requirements of the application side. The application side object of this method is the input parameters of the production and marketing integration model. By comparing the input parameters with the atomic indicators in the fields in the DWD layer basic library, it is found that the data indicator calculation in this method does not require data analysis operations with relatively complex calculation logic such as aggregation operations, field merging operations, and field splitting operations. Instead, it only needs to extract the atomic indicator results from some fields in the DWD layer basic database table and perform conventional association analysis calculations and data synchronization to meet the data requirements of the algorithm model input parameters. Therefore, the DWS layer thematic library stores the indicator wide table data and synchronized data calculated from the DWD layer basic library table and the DIM layer dimension table, specifically including the power-electricity price forecast wide table formed by associating power forecast and electricity price forecast data and synchronizing annual maintenance plan data, unplanned fault maintenance data and cost estimation data. The processing logic of the above two calculation methods of association analysis calculation and data synchronization is as follows.

[0302] (1) Correlation calculation of power forecast and electricity price forecast data wide table

[0303] select

[0304] A.xzdc as administrative power plant,

[0305] A.bzdz as standard power station,

[0306] A.fjbhas fan number,

[0307] B.yclxas electricity price forecast type,

[0308] A.ycsj as prediction time,

[0309] A.ycglas prediction power,

[0310] B.ycdj as predicted electricity price

[0311] from{DWD layer basic library. Power prediction data table A}

[0312] inner join {DWD layer basic library. Electricity price forecast data table B}

[0313] on A.xzdc=B.xzdc and A.bzdz=B.bzdz and A.ycsj=B.ycsj;

[0314] (2) Synchronization of annual maintenance plan, unplanned fault maintenance and cost estimate data

[0315] insert into {DWS layer annual maintenance plan data table}

[0316] select distinct

[0317] jxjhnd,xzdc,bzdz,jxxh,jxxmmc,jhkgsj,jhwgsj,gq,jhfy,sfkz

[0318] from{DWD layer basic library.Annual maintenance plan data table};

[0319] insert into {DWS layer unplanned fault repair data table}

[0320] select distinct

[0321] xzdc,bzdz,gzlx,fjbh,gzjcfssj,zdjxksjg,gzpjxfsj

[0322] from{DWD layer basic library.unplanned fault repair data table};

[0323] insert into {DWS layer cost estimation data table}

[0324] select distinct

[0325] ygnd,xzdc,bzdz,fjbh,clcb

[0326] from{DWD layer base library.cost estimation data table}.

[0327] S7, pushes the calculation results to the existing production and marketing integrated application module.

[0328] Preferably, step S7 is to share the indicator result data calculated at the DWS layer in step S6 as input parameters to the production and marketing integration model for model calculation.

[0329] Preferably, after the DWS layer generates the indicator result data, it is necessary to provide data services to the production and marketing integration model through data sharing methods such as database tables and API interfaces at the ADS layer. Considering the convenience and security of using data for the input parameters of the production and marketing integration model, the API interface is used to pass parameters to the algorithm model, and the Redis cache database is used as the API interface persistence layer object to improve the efficiency of API interface access.

[0330] Four interfaces are constructed at the ADS layer, namely, annual maintenance plan query interface, unplanned fault maintenance query interface, power-electricity price forecast query interface and cost estimation query interface.

[0331] The request method of the annual maintenance plan query interface is post. The input parameters are maintenance plan year, administrative power plant, and standard power plant. The output parameters are maintenance serial number, maintenance item, planned start time, planned completion time, construction period, planned cost, and whether to carry out. The URL request format is https: / / {ip:port} / ndjxjhInfo.

[0332] The request method of the unplanned fault maintenance query interface is post. The input parameters are administrative power plants and standard power plants. The output parameters are wind turbine number, fault detection occurrence time, maximum maintenance start interval, and mean fault repair time. The URL request format is https: / / {ip:port} / fjhgzjxInfo.

[0333] The request method of the power-electricity price forecast query interface is post. The input parameters are administrative power plants and standard power plants. The output parameters are wind turbine number, forecast time, forecast power, electricity price forecast type, and forecast electricity price. The URL request format is https: / / {ip:port} / gldjycInfo.

[0334] The request method of the cost estimate query interface is post. The input parameters are estimated year, administrative power plant, and standard power plant. The output parameters are wind turbine number, material cost, labor cost, and safety cost. The URL request format is https: / / {ip:port} / cbygInfo.

[0335] S8, distributes the prediction result data of the production and marketing integration application module.

[0336] Preferably, step S8 shares the calculation result data of the production and marketing integration model with the intelligent production control system for guiding the dispatch of maintenance work orders.

[0337] Preferably, the output parameters of the production and marketing integration model are the maintenance start time and the total maintenance cost. The calculation results will eventually be provided to the intelligent production control system to guide the dispatch of maintenance work orders. Therefore, the calculation result data must flow back to the data warehouse and the returned data must be pushed to the front exchange database of the intelligent production control system in a timely manner. This process includes two data synchronization calculation processes, and its calculation logic is consistent with the principle of the data screening synchronization calculation logic in step S6. The specific calculation logic is as follows.

[0338] create table{ADS layer production and marketing integration model calculation result data table}{

[0339] `fjbh`varchar(50)NOT NULL COMMENT'Fan number'

[0340] `jxkssj`datetime NOT NULL COMMENT'Maintenance start time',

[0341] `jxzcb`float(20,2)NOT NULL COMMENT'Total cost of maintenance',

[0342] };

[0343] insert into{ADS layer production and marketing integration model calculation result data table}

[0344] select distinct

[0345] fjbh,jxkssj,jxzcb

[0346] from{output parameter result set of production and marketing integration model};

[0347] insert into{intelligent production control system front-end exchange library. production and marketing integration model calculation result data table}

[0348] selectdistinct

[0349] fjbh,jxkssj,jxzcb

[0350] from{ADS layer production and marketing integration model calculation results data table}.

[0351] Through the above scenario examples, based on the multi-business data integration and processing method based on new energy provided in this specification, first, the data management technology based on the enterprise-level big data governance methodology is used to solve the problem of insufficient management of production and marketing data in standardization and quality control; and, a production and marketing data standardization model for production and marketing integrated business is constructed; furthermore, a data quality verification model based on the production and marketing data standardization model is also constructed. Compared with the complexity of the many-to-many data sharing and exchange network structure between each system in the prior art, the present invention optimizes the data sharing and exchange mode, that is, a one-to-many method is used for data sharing, which effectively improves the efficiency of data circulation and sharing. In addition, compared with the problem of insufficient governance of production and marketing data in the prior art, the present invention improves the quality of data required for the production and marketing integrated analysis business scenario by constructing a production and marketing data standardization model and a data quality verification model, thereby ensuring the accuracy of the production and marketing integrated prediction and analysis results.

[0352] Although the present specification provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.

[0353] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0354] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer-readable storage media including storage devices.

[0355] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that the present specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can essentially be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0356] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0357] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A multi-business data integration processing method based on new energy, characterized in that: A node server applied to a multi-business data integration processing system, wherein the multi-business data integration processing system further comprises: a big data storage and computing engine module, a data development tool, and a data management tool. The multi-business data integration processing system is connected to a new energy plant terminal through a first connection and to a power grid market terminal through a second connection. The method comprises: Receive and respond to target joint business data processing requests, and call data development tools to generate corresponding target business processing flows; According to the target business processing flow, the data management tool is called to obtain the power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and the power grid market data related to the power grid market through the second connection; and the power generation capacity data, plant equipment operation data and power grid market data are combined to obtain the target joint business data; According to the target business processing flow, the big data storage and computing engine module is called to create a corresponding target data warehouse; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and the target joint business data is stored in the original data layer; According to the target business processing flow, call the data management tool to read the target joint business data through the original data layer; pre-process the target joint business data to obtain the pre-processed target joint business data; and store the pre-processed target joint business data in the detailed data layer; According to the target business processing flow, the big data storage and computing engine module is called to read the pre-processed target joint business data through the detailed data layer, combined with the public attribute data in the public dimension layer, and through joint calculation, the business processing strategy for the target time period is predicted as the target processing result; and the target processing result is stored in the summary data layer; Based on the target processing result, determining the associated target business implementation end; storing the target processing result in the application data layer; and establishing a third connection between the target business implementation end and the application data layer; Sending target prompt information about the target processing result to the target business implementation end; wherein the target prompt information carries at least an interface identifier of the data interface associated with the third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the relevant business of executing the target time period according to the target processing result.

2. The method according to claim 1, characterized in that After calling the data management tool to obtain power generation capacity data and plant equipment operation data related to the new energy plant through the first connection, and obtaining power grid market data related to the power grid market through the second connection, the method further includes: Calling the data management tool to detect whether there is sensitive data in the power generation capacity data, plant equipment operation data, and power grid market data according to the preset sensitive information data table; and if it is determined that sensitive data exists, encrypting the sensitive data according to the preset homomorphic encryption rules to obtain the ciphertext data of the sensitive data; Call the data management tool to replace the sensitive data in power generation capacity data, plant equipment operation data, and power grid market data with the ciphertext data of the sensitive data; and destroy the sensitive data.

3. The method according to claim 1, characterized in that Call the data management tool to obtain power generation capacity data related to new energy plants through the first connection, including: Detect and identify the data type of power generation capacity data; For streaming data in power generation capacity data, call the real-time data integration development function component in the data development tool, and build the corresponding first connection in combination with Flink CDC and Kafka; call the data management tool to obtain and cache the streaming data in power generation capacity data through the first connection; For batch data in power generation capacity data, call the offline data integration development function component in the data development tool to build a corresponding first connection; call the data management tool to obtain the batch data in the power generation capacity data through the first connection.

4. The method according to claim 1, characterized in that: Preprocess the target joint business data, including: Call the data management tool to standardize the field type, field name, field length, data type, and data format in the target joint business data according to the preset standardization rules, and obtain the standardized target joint business data; According to the target business processing flow, call the data management tool to build data tables and data models related to the target joint business data processing.

5. The method according to claim 4, characterized in that The data models include: power generation capacity prediction model, distribution network power prediction model, and production and marketing decision model.

6. The method according to claim 5, characterized in that According to the target business processing flow, the big data storage and computing engine module is called to read the target joint business data through the detailed data layer and combine it with the public attribute data in the public dimension layer through joint operation to predict the business processing strategy for the target time period as the target processing result, including: According to the target business processing flow, the big data storage and computing engine module is called to read the target joint business data through the detailed data layer and run the power generation capacity prediction model and the distribution network power prediction model respectively based on the pre-processed target joint business data and the public attribute data in the public dimension layer to obtain the power generation capacity prediction results and the distribution network demand power prediction results for the target time period; According to the target business processing flow, the big data storage and computing engine module is called to combine the power generation capacity forecast result of the target time period and the distribution network demand power forecast result of the target time period to obtain the target integrated data group for the target time period; According to the target business processing flow, the big data storage and computing engine module is called to utilize the target integrated data group, combined with the public attribute data in the public dimension layer, to run the production and marketing decision model, and predict the production and marketing joint strategy for the target time period as the target processing result.

7. The method according to claim 1, characterized in that The method further comprises: Calling the data management tool to determine the preset data quality detection rules corresponding to the relevant data involved in the target business processing flow from the preset data quality detection rule set, and combining them to obtain the target data quality detection rules; Call the data management tool to perform quality inspection on the relevant data appearing in the multi-business data integration processing system according to the target data quality inspection rules; According to the quality inspection results, the relevant data that failed the inspection will be marked as abnormal data; and the abnormal data will be traced and investigated according to the target quality rules and target business processing flow to determine the cause and / or location of the abnormality.

8. A multi-business data integration processing device based on new energy, characterized in that: A node server applied to a multi-business data integration processing system, wherein the multi-business data integration processing system further comprises: a big data storage and computing engine module, a data development tool, and a data management tool. The multi-business data integration processing system is connected to a new energy plant terminal through a first connection and to a power grid market terminal through a second connection. The device comprises: The first calling module is used to receive and respond to the target joint business data processing request, and call the data development tool to generate the corresponding target business processing flow; The second calling module is used to call the data management tool to obtain the power generation capacity data and plant equipment operation data related to the new energy plant through the first connection according to the target business processing flow, and to obtain the power grid market data related to the power grid market through the second connection; and combine the power generation capacity data, plant equipment operation data and power grid market data to obtain the target joint business data; The third calling module is used to call the big data storage and computing engine module to create a corresponding target data warehouse according to the target business processing flow; wherein the target data warehouse at least includes: an original data layer, a detailed data layer, a common dimension layer, a summary data layer and an application data layer; and store the target joint business data in the original data layer; The fourth calling module is used to call the data management tool to read the target joint business data through the original data layer according to the target business processing flow; pre-process the target joint business data to obtain the pre-processed target joint business data; and store the pre-processed target joint business data into the detailed data layer; The fifth calling module is used to call the big data storage and calculation engine module to read the pre-processed target joint business data through the detailed data layer according to the target business processing flow, and combine the public attribute data in the public dimension layer to predict the business processing strategy for the target time period as the target processing result through joint calculation; and store the target processing result in the summary data layer; A determination module, for determining an associated target business implementation end based on the target processing result; storing the target processing result in the application data layer; and establishing a third connection between the target business implementation end and the application data layer; A sending module is used to send target prompt information about the target processing result to the target business implementation end; wherein the target prompt information carries at least an interface identifier of a data interface associated with a third connection; the target business implementation end obtains the target processing result through the third connection according to the interface identifier; the target business implementation end also adjusts the relevant business of executing the target time period according to the target processing result.

9. A server, characterized in that: The method comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the instructions.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images