Multi-heterogeneous data management system based on software robot technology

Abstract

The application discloses a multi-element heterogeneous data management system based on software robot technology and relates to the technical field of multi-element heterogeneous data processing. In the multi-element heterogeneous data management process, extraction operation is performed on multi-element heterogeneous data obtained from different thermal power operation management systems, and the extracted multi-element heterogeneous data is summarized and represented as an original multi-element heterogeneous data set. Automatic cleaning and verification operation is performed on the original multi-element heterogeneous data set obtained from a software robot data extraction module. The standardized multi-element heterogeneous data obtained from a multi-element heterogeneous data processing module is written into an API service interface, which is used for unified calling, querying and application of the multi-element heterogeneous data, and solves the problem that the efficiency, accuracy and collaborative ability of the thermal power operation management system operation cannot be improved in the prior art.

Claims

1. A multi-dimensional heterogeneous data management system based on software robot technology, characterized in that: The system includes: a software robot data extraction module, a multi-dimensional heterogeneous data processing module, and a multi-dimensional heterogeneous data management module; The software robot data extraction module is used to perform extraction operations on the multi-dimensional heterogeneous data obtained from different thermal power operation and management systems during the multi-dimensional heterogeneous data management process, and to summarize the extracted multi-dimensional heterogeneous data into an original multi-dimensional heterogeneous dataset. The multi-variable heterogeneous data processing module is used to process the original multi-variable heterogeneous dataset obtained from the software robot data extraction module to form standardized multi-variable heterogeneous data. The multi-variable heterogeneous data management module is used to write the standardized multi-variable heterogeneous data obtained by the multi-variable heterogeneous data processing module into the API service interface, and then determine whether the standardized multi-variable heterogeneous data management is qualified based on the written standardized multi-variable heterogeneous data.

2. The multi-source heterogeneous data management system based on software robot technology as described in claim 1, characterized in that, The specific process for extracting the diverse and heterogeneous data obtained from different thermal power plant operation and management systems is as follows: The thermal power plant operation response delay is defined as the time interval between the thermal power plant operation management system receiving the operation command and the thermal power plant operation management system responding to the operation command. The response time of thermal power plant operation is compared with the preset response time of thermal power plant operation; If the thermal power plant operation response delay is less than or equal to the preset thermal power plant operation response delay, the status of the software robot can be viewed through the management page based on the software robot. Conversely, it indicates that the thermal power operation management system is malfunctioning, and the coverage of multi-dimensional heterogeneous data extraction is obtained. The multi-variable heterogeneous data extraction coverage is represented by the ratio of the number of successfully extracted multi-variable heterogeneous data types to the total number of heterogeneous data types to be extracted in the thermal power operation and management system. Determine whether the multivariate heterogeneous data extraction coverage is greater than or equal to the preset multivariate heterogeneous data extraction coverage; If so, the corresponding heterogeneous data will be aggregated into the original heterogeneous dataset, and a heterogeneous data processing qualification assessment will be performed to achieve the standardization transformation of heterogeneous data. Conversely, multi-dimensional heterogeneous data extraction is enhanced.

3. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 2, characterized in that, The specific process of extracting and enhancing multi-source heterogeneous data is as follows: Based on the software robot, the page node, interface address and data storage location of the heterogeneous data types that were not successfully extracted are relocated; For multi-variable heterogeneous data types that were not successfully extracted, adjust the extraction frequency of multi-variable heterogeneous data. The frequency of extracting diverse and heterogeneous data specifically refers to the total number of times that diverse and heterogeneous data in the thermal power operation and management system is extracted within a preset extraction time. The specific adjustment process is as follows: If the number of extraction failures corresponding to the unsuccessfully extracted heterogeneous data is greater than or equal to the preset failure threshold, the frequency of heterogeneous data extraction will be gradually increased based on the proportion of heterogeneous data extraction, and the coverage of heterogeneous data extraction will be obtained in real time. The multivariate heterogeneous data extraction ratio is expressed as the ratio of the absolute value of the difference between the multivariate heterogeneous data extraction coverage rate and the preset multivariate heterogeneous data extraction coverage rate to the preset multivariate heterogeneous data extraction coverage rate. If the number of extraction failures corresponding to the unsuccessfully extracted multivariate heterogeneous data is less than the preset failure threshold, the current multivariate heterogeneous data extraction frequency will be maintained, and the multivariate heterogeneous data extraction coverage will be obtained in real time. If, within the preset extraction enhancement period, the extraction coverage rate of the supplementary extraction of heterogeneous multivariate data is greater than or equal to the extraction coverage rate of heterogeneous multivariate data, then the corresponding heterogeneous multivariate data will be aggregated into the original heterogeneous multivariate dataset, and a qualification assessment of heterogeneous multivariate data processing will be performed. Otherwise, a heterogeneous multivariate data extraction anomaly prompt will be sent to the preset personnel.

4. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 3, characterized in that, The specific process for the qualification assessment of the multi-source heterogeneous data processing is as follows: The process of obtaining the number of valid data entries corresponding to the diverse and heterogeneous data after complex processing, cleaning, and manipulation specifically refers to: The original heterogeneous data collected from various thermal power operation management systems are sequentially processed by format verification, missing value filtering, duplicate value removal, outlier identification and field normalization. Invalid records that do not conform to the preset data rules are removed, and heterogeneous data with consistent format and complete content are retained. The number of valid data records after processing is then counted. The ratio of the number of valid data entries corresponding to the acquired heterogeneous multivariate data to the total number of original heterogeneous multivariate data entries is defined as the heterogeneous multivariate data integrity rate. Compare the completeness rate of multivariate heterogeneous data with the preset completeness rate of multivariate heterogeneous data; If the completeness rate of heterogeneous multivariate data is greater than or equal to the preset completeness rate of heterogeneous multivariate data, the acquired heterogeneous multivariate data will be represented as standardized heterogeneous multivariate data, and the heterogeneous multivariate data writing will be performed. Conversely, optimization of multi-dimensional heterogeneous data processing should be carried out.

5. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 4, characterized in that, The specific process for optimizing the processing of multi-variable heterogeneous data is as follows: Based on field similarity, redundant multi-dimensional heterogeneous data is identified. Incremental data and full data are acquired by combining preset scheduling period and preset number of fields. The acquired multi-dimensional heterogeneous data is then subjected to multi-dimensional heterogeneous data denoising processing. The specific process of acquiring incremental and full data by combining scheduling parameters refers to: A preset number of scheduling cycles are pre-set by designated personnel. Within each scheduling cycle, the standardized multivariate heterogeneous data of adjacent scheduling cycles are used as a comparison benchmark to determine whether the number of fields that completely match within adjacent scheduling cycles is greater than or equal to the preset number of fields. If it is greater than or equal to the preset number of fields, incremental data acquisition is triggered; otherwise, full data acquisition is triggered. The specific process of denoising the multivariate heterogeneous data is as follows: The system identifies and classifies multi-variable heterogeneous data based on deep learning algorithms and deep neural network models. It trains and optimizes the deep neural network model based on historical running data, obtains cleaned multi-variable heterogeneous data, represents the cleaned multi-variable heterogeneous data as standardized multi-variable heterogeneous data, and performs multi-variable heterogeneous data writing.

6. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 5, characterized in that, The specific process of writing the multi-source heterogeneous data is as follows: Standardized heterogeneous data is written through an API service interface, and the success rate of writing heterogeneous data is obtained. The number of standardized data entries successfully written to the API service interface is represented by the number of standardized data entries when writing to the API service interface during the writing operation of standardized heterogeneous data. The ratio of the number of standardized data entries successfully written to the API service interface to the total number of standardized data entries to be written is expressed as the success rate of writing multi-variable heterogeneous data. Determine whether the success rate of writing multi-dimensional heterogeneous data is greater than or equal to the preset success rate of writing multi-dimensional heterogeneous data. If so, it indicates that the writing of multi-dimensional heterogeneous data is qualified, and a qualification measurement of multi-dimensional heterogeneous data management should be carried out to assess the overall situation of multi-dimensional heterogeneous data management. Conversely, if the write of heterogeneous data is abnormal, optimization of the write of heterogeneous data should be carried out.

7. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 6, characterized in that, The specific process of optimizing the writing of multi-variable heterogeneous data is as follows: After identifying the anomaly type in the standardized multivariate heterogeneous data that was written abnormally, the standardized multivariate heterogeneous data was rewritten. The process of splitting standardized heterogeneous data into a preset number of heterogeneous data segments and writing them sequentially refers to: Based on a pre-set threshold for the number of heterogeneous data segments in a single batch, the complete standardized heterogeneous data to be written is divided into heterogeneous data segments according to the threshold. The heterogeneous data segments are written to the API service interface one by one in a preset order. After each heterogeneous data segment is written, the writing process of the next data segment is triggered until all the split data segments are written. Re-acquiring the success rate of writing multi-dimensional heterogeneous data; If the success rate of writing re-acquired heterogeneous data is greater than or equal to the preset success rate of writing heterogeneous data, then a qualification assessment of heterogeneous data management will be conducted. Conversely, the API service interface call frequency is adjusted based on the original API service interface call frequency. If the success rate of writing reacquired heterogeneous data is less than the first success rate threshold, the API service interface call frequency will be adjusted to the first API service interface call frequency set in advance by the preset personnel. When the success rate of re-acquired heterogeneous data writing is greater than or equal to the first success rate threshold and less than the preset heterogeneous data writing success rate, the API service interface call frequency will be adjusted to the second API service interface call frequency set in advance by the preset personnel. The system then checks whether the success rate of writing the newly acquired heterogeneous data is greater than or equal to the preset success rate of writing heterogeneous data. If the success rate of writing multi-dimensional heterogeneous data is greater than or equal to the preset success rate, then a qualification assessment of multi-dimensional heterogeneous data management will be conducted. If the success rate is still less than the preset success rate for writing heterogeneous data, an abnormal alarm will be triggered to notify the designated personnel.

8. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 7, characterized in that, The specific process for assessing the compliance of multi-source heterogeneous data management is as follows: Based on the standardized multi-dimensional heterogeneous data written, the coverage of multi-dimensional heterogeneous data management and the success rate of API interface calls are obtained; Get the number of standardized heterogeneous data types written and define it as the number of heterogeneous data types; The ratio of the number of heterogeneous data types received by the heterogeneous data management system to the preset number of heterogeneous data types is defined as the heterogeneous data management coverage rate. While comparing the coverage of multi-variable heterogeneous data management with the preset management coverage, the API interface call success rate is also compared with the preset interface call success rate. If the coverage rate of multi-dimensional heterogeneous data management and the success rate of API interface calls are both greater than or equal to the corresponding preset parameters, the multi-dimensional heterogeneous data management is deemed qualified, and a prompt is sent to the multi-dimensional heterogeneous data software robot to generate reports based on the multi-dimensional heterogeneous data. Conversely, if the data is not properly managed, the software robot will be notified that the task is not compliant. The software robot will then attempt to retry the task according to the number of attempts set by the user, until the task succeeds or the maximum number of failures is reached. If the task fails completely, an automatic alarm mechanism will be triggered.

9. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 1, characterized in that, The specific process for obtaining diverse and heterogeneous data from different thermal power plant operation and management systems is as follows: The ratio of the memory of the thermal power operation and management system during the preset monitoring period to the preset operating memory of the thermal power operation and management system set in advance by the preset personnel is expressed as the CPU utilization rate of the thermal power operation and management system. Determine whether the CPU utilization rate of the thermal power operation management system is less than or equal to the preset CPU utilization rate of the thermal power operation management system. If the CPU utilization is less than or equal to the preset upper limit threshold, the status of the software robot can be viewed through the management page based on the software robot. Conversely, the thermal power plant operation management system will trigger task downgrade processing.

10. The multi-dimensional heterogeneous data management system based on software robot technology as described in claim 9, characterized in that, The specific process of task downgrading is as follows: Based on the pre-set task priority tags in the thermal power operation and management system, the execution of low-priority tasks in the pre-set thermal power operation and management system is paused to reduce the memory usage of the thermal power operation and management system. Switch synchronous tasks to asynchronous tasks; If the CPU utilization rate of the thermal power operation management system is less than or equal to the preset upper limit threshold, the status of the software robot can be viewed through the management page based on the software robot. Conversely, the response delay of thermal power plant operation is obtained.

Images