Water conservancy model multi-source log processing method and device

Abstract

The invention provides a water conservancy model multi-source log processing method and device. The method comprises the steps of obtaining original log data of each model node task in a water conservancy model scheduling link; performing dimension mapping processing on the original log data based on the log dimension mapping logic to obtain log data of a target dimension; performing normalization processing on the log data of the target dimension to obtain normalized log data, and storing the normalized log data in a distributed search engine cluster; obtaining a query word input by a user side, and searching all normalized log data containing the query word; and obtaining the previous target quantity of log data from the log data sorting result as target log data, and returning the target log data to the user side. According to the method provided by the invention, the multi-source log processing efficiency of the water conservancy model can be effectively improved, and the target log data of the water conservancy model in different model node tasks can be quickly obtained, so that the water conservancy model scheduling link can be conveniently managed and maintained, and the use experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for processing multi-source logs of a water conservancy model, and also to an electronic device, a non-transitory computer-readable storage medium and a computer program product. Background Art

[0002] In recent years, with the rapid development of smart water conservancy construction, the digital twin basin platform that uses basins as units to improve water regime monitoring and intelligent scheduling capabilities has become an important task for the high-quality development of water conservancy in the new stage. The water conservancy models in the digital twin basin platform involve hydrological models that simulate rainfall and runoff, hydrodynamic models that simulate flood evolution, multi-objective joint scheduling models that guide the use of water conservancy projects, and disaster assessment models that analyze flooding losses. The connection and call between the above water conservancy models is the core of the intelligent simulation of the digital twin basin, but the call links between water conservancy models are long and the calculations are complex. The algorithm logic, call steps, parameter verification, and execution process of the water conservancy model are generally encapsulated in the program by the platform developers, which is a black box state for business operators. When the operation task fails, the operator cannot perceive which link has a problem with the task, and the developer needs to query the relevant logs in the background to check them one by one. The path to solve the problem is long and inefficient, which affects the development of flood control command and dispatch work in the basin. In addition, the "black box state" of the water conservancy model may cause users to face trust issues when relying on the prediction results of the water conservancy model to make decisions. Due to the lack of a clear understanding of the mechanism of the hydraulic model, it is difficult for users to fully trust the output of the hydraulic model, especially when the predicted results of the hydraulic model deviate from the actual situation, it is difficult to judge whether it is the limitation of the hydraulic model itself or the improper parameter setting. Therefore, the verification, validation and uncertainty analysis of the hydraulic model are particularly important, but these processes also require certain professional knowledge and technical means. Due to the large scale of the digital twin basin project, there are usually multiple hydraulic model manufacturers involved in the platform construction process. Different manufacturers and different models are independent and interdependent. Without unified management and effective integration, the call link layout appears messy, which brings great trouble to the subsequent maintenance and upgrade work. Therefore, how to design a more effective multi-source log processing solution for hydraulic models has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a multi-source log processing method for a water conservancy model, which is used to solve the defect in the prior art that the multi-source log processing efficiency of the water conservancy model is poor, resulting in the inability to quickly obtain target log data of the water conservancy model at different model node tasks.

[0004] The present invention provides a method for processing multi-source logs of a water conservancy model, comprising: Obtain the original log data of each model node task in the water conservancy model scheduling link; Performing dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension; Normalizing the log data of the target dimension to obtain normalized log data, and storing the normalized log data in a distributed search engine cluster; Obtain a query word input by a user terminal, and search for the query word in the term table corresponding to the distributed search engine cluster to obtain an inverted list corresponding to the query word; search for all normalized log data containing the query word according to the inverted list; sort all normalized log data containing the query word to obtain a log data sorting result; obtain the first target number of log data from the log data sorting result as target log data, and return it to the user terminal.

[0005] Furthermore, after obtaining the normalized log data, the following steps are also included: The normalized log data is stored in a log aggregation analysis library as log aggregation data; According to the preset aggregation dimension, the log aggregation data in the log aggregation analysis library is distributed to the corresponding aggregation operator corresponding to the aggregation dimension for aggregation processing, and the aggregation processing result of the log data is obtained, and the aggregation processing result of the log data is stored in the structured analysis library, so as to perform structured analysis according to the aggregation processing result of the log data stored in the structured analysis library and output the corresponding structured analysis result.

[0006] Furthermore, the original log data of each model node task in the water conservancy model scheduling link is obtained, including: Log embedding processing is performed on each model node task in the water conservancy model scheduling link to obtain the original log data during the execution of each model node task in the water conservancy model scheduling link through log embedding; wherein the original log data contains the identification of each model node task in different execution states.

[0007] Furthermore, the normalizing the log data of the target dimension to obtain the normalized log data specifically includes: Converting the log data of the target dimension into structured log data; Performing data cleaning on the structured log data to obtain cleaned log data; Perform interpolation processing on the cleaned log data to obtain interpolated log data; The interpolated log data is subjected to a deviation correction process to obtain normalized log data.

[0008] Further, after obtaining the first target number of log data from the log data sorting result as the target log data, the method further includes: In the case where the target log data has an abnormality, determining the log abnormality level corresponding to the target log data; Different log alarm objects and alarm notification methods are determined according to the log abnormality level.

[0009] Furthermore, the water conservancy model multi-source log processing method further includes: In the case where the target log data has an abnormality, determining a model node identifier of the target log data record; The corresponding model node task is determined according to the model node identifier; wherein the model node task includes at least one of a flood operation configuration task, a runoff calculation task, a reservoir rule scheduling calculation task, a one-dimensional hydrodynamic calculation task, an optimized scheduling operation task, an optimized scheduling calculation task and a two-dimensional hydrodynamic calculation task.

[0010] The present invention also provides a hydraulic model multi-source log processing device, comprising: A log data acquisition unit, used to acquire the original log data of each model node task in the water conservancy model scheduling link; A dimension mapping processing unit, configured to perform dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension; A normalization processing unit, used to perform normalization processing on the log data of the target dimension to obtain normalized log data, and store the normalized log data in a distributed search engine cluster; The target log data processing unit is used to obtain the query word input by the user terminal, and search for the query word in the term table corresponding to the distributed search engine cluster to obtain the inverted list corresponding to the query word; find all normalized log data containing the query word according to the inverted list; sort all the normalized log data containing the query word to obtain the log data sorting result; obtain the first target number of log data from the log data sorting result as the target log data, and return it to the user terminal.

[0011] Furthermore, after obtaining the normalized log data, the following steps are also included: Structured analysis unit, specifically used for: The normalized log data is stored in a log aggregation analysis library as log aggregation data; According to the preset aggregation dimension, the log aggregation data in the log aggregation analysis library is distributed to the corresponding aggregation operator corresponding to the aggregation dimension for aggregation processing, and the aggregation processing result of the log data is obtained, and the aggregation processing result of the log data is stored in the structured analysis library, so as to perform structured analysis according to the aggregation processing result of the log data stored in the structured analysis library and output the corresponding structured analysis result.

[0012] Furthermore, the log data acquisition unit is specifically used to: Log embedding processing is performed on each model node task in the water conservancy model scheduling link to obtain the original log data during the execution of each model node task in the water conservancy model scheduling link through log embedding; wherein the original log data contains the identification of each model node task in different execution states.

[0013] Furthermore, the normalization processing unit is specifically used to: Converting the log data of the target dimension into structured log data; Performing data cleaning on the structured log data to obtain cleaned log data; Perform interpolation processing on the cleaned log data to obtain interpolated log data; The interpolated log data is subjected to a deviation correction process to obtain normalized log data.

[0014] Further, after obtaining the first target number of log data from the log data sorting result as the target log data, the method further includes: The log exception processing unit is specifically used for: In the case where the target log data has an abnormality, determining the log abnormality level corresponding to the target log data; Different log alarm objects and alarm notification methods are determined according to the log abnormality level.

[0015] Furthermore, the hydraulic model multi-source log processing device further includes: In the case where the target log data has an abnormality, determining a model node identifier of the target log data record; The corresponding model node task is determined according to the model node identifier; wherein the model node task includes at least one of a flood operation configuration task, a runoff calculation task, a reservoir rule scheduling calculation task, a one-dimensional hydrodynamic calculation task, an optimized scheduling operation task, an optimized scheduling calculation task and a two-dimensional hydrodynamic calculation task.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-source log processing method for a water conservancy model as described in any one of the above items is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-source log processing method for a water conservancy model as described in any of the above items is implemented.

[0018] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-source log processing method for a water conservancy model as described in any of the above items.

[0019] The multi-source log processing method of a water conservancy model provided by the present invention obtains the original log data of each model node task in a water conservancy model scheduling link, performs dimension mapping processing on the original log data based on log dimension mapping logic, and obtains log data of a target dimension; normalizes the log data of the target dimension to obtain normalized log data, and stores it in a distributed search engine cluster; obtains a query word input by a user end, searches for all normalized log data containing the query word, obtains the first target number of log data as target log data from the log data sorting result, and returns it to the user end, which can effectively improve the multi-source log processing efficiency of the water conservancy model, quickly obtain the target log data of the water conservancy model at different model node tasks, thereby facilitating the management and maintenance of the water conservancy model scheduling link and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is one of the flow charts of the multi-source log processing method of the water conservancy model provided by the present invention.

[0022] Figure 2 This is the second flow chart of the multi-source log processing method of the water conservancy model provided by the present invention.

[0023] Figure 3 It is a schematic diagram of the log dimension mapping logic provided by the present invention.

[0024] Figure 4 It is a schematic diagram of the observation form of the log data service running status provided by the present invention.

[0025] Figure 5 This is the third flow chart of the multi-source log processing method of the water conservancy model provided by the present invention.

[0026] Figure 6 This is the fourth flow chart of the multi-source log processing method of the water conservancy model provided by the present invention.

[0027] Figure 7 This is the fifth flow chart of the multi-source log processing method of the water conservancy model provided by the present invention.

[0028] Figure 8 It is a structural schematic diagram of a hydraulic model multi-source log processing device provided by the present invention.

[0029] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Combine the following Figure 1-Figure 9 The present invention describes a method and device for processing multi-source logs of a water conservancy model, and describes an embodiment thereof in detail.

[0032] The following is a detailed description of an embodiment of the multi-source log processing method for a water conservancy model according to the present invention. Figure 1 As shown, it is one of the flow charts of the multi-source log processing method of the water conservancy model provided by the present invention, and the specific implementation process includes the following steps: Step 101: Obtain the original log data of each model node task in the water conservancy model scheduling link.

[0033] In an embodiment of the present invention, log embedding processing is performed on each model node task in the water conservancy model scheduling link to obtain the original log data during the execution of each model node task in the water conservancy model scheduling link by log embedding; wherein, the original log data contains the identification of each model node task in different execution states.

[0034] like Figure 2As shown, in the process of obtaining the original log data of each model node task in the water conservancy model scheduling link, the present invention provides multi-source and multi-path access channels for the water conservancy model scheduling link, including log SDK access channels and log API access channels. In the actual implementation process, the appropriate log access channel can be selected according to the conditions of the access subject itself. In addition, in the process of log point pre-calculation and disposal, the specific method is as follows: according to different water conservancy model scheduling links, a unique model node identifier is designed for different water conservancy model execution nodes, and the state information before, during and after the execution of the point is buried. The log SDK access channel adopts the AOP (aspect-oriented programming) method. Under the premise of not invading the water conservancy model code, the asynchronous state acquisition records the original log data of the state information and execution status information of each model node task of the water conservancy model; the log API access channel is compatible with multiple terminals and heterogeneous programming language environments, and the original log data is collected through the industry-standard network communication protocol.

[0035] Step 102: Perform dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension.

[0036] Specifically, in order to avoid the problem of dimensional expansion and redundancy of the original log data caused by field differences between different water conservancy model manufacturers, the present invention maps the dimensions during dimension extraction, that is, the original log data is dimensionally mapped based on a preset log dimension mapping logic. The log dimension mapping logic provides up to 6 custom filtering dimension items and supports the following: Figure 3The two log dimension mapping logics of 1 to 1 and many to 1 are shown. Through the log dimension mapping logic, the present invention can support multiple business parameters to participate in the dimension screening calculation, and at the same time ensure that the number of dimension columns will not expand indefinitely. It should be noted that business parameter 1, business parameter 2, business parameter 3, and business parameter 4 are enumerations of the use of mapping schemes in the log data mapping processing scheme, which are not limited to the above four types of business parameters, but more describe the mapping logic processing process. For example: in a 1 to 1 mapping scenario, business parameter 1 is the model node identifier A, operation time B, and model node status C, corresponding to monitoring dimension 1, and the reported value of business parameter 1 needs to be completely copied, that is, the original value is copied to monitoring dimension 1. In the many-to-1 mapping scenario, the reported value of business parameter 2 corresponds to the flood forecasting task number X and the reservoir joint scheduling task number Y of the model node task; the reported value of business parameter 3 corresponds to the runoff calculation L, one-dimensional hydrodynamic calculation M, and two-dimensional hydrodynamic calculation N of the model node task; the reported value of business parameter 4 corresponds to the problem of missing progress status data U, non-convergence of model calculation V, and completion of model calculation W. Specifically, during the calculation of the model node task, after collecting the reported values ​​of the above business parameters 2, 3, and 4, by querying the mapping relationship table, the model node task progress result can locate XLU corresponding to scenario 1: the problem of missing progress status of flood forecast task X during runoff calculation task L; XMV corresponding to scenario 2: the problem of model calculation non-convergence during the calculation of one-dimensional hydrodynamic task M of flood forecast task X; YNW corresponding to scenario 3: the model calculation of reservoir joint dispatch task Y during two-dimensional hydrodynamic calculation N is completed W (i.e., successfully executed). Among them, parameter 1 in the mapping relationship table is business parameter 1, parameter 2 is business parameter 2, parameter 3 is business parameter 3, and parameter 4 is business parameter 4.

[0037] Step 103: normalize the log data of the target dimension to obtain normalized log data, and store the normalized log data in a distributed search engine cluster.

[0038] In an embodiment of the present invention, the log data of the target dimension can be converted into structured log data, the structured log data can be cleaned to obtain cleaned log data, the cleaned log data can be interpolated to obtain interpolated log data, and the interpolated log data can be corrected to obtain normalized log data.

[0039] In the specific implementation process, the log SDK access channel and the log API access channel perform normalization processing on the log data of the target dimension of the accessed multi-source by defining the monitoring metadata, including defining a unified domain model, data cleaning, interpolation processing and correction processing, etc., in order to convert the log data of the target dimension of the multi-source of different scales or ranges into the same scale, so that it can be fairly compared and processed in the subsequent data analysis or link tracking process. In addition, in the embodiment of the present invention, the relationship between the model node identification and the model node task and the model node status is as follows: First, business definition: model log writing / storage; model scheduling task: manual interactive forecast. Model node number (i.e., model node identification): M010; model node task: flood operation configuration; model node status: start calculation log, success / failure log. Model node number: M020; model node task: runoff calculation; model node status: start calculation log, success / failure log. Model node number: M030; model node task: reservoir rule scheduling calculation; model node status: start calculation log, success / failure log. Model node number: M040; Model node task: one- and two-dimensional hydrodynamic calculation; Model node status: start calculation log, success / failure log. Model node number: M050; Model node task: optimization scheduling job configuration; Model node status: start calculation log, success / failure log. Model node number: M060; Model node task: optimization scheduling calculation; Model node status: start calculation log, success / failure log. Model node number: M070; Model node task: hydrodynamic calculation; Model node status: start calculation log, success / failure log. Model node number: M080; Model node task: submergence analysis calculation; Model node status: start calculation log, success / failure log.

[0040] In addition, the methods for data cleaning, interpolation and correction of the log data of the target dimension are as follows: the log data of the target dimension is parsed through regular expressions and JSON parsing libraries to convert it into structured data (such as JSON format, CSV format, etc.), and pre-cleaned according to the preset cleaning conditions or cleaning standards to remove redundant and abnormal log data; for each water conservancy model node link timing, combined with the actual business operation operation status, the operation status of each node is evaluated and calculated (normal, abnormal, completed), and the corresponding result records are output. For example: when a node is abnormal, the corresponding business conversion is calculated and handled in advance, and the corresponding technical and business details logs are generated to facilitate subsequent process calls. In addition, when the system faces extreme scenarios, service failures, network jitter, request congestion, etc., log monitoring data is prone to congestion / suspension, resulting in data delays, and data offsets when it is finally restored. In terms of the observation form of the service operation status, it is presented as follows Figure 4In the form shown. To prevent the log data from being offset, the present invention introduces a watermark monitoring mechanism on the log data receiving service end, that is, the maximum timestamp of the processed data is used to express the final state of the water conservancy model node task. For example: when the current water level advances to time t, it means that the data before time t has been processed and will not appear again; based on this setting, the present invention can determine that the node state before time t will not change again; however, in the actual production environment, due to the existence of long-tail data, it is difficult to achieve the ideal situation that all log data before time t has been processed, so the present invention will set a tolerance percentage (for example, 99.9%) for deviation correction. The monitoring data before the water level is a fixed value, and the data after the water level may change over time because it is still being processed, and finally eliminates the data offset problem caused by the server. Further, the cloud gateway of the present invention selects the corresponding communication protocol (RPC, MQTT, HTTP RESTful, etc.) for different log access methods, and performs access authentication and shields malicious network attacks to ensure the effective distribution and aggregation of log data into the warehouse.

[0041] like Figure 4 As shown, the subscription operator, aggregation operator and ES output operator describe data elements or data dimensions. Here, data aggregation is to perform data dimension reduction. Specifically, taking the statistical stability of the two-dimensional hydrodynamic model node task as an example, the subscription operator includes all the original log data of the reservoir joint scheduling task: model node identification, node input parameter data, node output parameter data, node start time, node calculation end time, node execution time, node status, etc.; in the aggregation stage, the two-dimensional hydrodynamic node status and execution time are used as aggregation operators; the overall proportion of nodes with successful calculations are statistically output, the proportion of node execution time within 5 minutes, 10 minutes, 20 minutes, 30 minutes, etc., the distribution of the start time of different execution times in 1 day and 24 hours, and the proportion of failed nodes, etc. The above aggregation results are used as data elements or data dimensions of each operator, and the data is structured and stored in the structured analysis library through the ES output operator, and finally the model operation monitoring chart data is formed.

[0042] After the log data of the target dimension is normalized and the normalized log data is obtained, the normalized log data can also be stored in a distributed search engine cluster (i.e., Elasticsearch cluster) to achieve massive data storage and efficient data indexing, full-text retrieval and analysis. The specific method is as follows: After the log data is stored in the distributed search engine cluster, an inverted index is generated for key indicators such as model node identification, time series information, and status information.

[0043] Step 104: obtain the query word input by the user terminal, and search for the query word in the term table corresponding to the distributed search engine cluster to obtain the inverted list corresponding to the query word; search for all normalized log data containing the query word according to the inverted list; sort all normalized log data containing the query word to obtain the log data sorting result; obtain the first target number of log data from the log data sorting result as the target log data, and return it to the user terminal.

[0044] Specifically, when a user enters a query term, the query term is first searched in the term table to obtain its corresponding inverted list, and then all log data (i.e., log records) containing the query term are quickly found based on the inverted list. These log data are further ranked based on a preset relevance algorithm (such as TF-IDF, BM25, etc.) and time series, and the first target number of log data is obtained from the log data sorting results as the target log data. Finally, the most relevant results (i.e., target log data) are returned to the user end.

[0045] In an embodiment of the present invention, after obtaining the normalized log data, it also includes: storing the normalized log data as log aggregation data in a log aggregation analysis library, distributing the log aggregation data in the log aggregation analysis library to the corresponding aggregation operator corresponding to the aggregation dimension for aggregation processing according to a preset aggregation dimension, obtaining the aggregation processing result of the log data, and storing the aggregation processing result of the log data in a structured analysis library, so as to perform structured analysis according to the aggregation processing result of the log data stored in the structured analysis library, and output the corresponding structured analysis result.

[0046] Specifically, a large amount of normalized log data is aggregated and calculated according to certain rules or dimensions (such as time series, previous and next node dependencies, scheduling status, etc.) to obtain more meaningful analysis statistics (i.e., corresponding structured analysis results), and describe the model scheduling process and evaluate the link health status from different angles. Taking time aggregation as an example, the execution time of each node in the statistical model scheduling task is realized through log monitoring streaming tasks: Figure 5As shown in the figure, the preset subscription operator is used to distribute the log aggregation data through the hash value of the dimension combination, ensuring that the log data that needs to be aggregated in the same dimension will be distributed to the same aggregation operator to avoid the problem of multi-instance aggregation. The preset aggregation operator is used to continuously accumulate the aggregation value in the memory, and the statistics are triggered by the scheduled task to output the aggregation processing results of the log data, and the current accumulated value is cleared to calculate the next time window. Finally, the aggregation operator data forms a structured statistical result arranged in the order of the model task nodes (that is, the aggregation processing result of the log data), and is stored in the structured analysis library, which is provided for subsequent business use in the form of charts, dashboards, etc., which is convenient for monitoring process status, analyzing model node card points, and optimizing business.

[0047] In addition, it should be noted that the present invention can achieve technical decoupling with the water conservancy model. That is, in the actual operation process, it is not strongly bound to the water conservancy model, and the technical decoupling prevents the failure of log fusion after the model is abnormal, and the specific abnormal node and cause cannot be viewed. At the same time, it is not strongly bound to a specific ecology and technology, which facilitates the access to different models and model technologies.

[0048] After obtaining the first target number of log data as the target log data from the log data sorting result, it also includes: if the target log data has an abnormality, judging the log abnormality level corresponding to the target log data; determining different log alarm objects and alarm notification methods according to the log abnormality level. That is, the present invention supports log abnormality alarms: different log alarms can be made according to different log abnormality levels, and different log alarm objects and alarm notification methods are determined according to the levels, so as to avoid the operation risks in the water conservancy model in advance and deal with them in advance.

[0049] In addition, when there is an abnormality in the target log data, the model node identifier of the target log data record is determined; the corresponding model node task is determined according to the model node identifier; wherein the model node task includes at least one of a flood operation configuration task, a runoff calculation task, a reservoir rule scheduling calculation task, a one-dimensional hydrodynamic calculation task, an optimized scheduling operation task, an optimized scheduling calculation task, and a two-dimensional hydrodynamic calculation task.

[0050] Formulate corresponding node call access rules from each node of the water conservancy model dispatch: Model node identification: M010; Model node name: Flood operation configuration; Log writing user: Current user. Model node identification: M020; Model node name: Runoff calculation; Log writing user: Current user. Model node identification: M030; Model node name: Reservoir rule dispatch calculation; Log writing user: Current user. Model node identification: M040; Model node name: One-dimensional hydrodynamic calculation; Log writing user: Current user. Model node identification: M050; Model node name: Optimization dispatch operation; Log writing user: Current user. Model node identification: M060; Model node name: Optimization dispatch calculation; Log writing user: Current user. Model node identification: M070; Model node name: Hydrodynamic calculation; Log writing user: Current user. Model node identification: M080; Model node name: Inundation analysis calculation; Log writing user: Current user.

[0051] The present invention sorts out the manual interaction forecasting job execution progress sequence diagram from the perspective of model process initiation, progress status query, log detail viewing, execution node matching initiation process and other functional design aspects. Figure 6 and 7 As shown. The water conservancy model starts the runoff calculation and reservoir rule scheduling calculation nodes according to the application system configuration, and the service log points are buried to record the different execution nodes, different execution status technologies, and business logs of the corresponding model node tasks. Standardize the scheduling time access attributes: The model execution node points are uniformly stored, installed, and managed. Each log definition must include a unified process number (logUniCode), a model node identifier (modelCode), and a success / failure mark (invokeSuccessed, 9-start calculation, 1-calculation success, 0-calculation abnormality) to facilitate the execution of model node task progress, status, and detailed inquiries.

[0052] Specifically, attribute: id; Description: unique primary key of index; Notes: unique. Attribute: logUniCode; Description: unified number of call chain log; Notes: unified maintenance. Attribute: logSource; Description: log data source; Notes: maintained according to actual situation. Attribute: logUser; Description: record user; Notes: system record. Attribute: logTime; Description: log time; Notes: system record. Attribute: modelCode; Description: water conservancy model number; Notes: unified issuance by the system. Attribute: modelName; Description: water conservancy model node name; Notes: unified issuance by the system. Attribute: timeConsume; Description: service calculation time; Notes: buried point calculation time. Attribute: upperCaseNo; Description: upper task call document; Notes: record if there is one. Attribute: currentCaseNo; Description: current level task call document. Attribute: invokeUrl; Description: call service link. Attribute: invokeSuccessed; Description: tracking point identification; Notes: 9-start 0-failure 1-success 2-calculating. Attribute: responseMsg; Description: business description; Notes: business language record needs to be converted. Attribute: requestParams; Description: model input parameter. Attribute: responseValue; Description: model output parameter. Attribute: exceptionInfo; Description: exception information.

[0053] The present invention uses the water conservancy model scheduling log data to fully automatically manage the water conservancy model business flow life cycle online: Based on a deep understanding of the water conservancy model scheduling mechanism, the present invention uniformly and standardizedly numbers the water conservancy model scheduling nodes, supports the fusion management and storage of any model node logs during the scheduling process, synchronizes with the actual operation nodes of the model, and converts the operation status of the model mechanism into the foreground technology and business language, solves the problems of log data dimension expansion and data offset, and performs visual application analysis and utilization. For the construction of digital twin water conservancy, the model call chain is uniformly managed and maintained, and technical standards are formulated, which truly achieves the use from the background to the foreground, from the settings to the business applications, which not only standardizes the technical call process, but also satisfies the use of business personnel.

[0054] The present invention adopts a method combining structured pruning and unstructured pruning. By acquiring the original log data of each model node task in the water conservancy model scheduling link, the original log data is dimensionally mapped based on the log dimension mapping logic to obtain the log data of the target dimension; the log data of the target dimension is normalized to obtain the normalized log data, and stored in a distributed search engine cluster; the query word input by the user end is obtained, all the normalized log data containing the query word are searched, and the log data of the first target number is obtained from the log data sorting result as the target log data, and returned to the user end. The target log data of the water conservancy model at different model node tasks can be quickly obtained, thereby facilitating the management and maintenance of the water conservancy model scheduling link and improving the user experience.

[0055] The multi-source log processing device for a hydraulic model provided by the present invention is described below. The multi-source log processing device for a hydraulic model described below and the multi-source log processing method for a hydraulic model described above can be referred to in correspondence with each other. Figure 8 As shown, it is a schematic diagram of the structure of the hydraulic model multi-source log processing device provided by the present invention. The hydraulic model multi-source log processing device of the present invention specifically includes the following parts: The log data acquisition unit 801 is used to acquire the original log data of each model node task in the water conservancy model scheduling link.

[0056] The dimension mapping processing unit 802 is used to perform dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension.

[0057] The normalization processing unit 803 is used to perform normalization processing on the log data of the target dimension to obtain normalized log data, and store the normalized log data in the distributed search engine cluster.

[0058] The target log data processing unit 804 is used to obtain the query word input by the user terminal, and search for the query word in the term table corresponding to the distributed search engine cluster to obtain the inverted list corresponding to the query word; find all normalized log data containing the query word according to the inverted list; sort all the normalized log data containing the query word to obtain the log data sorting result; obtain the first target number of log data from the log data sorting result as the target log data, and return it to the user terminal.

[0059] Furthermore, after obtaining the normalized log data, the following steps are also included: Structured analysis unit, specifically used for: The normalized log data is stored in a log aggregation analysis library as log aggregation data; According to the preset aggregation dimension, the log aggregation data in the log aggregation analysis library is distributed to the corresponding aggregation operator corresponding to the aggregation dimension for aggregation processing, and the aggregation processing result of the log data is obtained, and the aggregation processing result of the log data is stored in the structured analysis library, so as to perform structured analysis according to the aggregation processing result of the log data stored in the structured analysis library and output the corresponding structured analysis result.

[0060] Furthermore, the log data acquisition unit is specifically used to: Log embedding processing is performed on each model node task in the water conservancy model scheduling link to obtain the original log data during the execution of each model node task in the water conservancy model scheduling link through log embedding; wherein the original log data contains the identification of each model node task in different execution states.

[0061] Furthermore, the normalization processing unit is specifically used to: Converting the log data of the target dimension into structured log data; Performing data cleaning on the structured log data to obtain cleaned log data; Perform interpolation processing on the cleaned log data to obtain interpolated log data; The interpolated log data is subjected to a deviation correction process to obtain normalized log data.

[0062] Further, after obtaining the first target number of log data from the log data sorting result as the target log data, the method further includes: The log exception processing unit is specifically used for: In the case where the target log data has an abnormality, determining the log abnormality level corresponding to the target log data; Different log alarm objects and alarm notification methods are determined according to the log abnormality level.

[0063] Furthermore, the hydraulic model multi-source log processing device further includes: In the case where the target log data has an abnormality, determining a model node identifier of the target log data record; The corresponding model node task is determined according to the model node identifier; wherein the model node task includes at least one of a flood operation configuration task, a runoff calculation task, a reservoir rule scheduling calculation task, a one-dimensional hydrodynamic calculation task, an optimized scheduling operation task, an optimized scheduling calculation task and a two-dimensional hydrodynamic calculation task.

[0064] The multi-source log processing device for a water conservancy model provided by the present invention obtains the original log data of each model node task in a water conservancy model scheduling link, performs dimension mapping processing on the original log data based on log dimension mapping logic, and obtains log data of a target dimension; normalizes the log data of the target dimension to obtain normalized log data, and stores it in a distributed search engine cluster; obtains a query word input by a user end, searches for all normalized log data containing the query word, obtains the first target number of log data as target log data from the log data sorting result, and returns it to the user end, which can effectively improve the multi-source log processing efficiency of the water conservancy model, quickly obtain the target log data of the water conservancy model at different model node tasks, thereby facilitating the management and maintenance of the water conservancy model scheduling link and improving the user experience.

[0065] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device (ie, electronic device) may include: a processor (processor) 901, a communication interface (Communications Interface) 904, a memory (memory) 902 and a communication bus 903, wherein the processor 901, the communication interface 904, and the memory 902 communicate with each other via the communication bus 903. The processor 901 can call the logic instructions in the memory 902 to execute the multi-source log processing method of the water conservancy model, which includes: obtaining the original log data of each model node task in the water conservancy model scheduling link; performing dimension mapping processing on the original log data based on the preset log dimension mapping logic to obtain log data of the target dimension; normalizing the log data of the target dimension to obtain normalized log data, and storing the normalized log data in the distributed search engine cluster; obtaining the query word input by the user end, and searching the query word in the term table corresponding to the distributed search engine cluster to obtain the inverted list corresponding to the query word; searching for all normalized log data containing the query word according to the inverted list; sorting all normalized log data containing the query word to obtain the log data sorting result; obtaining the first target number of log data from the log data sorting result as the target log data, and returning it to the user end.

[0066] In addition, the logic instructions in the above-mentioned memory 902 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0067] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the water conservancy model multi-source log processing method provided by the above methods, the method including: obtaining the original log data of each model node task in the water conservancy model scheduling link; performing dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension; normalizing the log data of the target dimension to obtain normalized log data, and storing the normalized log data in a distributed search engine cluster; obtaining a query term input by a user terminal, and searching for the query term in a term table corresponding to the distributed search engine cluster to obtain an inverted list corresponding to the query term; searching for all normalized log data containing the query term according to the inverted list; sorting all normalized log data containing the query term to obtain a log data sorting result; obtaining the first target number of log data from the log data sorting result as the target log data, and returning it to the user terminal.

[0068] On the other hand, the present application also provides a computer-readable storage medium, which includes a stored program, wherein the program, when running, executes the water conservancy model multi-source log processing method provided by the above methods, the method comprising: obtaining the original log data of each model node task in the water conservancy model scheduling link; performing dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension; normalizing the log data of the target dimension to obtain normalized log data, and storing the normalized log data in a distributed search engine cluster; obtaining a query term input by a user terminal, and searching for the query term in the term table corresponding to the distributed search engine cluster to obtain an inverted list corresponding to the query term; searching for all normalized log data containing the query term according to the inverted list; sorting all normalized log data containing the query term to obtain a log data sorting result; obtaining the first target number of log data from the log data sorting result as the target log data, and returning it to the user terminal.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing multi-source logs of a water conservancy model, characterized in that: include: Obtain the original log data of each model node task in the water conservancy model scheduling link; Performing dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension; Normalizing the log data of the target dimension to obtain normalized log data, and storing the normalized log data in a distributed search engine cluster; Obtain a query word input by a user terminal, and search for the query word in the term table corresponding to the distributed search engine cluster to obtain an inverted list corresponding to the query word; search for all normalized log data containing the query word according to the inverted list; sort all normalized log data containing the query word to obtain a log data sorting result; obtain the first target number of log data from the log data sorting result as target log data, and return it to the user terminal.

2. The multi-source log processing method of the water conservancy model according to claim 1 is characterized in that: After obtaining the normalized log data, it also includes: The normalized log data is stored in a log aggregation analysis library as log aggregation data; According to the preset aggregation dimension, the log aggregation data in the log aggregation analysis library is distributed to the corresponding aggregation operator corresponding to the aggregation dimension for aggregation processing, and the aggregation processing result of the log data is obtained, and the aggregation processing result of the log data is stored in the structured analysis library, so as to perform structured analysis according to the aggregation processing result of the log data stored in the structured analysis library and output the corresponding structured analysis result.

3. The multi-source log processing method of the hydraulic model according to claim 1 is characterized in that: The method of obtaining the original log data of each model node task in the water conservancy model scheduling link includes: Log embedding processing is performed on each model node task in the water conservancy model scheduling link to obtain the original log data during the execution of each model node task in the water conservancy model scheduling link through log embedding; wherein the original log data contains the identification of each model node task in different execution states.

4. The method for processing multi-source logs of a hydraulic model according to claim 1, characterized in that: The normalizing the log data of the target dimension to obtain the normalized log data specifically includes: Converting the log data of the target dimension into structured log data; Performing data cleaning on the structured log data to obtain cleaned log data; Perform interpolation processing on the cleaned log data to obtain interpolated log data; The interpolated log data is subjected to a deviation correction process to obtain normalized log data.

5. The method for processing multi-source logs of a hydraulic model according to claim 1, characterized in that: After obtaining the first target number of log data from the log data sorting result as the target log data, the method further includes: In the case where the target log data has an abnormality, determining the log abnormality level corresponding to the target log data; Different log alarm objects and alarm notification methods are determined according to the log abnormality level.

6. The method for processing multi-source logs of a hydraulic model according to claim 1, characterized in that: Also includes: In the case where the target log data has an abnormality, determining a model node identifier of the target log data record; The corresponding model node task is determined according to the model node identifier; wherein the model node task includes at least one of a flood operation configuration task, a runoff calculation task, a reservoir rule scheduling calculation task, a one-dimensional hydrodynamic calculation task, an optimized scheduling operation task, an optimized scheduling calculation task and a two-dimensional hydrodynamic calculation task.

7. A multi-source log processing device for a water conservancy model, characterized in that: include: A log data acquisition unit, used to acquire the original log data of each model node task in the water conservancy model scheduling link; A dimension mapping processing unit, configured to perform dimension mapping processing on the original log data based on a preset log dimension mapping logic to obtain log data of a target dimension; A normalization processing unit, used to perform normalization processing on the log data of the target dimension to obtain normalized log data, and store the normalized log data in a distributed search engine cluster; The target log data processing unit is used to obtain the query word input by the user terminal, and search for the query word in the term table corresponding to the distributed search engine cluster to obtain the inverted list corresponding to the query word; find all normalized log data containing the query word according to the inverted list; sort all the normalized log data containing the query word to obtain the log data sorting result; obtain the first target number of log data from the log data sorting result as the target log data, and return it to the user terminal.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the water conservancy model multi-source log processing method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multi-source log processing method of the water conservancy model as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the multi-source log processing method of the water conservancy model as described in any one of claims 1 to 6.

Images