Log data storage method, electronic device, storage medium, and program product

Abstract

The present disclosure provides a log data storage method, an electronic device, a readable storage medium and a program product. The log data storage method comprises: determining a first execution parameter and a second execution parameter according to a received log data storage request; storing target log data to a preset disk according to the first execution parameter; and storing the target log data stored in the preset disk to a preset database according to the second execution parameter when a preset condition is met.

Description

Technical Field

[0001] This disclosure relates to the field of data storage technology, and in particular to a log data storage method, electronic device, storage medium, and program product. Background Technology

[0002] With the continuous development of computer technology, the amount of electronic data is increasing, and data storage is becoming increasingly important.

[0003] Currently, related technologies rely on cumbersome manual methods to store log data. As the volume of log data increases, this requires a significant investment of manpower. Furthermore, errors in manual operation can lead to data storage failures or errors in log data storage. In other words, these technologies suffer from high labor costs, high storage error rates, and high storage failure rates. Summary of the Invention

[0004] This disclosure provides a log data storage method, electronic device, storage medium, and program product.

[0005] The first aspect of this disclosure provides a log data storage method, comprising: determining a first execution parameter and a second execution parameter according to a received log data storage request; storing a target log data to a preset disk according to the first execution parameter; and storing the target log data stored in the preset disk to a preset database according to the second execution parameter when a preset condition is met.

[0006] In some implementations, determining the first execution parameter and the second execution parameter based on the received log data storage request includes: determining the identifier information of the source message queue of the target log data, the flow rate of the target log data, the ingestion field of the target log data, and the target storage permissions based on the received log data storage request; generating the first execution parameter based on the identifier information of the source message queue, the flow rate of the target log data, and the target storage permissions; and generating the second execution parameter based on the ingestion field.

[0007] In some implementations, generating the first execution parameter based on the identifier information of the source message queue, the traffic of the target log data, and the target storage permissions includes: determining the running memory size based on the traffic of the target log data; determining the disk write path of the target log data based on the identifier information of the source message queue and a preset storage path; determining the number of target partitions corresponding to the identifier information of the source message queue based on message queue configuration data, wherein the message queue configuration data includes the identifier information of multiple message queues and their corresponding number of partitions; using the number of target partitions as the parallelism; determining the target disk write execution cluster information corresponding to the target storage permissions based on permission configuration data, wherein the permission configuration data includes multiple storage permissions and their corresponding disk write execution cluster information; and using one or more of the running memory size, the disk write path of the target log data, the parallelism, and the target disk write execution cluster information as the first execution parameter.

[0008] In some implementations, generating the second execution parameter based on the input field includes: generating a data acquisition function based on the input field, the data acquisition function being used to acquire data matching the input field; generating a target statement based on the input field, the target statement being used to write the data matching the input field into the preset database; and using one or more of the data acquisition function and the target statement as the second execution parameter.

[0009] In some implementations, the second execution parameter includes an input field and a target statement. The preset database includes a raw data layer and a data detail layer. The step of storing the target log data stored in the preset disk into the preset database according to the second execution parameter includes: storing the target log data stored in the preset disk into the raw data layer; parsing the target log data stored in the raw data layer to obtain structured data of the target log data; and storing the structured data into the data detail layer according to the second execution parameter.

[0010] In some implementations, the second execution parameters include a data acquisition function and a target statement. The step of storing the structured data into the data detail layer according to the second execution parameters includes: executing the data acquisition function to obtain data matching the input field from the structured data; and executing the target statement to store the obtained data matching the input field into the data detail layer.

[0011] In some implementations, storing the target log data stored in the preset disk into the preset database according to the second execution parameter includes: determining the target time of the target log data stored in the preset disk based on the time format and regular expression; and storing the target log data in the preset disk whose target time is within a preset time period into the preset database according to the second execution parameter.

[0012] In some implementations, before storing the target log data stored in the preset disk into the preset database according to the second execution parameter, the method further includes: determining the data label of the input field; and storing the data label into a data map system.

[0013] In some implementations, the method further includes: in response to detecting abnormal operating data, determining a target abnormality type based on the abnormal operating data; determining a target alarm method and a target alarm object corresponding to the target abnormality type based on alarm configuration data, wherein the alarm configuration data includes multiple abnormality types and their corresponding alarm methods and alarm objects; and issuing an alarm to the target alarm object according to the target alarm method.

[0014] A second aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a log data storage method according to any embodiment of this disclosure.

[0015] A third aspect of this disclosure provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the log data storage method described in any embodiment of this disclosure.

[0016] The fourth aspect of this disclosure provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the log data storage method described in any embodiment of this disclosure. Attached Figure Description

[0017] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0018] Figure 1 This is a flowchart illustrating some embodiments of the log data storage method disclosed herein.

[0019] Figure 2 This is a schematic diagram illustrating the process of determining the first execution parameter and the second execution parameter in some embodiments of this disclosure.

[0020] Figure 3 This is a schematic diagram illustrating the process of generating the first execution parameter in some embodiments of this disclosure.

[0021] Figure 4 This is a schematic diagram illustrating the process of generating the second execution parameter in some embodiments of this disclosure.

[0022] Figure 5 This is a schematic diagram illustrating the process of storing target log data according to some embodiments of this disclosure.

[0023] Figure 6 This is a schematic diagram illustrating the process of storing structured data in some embodiments of this disclosure.

[0024] Figure 7 This is another schematic diagram illustrating the process of storing target log data in some embodiments of this disclosure.

[0025] Figure 8 This is another flowchart illustrating a log data storage method according to some embodiments of this disclosure.

[0026] Figure 9 This is a visualization example of data labels for some embodiments of this disclosure.

[0027] Figure 10 This is yet another flowchart illustrating a log data storage method according to some embodiments of this disclosure.

[0028] Figure 11 This is a schematic block diagram of a log data storage device implemented using a processor, according to one embodiment of the present disclosure. Detailed Implementation

[0029] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0030] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0032] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0033] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0034] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0035] The log data storage method disclosed herein can be applied to the log data storage device disclosed herein, which can be configured on an electronic device. The electronic device can be a server or a terminal device, and the terminal device can be a mobile terminal, such as a mobile phone, tablet computer, personal digital assistant, or other hardware device with various operating systems.

[0036] The following text combines Figures 1 to 11 The log data storage method and log data storage device disclosed herein are described in detail.

[0037] Figure 1 This is a flowchart illustrating some embodiments of the log data storage method disclosed herein. Please refer to... Figure 1 The log data storage method S100 provided in this disclosure may include steps S110, S120 and S130.

[0038] S110: Determine the first execution parameter and the second execution parameter based on the received log data storage request.

[0039] S120: Store the target log data to the preset disk according to the first execution parameter.

[0040] S130: When the preset conditions are met, the target log data stored in the preset disk is stored in the preset database according to the second execution parameter.

[0041] The log data storage method of this disclosure automatically determines a first execution parameter and a second execution parameter based on the received log data storage request, automatically stores the target log data to a preset disk according to the first execution parameter, and automatically stores the target log data stored in the preset disk to a preset database according to the second execution parameter when preset conditions are met. This avoids tedious manual operations and saves labor costs. Furthermore, since the first and second execution parameters are automatically generated rather than manually set, it helps reduce storage error rates and storage failure rates. In addition, storing the target log data on the preset disk meets the real-time requirement for target log data storage, and storing the target log data in the preset disk to the preset database when preset conditions are met meets the long-term storage requirement for target log data, while also helping to alleviate the pressure on the preset database.

[0042] Optionally, log data storage requests can be triggered by the user, or automatically during or after business operations; no limitation is made here.

[0043] For example, the first execution parameter can be understood as a parameter used to execute a real-time storage task. The real-time storage task can be understood as storing the received target log data to a preset disk. Accordingly, in step S120, the target log data is stored to the preset disk in real time according to the first execution parameter; that is, each time a target log data is received, the target log data is immediately stored to the preset disk.

[0044] For example, the second execution parameter can be understood as a parameter used to execute the offline storage task. The offline storage task can be understood as storing the target log data received within a certain period into a preset database. Given the same amount of target log data, the number of times the offline storage task is executed is less than or equal to the number of times the real-time storage task is executed.

[0045] Optionally, the preset condition can be that the current time is the preset entry time. For example, the hourly times of each day (e.g., 1:00, 2:00, ..., 24:00) are set as the preset entry time. In this way, the target log data stored in the preset disk is stored in the preset database once every hour according to the second execution parameter.

[0046] In one example, the preset entry time is 12:00 noon every day. Then, at 12:00 noon every day, the target log data stored in the preset disk between 12:00 noon of the previous day and 12:00 noon of today is stored in the preset database according to the second execution parameter.

[0047] In another example, the preset entry time is the hour of each day. Therefore, at 8:00 AM each day, the target log data stored between 7:00 AM and 8:00 AM on the preset disk is stored into the preset database according to the second execution parameter; at 9:00 AM each day, the target log data stored between 8:00 AM and 9:00 AM on the preset disk is stored into the preset database according to the second execution parameter; at 10:00 AM each day, the target log data stored between 9:00 AM and 10:00 AM on the preset disk is stored into the preset database according to the second execution parameter, and so on.

[0048] In some implementations, a log integration management system is built based on Spring Boot, which can then automatically generate and load the first and second execution parameters. This simplifies the user's workflow and improves the efficiency of the system.

[0049] Please see Figure 2 In some implementations, step S110 may include steps S111, S112 and S113.

[0050] S111: Determine the source message queue identification information of the target log data, the traffic of the target log data, the entry fields of the target log data, and the target storage permissions based on the received log data storage request.

[0051] For example, the target log data is the log data requested to be stored in the log data storage request. The source message queue can be understood as the message queue from which the target log data originates. The identification information of the source message queue may include the topic name of the source message queue. The traffic of the target log data can be understood as the amount of target log data generated per unit of time.

[0052] Optionally, the log data storage request may include the identification information of the source message queue of the target log data and the traffic of the target log data. Then, after receiving the log data storage request, the identification information of the source message queue of the target log data and the traffic of the target log data can be determined by parsing the received log data storage request.

[0053] For example, the target log data may include multiple fields, where the fields to be stored in the target log data can be understood as the fields among the multiple fields that are expected to be stored in a preset database. In one example, after receiving a log data storage request, the multiple fields of the target log data are parsed out and sent to the log data storage requester so that the log data storage requester can display the parsed multiple fields. The field selection result sent by the log data storage requester is received. The field selection result can be triggered by the user and may include the selected field. The selected field in the field selection result can then be used as the field to be stored in the target log data.

[0054] For example, a log data storage request may include the identifier of the user or service that triggered the request. By looking up a preset correspondence, the target storage permission corresponding to the identifier of the user or service that triggered the request can be determined. The preset correspondence includes multiple preset identifiers and their corresponding preset storage permissions.

[0055] S112: Generate the first execution parameters based on the source message queue's identification information, the target log data's traffic, and the target storage permissions.

[0056] Please see Figure 3 In some implementations, step S112 may include steps S1121, S1122, S1123, S1124, S1125 and S1126.

[0057] S1121: Determine the running memory size based on the traffic of the target log data.

[0058] For example, the running memory size corresponding to the traffic of the target log data can be determined by looking up a preset mapping relationship. The preset mapping relationship may include multiple preset traffic ranges and their corresponding preset running memory sizes.

[0059] In one example, the runtime memory size is used to determine the JobManager and TaskManager parameters during the process of storing the target log data to a preset disk according to the first execution parameters.

[0060] S1122: Determine the disk path for the target log data based on the identifier information of the source message queue and the preset storage path.

[0061] For example, the preset storage path can be a pre-set storage path or a default storage path. After determining the identification information of the source message queue, a folder can be created under the preset storage path with the identification information of the source message queue as the folder name. Therefore, the disk path for the target log data can be understood as the path where the target log data is stored in that folder.

[0062] S1123: Based on the message queue configuration data, determine the number of target partitions corresponding to the identification information of the source message queue. The message queue configuration data includes the identification information of multiple message queues and the number of their corresponding partitions.

[0063] S1124: Use the number of target partitions as the degree of parallelism.

[0064] In one example, parallelism is used to determine the TaskSlotNumber parameter during the process of storing the target log data to a preset disk according to the first execution parameter.

[0065] S1125: Based on the permission configuration data, determine the target disk execution cluster information corresponding to the target storage permission. The permission configuration data includes multiple storage permissions and their corresponding disk execution cluster information.

[0066] For example, the target storage permissions can be the permissions of the space where the user resides. It is understood that different users in different spaces have different resource permissions (i.e., the disk execution clusters they can use).

[0067] For example, the target disk execution cluster information may include the target disk execution cluster name (ClusterName) and the target disk execution message queue name (QueueName). ClusterName can be understood as the name of the cluster used to store the target log data to the preset disk. QueueName can be understood as the name of the message queue used to store the target log data to the preset disk.

[0068] S1126: Use one or more of the following as the first execution parameters: running memory size, target log data write path, parallelism, and target write-to-disk execution cluster information.

[0069] In this way, the first execution parameters can be generated automatically, avoiding tedious manual operations and saving labor costs.

[0070] For example, the running memory size, the disk path of the target log data, the parallelism, and the target disk execution cluster information are all used as the first execution parameters. Then, in step S120, the execution cluster and execution message queue corresponding to the target disk execution cluster information are used to store the target log data to the preset disk according to the running memory size and the parallelism.

[0071] S113: Generate the second execution parameter based on the input field.

[0072] Please see Figure 4 In some implementations, step S113 may include steps S1131, S1132 and S1133.

[0073] S1131: Generate a data retrieval function based on the input field. The data retrieval function is used to retrieve data that matches the input field.

[0074] For example, the data retrieval function can be the get_json_object function of Hive SQL.

[0075] For example, the field names for the input field can be automatically populated into a pre-configured data retrieval function template, which can automatically generate the data retrieval function. Data matching the input field can include fields that are the same as the input field and their corresponding field values.

[0076] S1132: Generate target statement based on the input field. The target statement is used to write the data that matches the input field into the preset database.

[0077] For example, the target statement is a Structured Query Language (SQL) statement.

[0078] For example, the field names for data entry can be automatically populated into a pre-configured target statement template to generate the target statement.

[0079] S1133: Use one or more of the data acquisition function and the target statement as the second execution parameter.

[0080] For example, both the data retrieval function and the target statement are used as the second execution parameters. This allows for the automatic generation of the second execution parameters, avoiding tedious manual operations and saving labor costs.

[0081] The log data storage method described above determines the identifier information of the source message queue, the flow rate of the target log data, and the ingestion fields of the target log data based on the received log data storage request. It then generates first execution parameters based on the source message queue identifier, the flow rate of the target log data, and the target storage permissions, and generates second execution parameters based on the ingestion fields. This automatically generates the first and second execution parameters, avoiding tedious manual operations and saving labor costs. It also helps reduce storage error and failure rates. It is understood that in related technologies, the first and second execution parameters are set manually. However, if the user is unfamiliar with the parameters, manual setting may lead to errors, resulting in storage errors or failures. Alternatively, if both the first and second execution parameters use default parameters without specific settings, they cannot be applied to various target log data, leading to higher storage error and failure rates.

[0082] In some implementations, the pre-defined database includes a raw data layer and a data detail layer. See accordingly. Figure 5 Step S130 may include steps S131, S132 and S133.

[0083] S131: Store the target log data stored in the preset disk to the raw data layer.

[0084] For example, a Hive table is automatically created in the raw data layer, and the target log data stored in the preset disk is stored in the Hive table.

[0085] S132: Parse the target log data stored in the raw data layer to obtain the structured data of the target log data.

[0086] For example, the target log data stored in the original data layer is parsed using Jackson. The JSON data obtained after JSON parsing is the structured data of the target log data. The structured data of the target log data includes key and value, where the key represents the field name of Hive and the value represents the field value.

[0087] S133: Store the structured data to the data detail layer according to the second execution parameter.

[0088] In some implementations, the second execution parameter includes a data acquisition function and a target statement. See accordingly. Figure 6 Step S133 may include steps S1331 and S1332.

[0089] S1331: Execute the data retrieval function to retrieve data from the structured data that matches the fields entered into the database.

[0090] For example, the structured data includes multiple fields and their corresponding field values. The data retrieval function is the get_json_object function of HiveSQL. Then, in step S1331, the get_json_object function is executed to obtain the fields in the structured data that are the same as the fields entered into the database and their corresponding field values. The fields that are the same as the fields entered into the database and their corresponding field values ​​are the data that match the fields entered into the database.

[0091] S1332: Execute the target statement to store the data that matches the retrieved input fields into the data detail layer.

[0092] In this way, the structured data corresponding to the input fields can be automatically stored in the data detail layer based on the data retrieval function and the target statement, thereby improving data usage efficiency, lowering the threshold for users to use data, and facilitating the subsequent use of data that matches the input fields.

[0093] The log data storage method described above stores the target log data stored in the preset disk directly into the original data layer. It automatically parses the target log data stored in the original data layer to obtain structured data, and then stores this structured data into the data detail layer according to the second execution parameters. This ensures that the data stored in the data detail layer is structured, achieving log data availability and facilitating subsequent execution of corresponding business operations based directly on the structured data stored in the data detail layer, thus reducing log usage costs. It is understandable that in related technologies, it is impossible to automatically perform structured parsing of the target log data and directly store it into a preset database. Because the target log data is semi-structured, even if it is stored in the preset database, it cannot be used immediately. Manual structured parsing is required before reuse, which increases the cost of using the target log data.

[0094] Please see Figure 7 In some implementations, step S130 may include steps S134 and S135.

[0095] S134: Determine the target time of the target log data stored in the preset disk based on the time format and regular expression.

[0096] Optionally, the time format may include one or more of consumption time and event time.

[0097] For example, consumption time can be understood as the time when data is used, accessed, or consumed. In one example, if target log data is stored in a preset disk at 10:39, then the consumption time of the target log data is 10:39.

[0098] For example, event time can be understood as the time when the event or situation related to the data occurred. In one example, although the target log data is stored in the preset disk at 10:39, the target log data was generated at 08:39, then the event time of the target log data is 08:39.

[0099] For example, a regular expression can be understood as a way to extract the time from the target log data. In one example, the regular expression is / d{12}, which will extract 12 consecutive digits as the target time of the target log data.

[0100] Furthermore, if the time format is consumption time, the consumption time of the target log data stored in the preset disk is extracted using a regular expression and used as the target time of the target log data; if the time format is event time, the event time of the target log data stored in the preset disk is extracted using a regular expression and used as the target time of the target log data.

[0101] S135: According to the second execution parameter, store the target log data in the preset disk within the preset time period to the preset database.

[0102] Optionally, the end time of the preset time period is the current time, and the duration of the preset time period is the interval between two adjacent offline storage tasks. For example, if the interval is 1 hour, that is, an offline storage task (to store the target log data stored in the preset disk to the preset database) is executed every 1 hour, then the duration of the preset time period is 1 hour.

[0103] In one example, the preset time period is 1 hour in length and the time format is consumption time. If the current time is 9 o'clock, then according to the second execution parameter, multiple target log data stored in the preset disk between 8 o'clock and 9 o'clock will be stored in the preset database.

[0104] The log data storage method described above can accurately determine the target time of the target log data stored in the preset disk based on the time format and regular expression, and then automatically and accurately store the target log data in the preset disk within the preset time period according to the second execution parameter, thereby improving the accuracy and success rate of target log data storage.

[0105] Please see Figure 8 In some embodiments, steps S140 and S150 are included before step S130.

[0106] S140: Determine the data label for the input field.

[0107] For example, data tags are used to describe objective information about the inbound fields. Data tags can be tags automatically generated after parsing the target log data, or tags entered by the user from the target log data; there is no limitation here. Data tags based on the inbound fields can enrich the business scenarios for those fields.

[0108] Please combine Figure 9 In one example, the data label for an input field may include the field's Chinese name, field type, field description, enumerated and default values, security level, whether the field is encrypted, whether the field is nullable, and whether the field is a primary key. Data labels that can be automatically parsed can be automatically parsed by the electronic device; data labels that cannot be automatically parsed can prompt the user to fill them in.

[0109] S150: Store the data labels in the data map system.

[0110] For example, the data mapping system is used to store data labels, which are used to describe the characteristics of the data stored in the data detail layer.

[0111] For example, data labels can be written into a data mapping system using the Java language.

[0112] The log data storage method described above stores the data tags of the input fields in the data map system, thereby facilitating the determination of the characteristics of the corresponding input fields in the data detail layer based on the data tags stored in the data map system, and improving the convenience of data use.

[0113] Please see Figure 10 In some implementations, steps S160, S170 and S180 are also included.

[0114] S160: In response to the detection of abnormal operating data, determine the target abnormality type based on the abnormal operating data.

[0115] For example, during the storage of target log data, when the storage task fails, or the target log data to be stored is severely backed up, or there is insufficient memory, or other abnormalities occur, abnormal operation data will be generated. All generated abnormal operation data will be sent to the same pre-set message queue. By listening to this pre-set message queue, abnormal operation data can be detected in a timely manner.

[0116] For example, abnormal operation data may include an exception flag field. By looking up a pre-stored exception type mapping relationship, the target exception type corresponding to the exception flag field in the abnormal operation data can be determined. The exception type mapping relationship may include multiple preset exception flag fields and their corresponding preset exception types.

[0117] For example, one preset exception type can correspond to multiple preset exception flag fields, or one preset exception type can correspond to one preset exception flag field; this is not limited here. It is understood that when one preset exception type corresponds to multiple preset exception flag fields, subsequent configuration only requires setting a mapping relationship between the preset exception type, alarm method, and alarm object to complete the alarm configuration for multiple preset exception flag fields, without needing to configure the alarm configuration for each preset exception flag field separately. This is more convenient and helps simplify the operational logic.

[0118] In one example, if the exception flag field in the abnormal operation data is detected as oom (out of memory), the target exception type can be determined to be insufficient memory; if the exception flag field in the abnormal operation data is detected as failure, the target exception type can be determined to be storage task failure.

[0119] S170: Based on the alarm configuration data, determine the target alarm method and target alarm object corresponding to the target anomaly type. The alarm configuration data includes multiple anomaly types and their corresponding alarm methods and alarm objects.

[0120] For example, the target alarm method may include one or more of the following: sending a WeChat message, sending a WeChat Work message, sending an SMS message, sending an email, and voice broadcast.

[0121] For example, the target alarm object may include one or more of the following: mobile phone number and email address.

[0122] S180: Send an alarm to the target alarm object according to the target alarm method.

[0123] For example, the target anomaly type and abnormal operation data are output to the target alarm object according to the target alarm method, so as to send an alarm to the target alarm object.

[0124] For example, Flink and Kafka can be used to monitor abnormal operational data during the target log data storage process in real time, automatically issuing alerts when abnormal operational data is detected, and performing automated post-processing based on the target exception type. For instance, when the target exception type is storage task failure, the automated post-processing could be automatic restart, which can effectively resolve the storage task failure issue. Thus, in the event of an exception, in addition to timely alerts, the exception can also be handled promptly through automated post-processing. Furthermore, since all abnormal operational data is sent to the same pre-configured message queue, multiple identical exceptions can be deduplicated, resulting in only one alert being issued, which facilitates rapid alert response.

[0125] The log data storage method described above can automatically issue alarms when abnormal operating data is detected, thus improving the degree of automation. On the other hand, it can accurately send alarms to the target alarm object based on the alarm configuration data, ensuring the directionality of task alarms.

[0126] Figure 11 This is a schematic block diagram of a log data storage device implemented using a processor, according to one embodiment of the present disclosure.

[0127] The log data storage device may include corresponding modules that perform one or more steps in the flowchart above. Therefore, each or more steps in the flowchart above can be performed by a corresponding module, and the device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0128] The hardware architecture of the log data storage device disclosed herein can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0129] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0130] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0131] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0132] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0133] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0135] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0136] refer to Figure 11 According to one embodiment of the present disclosure, the log data storage device 1000 of the present disclosure includes a determination module 1002, a first storage module 1004 and a second storage module 1006.

[0137] The determination module 1002 is used to determine the first execution parameter and the second execution parameter according to the received log data storage request; the first storage module 1004 is used to store the target log data to a preset disk according to the first execution parameter; and the second storage module 1006 is used to store the target log data stored in the preset disk to a preset database according to the second execution parameter when the preset conditions are met.

[0138] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, causing the processor or other hardware module to perform the above-described method.

[0139] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the methods described above.

[0140] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the methods described above.

[0141] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0142] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0143] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0144] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0145] At the same time, it is understood that the data involved in this disclosed technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0146] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0148] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A log data storage method, characterized in that, include: The first and second execution parameters are automatically determined based on the received log data storage request; The first execution parameter is used to execute the real-time storage task, and the second execution parameter is used to execute the offline storage task. The target log data is automatically stored to a preset disk according to the first execution parameter; as well as When the preset conditions are met, the target log data stored in the preset disk will be automatically stored in the preset database according to the second execution parameters; The step of determining the first execution parameter and the second execution parameter based on the received log data storage request includes: Based on the received log data storage request, determine the source message queue identification information of the target log data, the traffic of the target log data, the entry fields of the target log data, and the target storage permissions; The first execution parameters are generated based on the identifier information of the source message queue, the traffic of the target log data, and the target storage permissions; and The second execution parameter is generated based on the input field.

2. The log data storage method according to claim 1, characterized in that, The step of generating the first execution parameter based on the identifier information of the source message queue, the traffic of the target log data, and the target storage permissions includes: The running memory size is determined based on the traffic of the target log data; The disk path for the target log data is determined based on the identification information of the source message queue and the preset storage path; Based on the message queue configuration data, determine the number of target partitions corresponding to the identification information of the source message queue. The message queue configuration data includes the identification information of multiple message queues and the number of their corresponding partitions. The number of target partitions is used as the degree of parallelism. Based on the permission configuration data, the target disk execution cluster information corresponding to the target storage permission is determined; the permission configuration data includes multiple storage permissions and their corresponding disk execution cluster information; and One or more of the following are used as the first execution parameters: the running memory size, the disk path of the target log data, the parallelism, and the target disk execution cluster information.

3. The log data storage method according to claim 1, characterized in that, The step of generating the second execution parameter based on the input field includes: A data retrieval function is generated based on the input field, and the data retrieval function is used to retrieve data that matches the input field. A target statement is generated based on the input fields; the target statement is used to write data matching the input fields into the preset database; and One or more of the data acquisition function and the target statement are used as the second execution parameter.

4. The log data storage method according to claim 1, characterized in that, The second execution parameters include the input field and the target statement. The preset database includes a raw data layer and a data detail layer. The step of storing the target log data stored in the preset disk into the preset database according to the second execution parameters includes: The target log data stored in the preset disk is stored in the original data layer; The target log data stored in the original data layer is parsed to obtain structured data of the target log data; and The structured data is stored in the data detail layer according to the second execution parameter.

5. The log data storage method according to claim 1, characterized in that, The step of storing the target log data stored in the preset disk into the preset database according to the second execution parameter includes: The target time of the target log data stored in the preset disk is determined based on the time format and regular expressions; and According to the second execution parameter, the target log data in the preset disk within the preset time period is stored in the preset database.

6. The log data storage method according to any one of claims 1 to 5, characterized in that, Also includes: In response to the detection of abnormal operating data, the target abnormality type is determined based on the abnormal operating data; Based on the alarm configuration data, the target alarm method and target alarm object corresponding to the target anomaly type are determined. The alarm configuration data includes multiple anomaly types and their corresponding alarm methods and alarm objects; and An alarm is sent to the target alarm object according to the target alarm method.

7. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the log data storage method according to any one of claims 1 to 6.

8. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the log data storage method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the log data storage method according to any one of claims 1 to 6.

Images