Log analysis method and device, storage medium and electronic equipment

Abstract

The invention discloses a log analysis method and device, a storage medium and electronic equipment. The method comprises the following steps: based on a pre-created parameter object list, obtaining at least one log file formed by message receipts provided by at least one message service party; through at least one thread which is executed in parallel, a starting position variable and an ending position variable in a current analysis task are obtained, and the starting position variable is a starting log line number determined according to an ending log line number of a previous analysis task; the initial log line number represents the line number of the first log in the next analysis task in the to-be-analyzed log file, and the end position variable is the end log line number of the current analysis task; and performing synchronous analysis on the to-be-analyzed log file according to the actual starting log line number and the actual ending log line number to obtain a target analysis result. According to the method and the device, the technical problem of relatively low efficiency in a large-scale log file analysis process is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a log analysis method and apparatus, storage medium and electronic device. Background Technology

[0002] In related technologies, the basic process for message-related business processing involves connecting to multiple message service providers (such as SMS vendors). When the business side initiates an SMS request, the service side receives the request, records it in the database, and submits the request to different vendors. For each successfully submitted SMS to a vendor, a status receipt is returned. By saving the message receipts pushed by the vendors to a log file according to the date, the status is then analyzed offline to obtain and update the receipt status of each request.

[0003] However, because message receipts for the current date (usually daily) need to be written to the log file, and the individual receipt information in the log file needs to be analyzed offline the next day, this operation method has a significant lag. Furthermore, the daily log entries can reach millions of records, increasing log analysis time and leading to inefficiencies during large-scale log file analysis.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a log analysis method and apparatus, storage medium and electronic device to at least solve the technical problem of low efficiency in the analysis of large-scale log files.

[0006] According to one aspect of the embodiments of this application, a log analysis method is provided, comprising: obtaining at least one log file composed of message receipts provided by at least one message service provider based on a pre-created list of parameter objects; obtaining a start position variable and an end position variable in the current analysis task through at least one thread executed in parallel, wherein the start position variable is a start log line number determined according to the end log line number of the previous analysis task, the start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed, the at least one log file includes the log file to be analyzed, and the end position variable is the end log line number of the current analysis task; and synchronously analyzing the log file to be analyzed according to the actual start log line number indicated by the start position variable and the actual end log line number indicated by the end position variable to obtain a target analysis result.

[0007] Optionally, the above-mentioned acquisition of the start position variable and end position variable in the current analysis task through at least one thread executed in parallel includes: in the current analysis task, repeatedly performing the following processing for each of the at least one message service provider: querying the location file corresponding to the current message service provider to obtain the target query result; if the target query result indicates the existence of a current location file corresponding to the current message service provider, reading the historical start line number and historical end line number from the previous analysis task from the current location file; and determining the current start log line number and the current end log line number in the current log file to be analyzed based on the historical start line number and the historical end line number.

[0008] Optionally, determining the current starting log line number and the current ending log line number in the current log file to be analyzed based on the historical starting line number and the historical ending line number includes: determining the historical ending line number as the initial starting log line number; determining the initial ending log line number by the sum of the initial starting log line number and the incremental step size, and determining the log line number of the last log in the current log file to be analyzed as the actual ending log line number, wherein the incremental step size is a pre-set total number of log lines in a single analysis task; and determining the current starting log line number and the current ending log line number based on the initial starting log line number, the initial ending log line number, and the actual ending log line number.

[0009] Optionally, determining the current start log line number and the current end log line number based on the initial start log line number, the initial end log line number, and the actual end log line number includes at least one of the following: if the initial start log line number is greater than the actual end log line number, the current start log line number is set to 0; if the actual end log line number is greater than the initial end log line number, the initial end log line number is set to the current end log line number; if the initial start log line number is equal to the actual end log line number, both the current start log line number and the current end log line number are set to 0; if the initial start log line number is less than the actual end log line number and the initial end log line number is less than the actual end log line number, the initial start log line number is set to the current start log line number and the initial end log line number is set to the current end log line number.

[0010] Optionally, the above method further includes: if the target query result indicates that the file at the current location does not exist, setting the current starting log line number to 0; setting the sum of the initial starting log line number and the incremental step size to the initial ending log line number, and setting the log line number of the last log in the current log file to be analyzed to the actual ending log line number; and setting the initial ending log line number to the current ending log line number if the actual ending log line number is greater than the initial ending log line number.

[0011] Optionally, the synchronous analysis of the log file to be analyzed according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable includes: when the actual starting log line number is less than the actual ending log line number, sequentially reading each log line from the actual starting log line number to the actual ending log line number to obtain a log content set, wherein the actual ending log line number is equal to the sum of the actual starting log line number and the incremental step size; extracting a portion of the content of each log line in the log content set according to preset fields, and adding the portion of the content to the log object list; and updating the log object list to the target database to complete the current analysis task.

[0012] Optionally, the above-mentioned method of obtaining at least one log file composed of message receipts provided by at least one message service provider based on a pre-created list of parameter objects includes: obtaining multi-dimensional attribute information corresponding to each message service provider among the at least one message service provider, wherein the multi-dimensional attribute information includes the service provider name, location file name, log file name, and start time; encapsulating the multi-dimensional attribute information to obtain multiple parameter objects; adding the multiple parameter objects to the parameter object list in sequence, and obtaining at least one log file and location file corresponding to each message service provider through the parameter object list.

[0013] According to another aspect of the embodiments of this application, a log analysis apparatus is also provided, comprising: a first acquisition unit, configured to acquire at least one log file composed of message receipts provided by at least one message service provider based on a pre-created list of parameter objects; a second acquisition unit, configured to acquire a start position variable and an end position variable in the current analysis task through at least one thread executed in parallel, wherein the start position variable is a start log line number determined according to the end log line number of the previous analysis task, the start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed, the at least one log file includes the log file to be analyzed, and the end position variable is the end log line number of the current analysis task; and an analysis unit, configured to synchronously analyze the log file to be analyzed according to the actual start log line number indicated by the start position variable and the actual end log line number indicated by the end position variable, to obtain a target analysis result.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program for executing the above-described log analysis method when the electronic device is run.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the log analysis method described above through the computer program.

[0017] Using the embodiments provided in this application, at least one log file composed of message receipts provided by various message service providers is obtained based on a pre-created list of parameter objects. Simultaneously, during the writing of log content into the log files, the actual start and end log line numbers in the current analysis task are accurately located according to the obtained start and end position variables and multiple threads, thereby enabling synchronous analysis of each log file. In other words, by introducing a list of parameter objects and multiple parallel threads, the traditional T+1 offline analysis mode is transformed into an online analysis mode where writing and analysis are executed synchronously. This solves the problem of excessively long processing times in traditional analysis schemes, reduces the lag in sending and responding to problem messages, and achieves the technical effect of improving the analysis efficiency of large-scale log files. Attached Figure Description

[0018] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of an optional log analysis method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of an optional log analysis method according to an embodiment of this application;

[0021] Figure 3 This is an overall architecture diagram of an optional log analysis method according to an embodiment of this application;

[0022] Figure 4 This is an overall flowchart of an optional log analysis and database entry according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of an optional log analysis device according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The technical solutions in this application will comply with legal regulations during implementation. When operating according to the technical solutions in the embodiments, the data used will not involve user privacy, ensuring that the operation process is compliant and legal while guaranteeing data security. In addition, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant regulations and standards of the relevant countries or regions.

[0028] According to one aspect of the embodiments of this application, a log analysis method is provided. As an optional implementation, the above-described log analysis method can be applied, but is not limited to, to applications such as... Figure 1 The application scenarios shown are as follows. In, for example... Figure 1In the application scenario shown, the target terminal 102 can communicate with the server 106 via network 104, but is not limited to this. The server 106 can perform operations on the database 108, such as write or read data operations. The target terminal 102 may include, but is not limited to, a human-computer interaction screen, a processor, and a memory. The human-computer interaction screen may be used to display target analysis results on the target terminal 102 for multiple log files corresponding to multiple message service providers, using the technical solution of this application. The processor may be used to respond to the human-computer interaction operations, execute corresponding operations, or generate corresponding instructions and send the generated instructions to the server 106. The memory is used to store relevant processing data, such as parameter object lists, receipt information, and start position variables.

[0029] Optionally, in this embodiment, the target terminal can be a terminal configured with a target client, which may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, etc. The target client may be a video client, instant messaging client, browser client, educational client, etc. The network may include, but is not limited to, wired network and wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that enable wireless communication. The server may be a single server, a server cluster composed of multiple servers, or a cloud server.

[0030] The technical solution presented in this application can be widely applied in the internet field, primarily for analyzing large files (such as log files), transforming offline analysis into real-time analysis. Examples include financial transaction log analysis, network monitoring and security, online advertising effectiveness evaluation, and smart factory equipment status monitoring.

[0031] As described in the above embodiments, traditional offline analysis methods are prone to low analysis efficiency when applied to large-scale log file analysis scenarios. To address this issue, this application proposes a log analysis method. Figure 2 This is a flowchart of a log analysis method according to an embodiment of this application, which includes the following steps S202 to S206.

[0032] It should be noted that the log analysis methods shown in steps S202 to S206 can be, but are not limited to, executed by electronic devices. The electronic devices can be, but are not limited to, devices such as… Figure 1 The target terminal or server shown.

[0033] Step S202: Based on a pre-created list of parameter objects, obtain at least one log file consisting of message receipts provided by at least one message service provider;

[0034] At least one log file includes receipt information received via real-time message sending.

[0035] Step S204: Obtain the start position variable and end position variable in the current analysis task through at least one thread executed in parallel. The start position variable is the start log line number determined according to the end log line number of the previous analysis task. The start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed. The at least one log file includes the log file to be analyzed. The end position variable is the end log line number of the current analysis task.

[0036] It should be noted that the above-mentioned start position variable is used to indicate the start log line number of the next analysis task, which is determined by the end log line number at the end of the previous analysis task during the continuous writing of message receipts to the log file. At least one log file includes the log file to be analyzed, and the end position variable is used to indicate the end log line number of the current analysis task, which is determined by the start position variable.

[0037] For example, suppose the first analysis starts with log line number startPos=0, increment step size=500, and ends with log line number endPos=500. This analysis task will then analyze the log from line 0 to line 499. Simultaneously, startPos=0 and endPos=500 are written to the location file. The second analysis will then cover lines 500 to 999. This process continues, analyzing 500 lines each time.

[0038] Step S206: Perform synchronous analysis on the log file to be analyzed according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable to obtain the target analysis result.

[0039] Before explaining the technical solutions in this embodiment, the technical terms involved in this application will be briefly introduced.

[0040] Message receipt: A series of feedback messages returned by the message service provider (which can also be understood as the message service provider) to the sender after a message (such as an SMS) is successfully or unsuccessfully sent to the user's terminal.

[0041] Real-time analysis: Analysis is performed as soon as the message receipt is generated, rather than waiting for all the data to accumulate to a certain scale before offline analysis.

[0042] At least one thread: the smallest unit in a computer program that can process tasks in parallel, used in this application embodiment to accelerate the analysis process of log files.

[0043] Parameter object: Contains key parameters required for the analysis process, such as log file name, analysis start line number, vendor information, etc.

[0044] Message service provider: This can be, but is not limited to, a third-party service provider designated by each operator. In this embodiment, it is mainly used to collect the message receipt status returned by the operator.

[0045] The process of obtaining at least one log file based on the parameter object list described above is the preparatory work for log analysis. The parameter object list contains relevant information for all log files to be analyzed, including but not limited to the message service provider's (hereinafter referred to as the vendor) name, location file name, target receipt file name, start time, end time, and other attributes. When the system starts up or at a preset time interval, the parameter object list is loaded, and then the corresponding real-time receipt log file—that is, at least one log file—is obtained based on the information of each vendor in the parameter object list.

[0046] After retrieving the log files, the system uses threads from a pre-created process pool to process the log files from various vendors in parallel. The start position variable is determined by the last log line number recorded at the end of the previous analysis task, and the end position variable is the next log line number after that last log line number. For example, assuming that vendor 1's log file was analyzed up to line 999 in the previous analysis task, then the starting log line number (referred to as the starting line number) for the next analysis task would be 1000. In this way, the system can ensure that each analysis continues from where it left off, avoiding duplicate analysis and missing data.

[0047] After determining the start and end positions, each log entry in the log file that is located after the actual start log line number and before the actual end log line number indicated by the start and end positions will be analyzed line by line. For example, starting from the line containing the specified actual start log line number, until the next preset end log line number.

[0048] In this embodiment, the target analysis results typically refer to key fields such as receipt status code, receipt time, and user terminal identifier (e.g., mobile phone number). After the analysis of each log line is completed, the extracted information is stored in a temporary data object. When all relevant lines of a log file have been analyzed, this temporary data object is batch-updated to the database to achieve real-time updates of SMS receipt status.

[0049] For example, if the system detects that 500 new receipt records have been added to a vendor's log file within one minute, it will immediately start analyzing these 500 records from line 1001. After the analysis is completed, the receipt status will be updated to the corresponding record in the database.

[0050] To facilitate understanding, the following will be combined with... Figure 3 The overall architecture diagram shown further explains the above log analysis method.

[0051] like Figure 3 As shown, the overall architecture for implementing the above log analysis method includes three parts: SMS (i.e. message) sending, log collection, and log analysis and storage. The following describes each part and the interaction process between them.

[0052] (a) Sending SMS messages

[0053] Step 1: The client or business side initiates a request to request the SMS service;

[0054] Step 2: Upon receiving the request, the SMS server performs basic checks, including the number of times to send, frequency control, and permissions.

[0055] Step 3: Assemble and send the content according to the template and signature;

[0056] Step 4: Store the content to be sent as a task in the message queue;

[0057] Step 5: Consumers read the tasks in the order they are received and submit them to one of the SMS providers according to a specific algorithm based on weights, while simultaneously sending details to the database.

[0058] Step 6: The manufacturer submits the sending task to the operator to complete the SMS sending;

[0059] Step 7: After the operator completes the transmission, it pushes the receipt status (which can also be understood as receipt information) to the manufacturer.

[0060] (ii) Collection of receipt documents

[0061] This can also be understood as the collection of log files containing the status information of each SMS receipt. This section mainly describes the collection of log files. Since multiple SMS providers are contracted annually, and each provider generates one receipt file per day, the number of providers determines the number of receipt files (log files) generated daily. The specific process is as follows:

[0062] Step 1: Assign each vendor an API address for push notifications, and deploy these API addresses on the same server;

[0063] Step 2: After receiving the receipt status from the operator, the manufacturer will call the pre-allocated address regardless of whether it is successful or not;

[0064] Step 3: After receiving the push request from the vendor, the log server will store the logs in files according to the vendor.

[0065] Step 4: Simultaneously, forward the logs to the specified directory on the file analysis server by vendor via rsyslog, such as / data / logs / repoert, and generate the target receipt file to be analyzed, with the filename "report-B.log", where B represents the vendor name;

[0066] Step 5: Compress the previous day's log files by date, with a filename such as "report-xxx.log-20250701.gz" for easy backtracking and to complete the collection of receipt files.

[0067] (III) Analysis and storage of receipt documents

[0068] The basic processing flow is as follows:

[0069] Step 1: Create multiple threads to execute in parallel;

[0070] Step 2: Create a list of parameter objects according to the manufacturer;

[0071] Its purpose is to integrate multi-dimensional attribute information from various manufacturers. For example, by defining a parameter object param, and adding the manufacturer name, location file name, target receipt file name, start time, and end time as attributes to the parameter object param={'report_file':report_file, 'position_file':position_file,'agent':agent,'start_day':start_day,'end_day':end_day}.

[0072] This involves encapsulating the multi-dimensional attribute information from various manufacturers into parameter objects with a standardized format, resulting in a list of parameter objects. This list of parameter objects is used to retrieve at least one log file.

[0073] It should be noted that in this embodiment, the number of parameter objects is equal to the number of preset manufacturers. That is, there are as many manufacturer parameter lists as there are manufacturers, which in turn means as many parameter objects as there are manufacturers, and consequently, as many log files as there are manufacturers.

[0074] Step 3: Obtain the start and end analysis positions;

[0075] That is, by obtaining the end log line number of the previous analysis task in real time, the starting analysis position of the current analysis task is determined based on this. The determination of the end analysis position can be referred to step 4.

[0076] Step 4: Obtain the analysis content of the current analysis task according to the starting analysis position;

[0077] The analysis includes logs from the starting line of the analysis to the ending line of the analysis.

[0078] Step 5: Perform content parsing on the obtained analysis content.

[0079] The parsing results typically include extracting key fields from each log entry, such as the receipt status code, receipt time, and user terminal identifier (e.g., mobile phone number).

[0080] After the log content is parsed, the list of log objects is updated in the database.

[0081] During the log analysis and storage process, multiple threads can be executed in parallel to obtain the starting position variable for each analysis task in real time. Based on the actual starting log line number in the current analysis task indicated by the starting position variable, the log files to be analyzed in at least one log file corresponding to at least one message service provider can be analyzed synchronously.

[0082] The reason for analyzing the log files synchronously is that, in cases involving multiple vendors, some vendors' log files may not have any new entries added that day, thus eliminating the need for analysis. Therefore, it is necessary to first identify the log files to be analyzed from at least one set of log files whose log content has been updated (incrementally reflected in the number of log line numbers). The detailed process of analyzing the log files will be described below with reference to specific embodiments.

[0083] By employing real-time log analysis and parallel processing, the feedback speed of message receipt status and data processing efficiency have been improved. Specifically, a dynamic starting position acquisition mechanism based on a parameter object list ensures that log file analysis and log content writing are executed synchronously online, enabling efficient parsing of large files. Furthermore, the multi-threaded parallel analysis design fully utilizes server resources, reducing the original T+1 offline analysis cycle (which previously required hours or even a day) to minute-level real-time processing, thus minimizing the lag in problem detection and response. This is crucial for the stability of SMS services and user experience, especially in notification SMS scenarios, enabling real-time monitoring and rapid adjustment of problematic vendors, effectively preventing SMS failures and ensuring service continuity and high efficiency.

[0084] As an optional example, the above-described method of obtaining the start and end position variables in the current analysis task through at least one thread executed in parallel includes: repeatedly performing the following process for each of the at least one message service provider in the current analysis task:

[0085] Query the location file corresponding to the current message service provider to obtain the target query results;

[0086] If the target query result indicates the existence of a current location file corresponding to the current message service provider, read the historical start line number and historical end line number from the previous analysis task from the current location file;

[0087] Based on the historical start line number and the historical end line number, determine the current start log line number and the current end log line number in the current log file to be analyzed.

[0088] Location files can be, but are not limited to, dedicated files used to record the log file analysis status of each message service provider (i.e., the message service provider), especially recording the start and end positions of the last analysis task. By querying these location files, the system can quickly locate the latest analysis point in each provider's log file, avoiding duplicate processing of already analyzed data. For example, if provider A's location file shows that the last analysis ended at line 999, then the system will start the current analysis from line 1000.

[0089] If the target query results indicate the existence of a current location file corresponding to the current message service provider, the historical start and end line numbers from the previous analysis task are read from the current location file. In other words, if the existence of a location file for this vendor is confirmed, key information is read from the location file. For example, the start and end line numbers of the previous analysis task. These line numbers are key parameters used to determine which data in the log file needs to be analyzed. For example, if the previous analysis started from line 1 and ended at line 500, then the current analysis task will start a new analysis from line 501 to obtain the latest generated receipt information.

[0090] After obtaining the log line number information from the previous analysis, this information is used to determine the starting point of the current analysis. This means that the analysis begins one position after the line number where the previous analysis ended, ensuring the continuity of the analysis. For example, if the previous ending line number was 500, then the starting line number for this analysis will be set to 501.

[0091] In addition, considering that log files may be truncated or reset due to server restarts or other reasons, the system also needs to have intelligent judgment capabilities. When it finds that the current file line number is less than the historical end line number, it should automatically reset the starting line number to 0 to ensure that no data is missed.

[0092] In an optional embodiment, the above-described method of obtaining the start and end position variables in the current analysis task through at least one thread executed in parallel can also be as follows: For each of the at least one message service providers, the following process is repeatedly performed to obtain the current start log line number and current end log line number in the log file to be analyzed corresponding to each message service provider in the current analysis task. The log file to be analyzed is a log file whose content has been incrementally updated within at least one log file within the current date: Query the position file corresponding to the current message service provider to obtain the target query result; if the target query result indicates the existence of a current position file corresponding to the current message service provider, read the historical start line number and historical end line number from the previous analysis task from the current position file; based on the historical start line number and historical end line number, determine the current start log line number and current end log line number in the current log file to be analyzed corresponding to the current message service provider.

[0093] The technical solution in this embodiment ensures that the system can efficiently and accurately locate the key to newly added log records. First, a list of parameter objects containing information on all message service providers is loaded. Then, a thread is allocated to each provider for parallel processing. Each thread can query whether a location file for the corresponding provider exists. If a location file exists, the end line number of the previous analysis is obtained, and the log file is read from that line number until the next preset end point (usually the current line number plus a fixed number of lines).

[0094] By using the above methods, the system can ensure that each thread only processes data added since the last analysis, avoiding duplicate processing of already analyzed data. At the same time, it can also promptly capture any line number resets caused by log file rotation, ensuring the integrity of the analysis.

[0095] As an optional implementation, the above method of determining the current start log line number and the current end log line number in the current log file to be analyzed based on the historical start line number and the historical end line number includes:

[0096] Set the end line number of the historical log as the initial start line number;

[0097] The initial start log line number is determined by the sum of the initial start log line number and the incremental step size, and the log line number of the last log in the current log file to be analyzed is determined as the actual end log line number. The incremental step size is the total number of log lines in a single analysis task that is set in advance.

[0098] The current start log line number and the current end log line number are determined based on the initial start log line number, the initial end log line number, and the actual end log line number.

[0099] When starting a new round of analysis, it will begin from the end position of the previous round of analysis. For example, if the last analysis of the log file provided by vendor A ended at line 1200, then the initial starting log line number for this analysis will be line 1201, ensuring that the system can continue the analysis from the last stopped position.

[0100] The initial and final log line numbers are determined by a preset increment step size (e.g., processing 500 lines of log data per analysis task). Continuing with the example above, if the increment step size is 500 lines and the initial starting log line number is 1201, then the initial ending log line number is set to 1700. The system then checks the actual length of the log file, i.e., the last log line number of the file currently being analyzed, to ensure that the actual file boundary is not exceeded. For example, if the file actually only has 1600 lines, then the actual ending log line number will be updated to 1600, avoiding unnecessary processing overhead.

[0101] Finally, the system calculates the range of log line numbers to be processed in this analysis task based on the actual file length and the preset increment step size. For example, if the initial starting log line number is 1201 and the preset initial ending log line number is 1700, but the actual file length is only 1600 lines, the system will adjust the current ending log line number for this analysis to 1600 lines, thus ensuring that each analysis neither exceeds the boundary nor leaves unprocessed data. This mechanism ensures that even as the log file continues to grow, the system can adjust its analysis strategy in a timely manner, maintaining efficient data processing capabilities.

[0102] The core of this embodiment lies in dynamically adjusting the start and end line numbers of the analysis task to better reflect the actual state of the current log file. At the start of each analysis task, the initial starting log line number is determined, continuing analysis from the end log line number of the previous analysis. Next, the initial ending log line number is calculated based on a preset incremental step size, while simultaneously reading the actual length of the current log file to avoid exceeding file boundaries. Finally, based on the actual file length, the precise start and end range of this analysis task is determined. This not only ensures the continuity and completeness of the analysis but also improves the real-time performance and efficiency of data processing. For example, when the file size grows rapidly, by flexibly adjusting the analysis range, the system can capture and process the latest generated receipt information at minute intervals, updating the database instantly and providing real-time data support for business decisions, effectively reducing the potential risks caused by information lag.

[0103] As an optional example, determining the current start log line number and the current end log line number based on the initial start log line number, the initial end log line number, and the actual end log line number includes at least one of the following:

[0104] If the initial start log line number is greater than the actual end log line number, the current start log line number is set to 0; and if the actual end log line number is greater than the initial end log line number, the initial end log line number is set to the current end log line number.

[0105] If the initial start log line number is equal to the actual end log line number, both the current start log line number and the current end log line number will be set to 0.

[0106] If the initial start log line number is less than the actual end log line number, and the initial end log line number is less than the actual end log line number, then the initial start log line number will be determined as the current start log line number, and the initial end log line number will be determined as the current end log line number.

[0107] like Figure 4 The process for obtaining the starting position of the analysis, as shown below, is as follows:

[0108] S41. If a position_file exists, it indicates that this is not the first analysis. Read the file directly to obtain the start position lastStartPos and end position lastEndPos of the previous analysis task.

[0109] S42, set the new start position to the previous end position startPos=lastEndPos, the new end position to the current total number of lines endPos=total, and the new temporary end position tmpPos=startPos+pageSize;

[0110] S43, determine whether startPos > endPos;

[0111] If so, set startPos=0 (this occurs during log rotation; in this scenario, log files are compressed daily, and after compression, they restart the writing mode for the next day, so endPos is very small or 0).

[0112] S44, determine if endPos > tmpPos;

[0113] If so, then endPos=tmpPos.

[0114] S45, write the new start position and the new end position to overwrite the original position file, and return to the start position, and jump to step S61 in the following embodiment.

[0115] S46. If startPos=endPos, then set startPos=0 and endPos=0, return to the starting position, and jump to step S61 (this happens because the log file is constantly increasing; if they are equal, it means that no new logs have been added during this period, and no analysis is needed).

[0116] In step S61, if startPos=0 and endPos=0, it means that no new files have been added, and the analysis ends.

[0117] Otherwise, further determine whether startPos <= endPos; if so, read the content of the startPos line from the target log file and parse the content of the current line.

[0118] As can be seen in an optional embodiment, when the last line number of the current log file to be analyzed (i.e., the actual end log line number) is detected to be less than the line number recorded at the end of the previous analysis (i.e., the historical end line number), it means that the log file may have undergone rotation (the transition time from today to tomorrow, such as early morning) or cleanup operations, resulting in the removal of historical data. In this case, the log file is actually starting from the beginning. In this situation, the current starting log line number should be reset to 0 to ensure that the system re-analyzes from the beginning of the file. For example, if the end log line number of the previous analysis was 1500, and in the new analysis, due to the normal rotation of the log file, the latest log line number of the file becomes 500, the system will set the current starting log line number to 0 and start the analysis from the first line of the file.

[0119] Meanwhile, to ensure the completeness and accuracy of log file analysis, if the actual end log line number is greater than the preset initial end log line number, the initial end log line number remains unchanged, and the excess increment will be allocated to the next analysis task for analysis until all logs are analyzed. The purpose is to ensure that even if the log file length exceeds the preset increment step, the system can still completely analyze the file to the end, preventing the omission of important data.

[0120] In another alternative embodiment, assuming that in an analysis task, the initial start log line number and the actual end log line number are exactly the same, it indicates that no new records have been added to the current log file since the last analysis. In this special case, to avoid unnecessary analysis work, both the current start log line number and the current end log line number are set to 0. This is equivalent to telling the system that no analysis operation needs to be performed at this time, and it can directly proceed to the next round of checks, saving computing resources.

[0121] In another optional embodiment, the system handles the situation where the initial start log line number is less than the actual end log line number, and the initial end log line number is less than the actual end log line number. This is a common scenario, indicating that there are new lines in the current log file that need to be analyzed. The system determines the start and end line numbers of the current analysis based on the end line number of the previous analysis (i.e., the historical end line number) and the preset increment step size. For example, if the historical end line number is 1000 after the previous analysis, the preset increment step size is 500 lines, and the latest line number of the current log file is 1500, then the system will start the analysis from line 1001 and continue until line 1500. This approach ensures the continuity of the analysis while effectively utilizing computing resources and avoiding overprocessing.

[0122] The above processing logic enhances the system's adaptability and robustness, enabling it to effectively handle various dynamic changes in log files, including but not limited to date rotations in file content and sudden surges in data. It ensures the efficiency and accuracy of real-time analysis, reduces analysis latency, and provides technical support for services requiring high-frequency, high-precision monitoring, such as SMS services. Simultaneously, by flexibly adjusting the analysis scope, unnecessary computational overhead is avoided while ensuring comprehensive coverage of log information.

[0123] As an optional example, the above method also includes:

[0124] If the target query results indicate that the file at the current location does not exist, the current starting log line number will be set to 0;

[0125] The initial start log line number is determined by the sum of the initial start log line number and the incremental step size, and the log line number of the last log in the current log file to be analyzed is determined as the actual end log line number.

[0126] If the actual end log line number is greater than the initial end log line number, the initial end log line number will be used as the current end log line number.

[0127] If a location file corresponding to a particular manufacturer exists, and the target query result indicates that the location file does not exist, then by... Figure 4 The branch flow following the "NOT" result of the "Does a location file exist?" judgment condition determines the current start log line number and the current end log line number. The specific flow is as follows:

[0128] S51, determine whether the current manufacturer's position_file exists;

[0129] If it does not exist, proceed to step S52; otherwise, proceed to step S41 in the above embodiment.

[0130] It should be noted that if the location file does not exist, it means that the current analysis task is the first analysis of the day, and the start position startPos is set to 0.

[0131] S52, set the current starting log line number startPos=0;

[0132] S53, read the total number of lines in the log file to be analyzed (total), and the actual end position (i.e., the actual end log line number) endPos=total (the total number of log lines in the current log file to be analyzed).

[0133] S54 sets the temporary end position (i.e., the initial end log line number) tmpPos=startPos+size, where size represents the number of lines analyzed in a single run (which can also be understood as the incremental step size), and the default value is 500.

[0134] S55, determine if endPos > tmpPos;

[0135] If so, set endPos=tmpPos;

[0136] S56, create a position file position_file, and write the start position (current starting log line number) and end position (current ending log line number) into the position file, a total of 2 lines, return the start position and end position, and jump to step S61 in the following embodiment.

[0137] As can be seen, when no matching location file is found for a specific messaging service provider, it means that no analysis of that provider's log files has been performed. To ensure complete data coverage, the current starting log line number is initialized to 0.

[0138] After determining the starting log line number, the ending log line number for this analysis needs to be set. This can be obtained by adding the initial starting log line number to the preset increment step size. Simultaneously, the actual length of the log file to be analyzed should be considered to ensure the analysis range does not exceed the limits. For example, if the increment step size is set to 500 lines, and the starting line number is 0 for the first analysis, then the initial ending log line number will be 500. The system also needs to read the last log line number of the file, i.e., the actual ending log line number, to verify the reasonableness of the analysis range. If the actual length of the log file is 600 lines, then the analysis range will cover the first 500 lines of the file.

[0139] In some cases, the log file length may exceed the preset increment step. In this case, to ensure that all newly added logs are analyzed, the current end log line number for this analysis should be adjusted according to the actual length of the log file to be analyzed.

[0140] This embodiment mainly describes the specific implementation method of determining the starting and ending log line numbers when no location file exists. By initializing the starting line number to 0 and flexibly setting and adjusting the ending line number, it ensures comprehensive coverage and efficiency optimization of log analysis, improves the adaptive capability of the log analysis system, constructs a complete and dynamic log analysis mechanism, and realizes efficient real-time processing of large file logs.

[0141] As an optional example, the above synchronous analysis of the log file to be analyzed, based on the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable, includes:

[0142] If the actual starting log line number is less than the actual ending log line number, read each log line from the actual starting log line number to the actual ending log line number in sequence to obtain a log content set, where the actual ending log line number is equal to the sum of the actual starting log line number and the incremental step size;

[0143] Extract a portion of each log entry from the log content collection according to preset fields, and add the portion of the content to the log object list;

[0144] The current analysis task is completed by updating the list of log objects to the target database.

[0145] If, in accordance with the above method, it is determined that a vendor has a log file to be analyzed in the current analysis task, and the actual start log line number and the actual end log line number of the analysis have been obtained, the log portion covered in the current analysis task can be analyzed through, but is not limited to, the following process.

[0146] S61. If startPos=0 and endPos=0, it means that no new files have been added. Jump to step S68 and end the analysis. Otherwise, execute step S62.

[0147] S62, determine if startPos <= endPos;

[0148] If so, proceed to step S63;

[0149] S63, Read the contents of line startPos from the log file to be analyzed;

[0150] S64, parse the content of the current line, extract the mobile phone number, message ID, receipt time and receipt status code, and add them to the data object data;

[0151] Among them, the data object can be understood as the log object.

[0152] S65, Add the log object to the log object list;

[0153] S66, set startPos=startPos+1, jump to step S63, read the next line of content, until all the lines set in this step have been read, parsed and added to the log object list list.

[0154] S67, batch update the log object list data to the database;

[0155] S68, End the current analysis.

[0156] It should be noted that during the execution of the current analysis task, the start position (actual start log line number) and end position (actual end log line number) of the current analysis are obtained in advance. The end position is predetermined according to the method in the above embodiment. Therefore, the end position may be equal to the sum of the actual start log line number and the increment step size, or it may be equal to the total number of log lines in the file (in which case, it is the last analysis, but if it is less than the increment step size of 500, the total number of lines in the log file to be analyzed is used as the end position).

[0157] After the log object list is constructed, a batch update operation will be performed to synchronize all log object information in the list to the target database, thereby completing the analysis and storage of newly added log lines. For example, if the log object list contains 500 log objects, each containing information such as a phone number, message ID, and receipt status code, the system will update all 500 records to the database at once.

[0158] This embodiment achieves efficient reading and processing of the dynamic range of log files, enabling a shift from T+1 (logs collected on the first day, analyzed offline on the second day) to T+0 (logs collected and analyzed simultaneously on the same day). This achieves efficient and real-time processing of log files, significantly improving information feedback speed and business processing capabilities.

[0159] As an optional example, the above method, based on a pre-created list of parameter objects, retrieves at least one log file consisting of message receipts provided by at least one message service provider, including:

[0160] Obtain multi-dimensional attribute information for each message service provider in at least one message service provider, wherein the multi-dimensional attribute information includes the service provider name, location file name, log file name, and start time;

[0161] By encapsulating multidimensional attribute information, multiple parameter objects are obtained;

[0162] Multiple parameter objects are added to the parameter object list in sequence, and at least one log file and location file corresponding to each message service provider are obtained through the parameter object list.

[0163] In this embodiment, each SMS message sent will be asynchronously inserted into the database, including the mobile phone number, sending time, message ID, and other information. If it is sent in batches, the message IDs of the same batch are the same. The message ID + mobile phone number uniquely identifies a sending record. The receipt time and receipt status code are updated after the receipt file is analyzed. At the same time, the receipt file is continuously written, even up to tens of millions of lines. The core of real-time analysis lies in how to implement batch small-scale analysis, record the starting line number of the current analysis each time, and involve multiple service providers to use parallel cleaning to improve efficiency.

[0164] To achieve the above process, the multidimensional attribute information of each message service provider needs to be preprocessed. This mainly includes obtaining the multidimensional attribute information of multiple message service providers according to a unified standard, and then creating parameter objects on a per-message-service-provided basis. Through these parameter objects, the multidimensional attribute information of each message service provider can be queried and obtained.

[0165] In other words, by integrating the acquired attribute information, a list of parameter objects is obtained, and this list is used as the basis for obtaining at least one log file and determining the existence of a location file during each analysis. The specific process is as follows:

[0166] S71, create a process pool pool=multiprocessing.Pool(4) to execute tasks in parallel (usually consistent with the number of CPU cores on the server).

[0167] S72, get the start time, the current time and push it forward 1 day to get the end time, the current time and push it forward 4 days to get the start time (used for the time limit when updating data, real-time analysis is generally the data of the day, the end time is the early morning of the next day, the receipt may be delayed by 72 hours, so the start time is pushed forward 4 days to supplement the receipt that was not received before the update).

[0168] S73, iterate through the list of manufacturers;

[0169] S74, query position file name position_file in batches by manufacturer (used to record the starting position of the row in the last analysis, file name (e.g. / data / logs / position / position-xxx.log-20250801).

[0170] S75, define the parameter object param;

[0171] Depending on the manufacturer, the manufacturer name, location file name, target receipt file name, start time, and end time are added as attributes to the parameter object, resulting in param={'report_file':report_file, 'position_file':position_file,'agent':agent,'start_day':start_day,'end_day':end_day}.

[0172] S76, Define a parameter object list paramList, add the parameter objects assembled in step S75 to the parameter list, paramList.append(param);

[0173] S77, Repeat step S74 until all manufacturers have been traversed, and add the parameters corresponding to each manufacturer to paramList;

[0174] S78 distributes tasks to multiple processes in the process pool for parallel execution, completing the current analysis.

[0175] The specific steps for executing a single task for a message service provider can be found in steps S41-S46, S51-S56, and S61-S68 of the above embodiments. They will not be repeated here.

[0176] As described above, when real-time analysis of log files from different message service providers is required, the primary task is to collect multi-dimensional attribute information related to these service providers. This information includes, but is not limited to, the service provider's name, the filename used to record the analysis location, the filename of the log file to be analyzed, and the start time used to limit the analysis time range.

[0177] After obtaining the aforementioned multi-dimensional attribute information, the next task is to encapsulate this information into parameter objects. Each parameter object will correspond to a message service provider, containing key attributes such as the service provider's name, location file name, log file name, and start time. Taking service provider A as an example, a parameter object will be created, stored in dictionary form. The purpose is to customize personalized analysis tasks for each service provider, ensuring the relevance and efficiency of the analysis process.

[0178] After encapsulating each parameter object, these parameter objects are sequentially added to a parameter object list, forming a unified data processing entry point. This list aggregates the analysis needs of all message service providers, enabling subsequent parallel processing or batch analysis.

[0179] Once the list of parameter objects is constructed, the corresponding log file and location file can be efficiently retrieved based on the index of each parameter object in the list. This process essentially maps the encapsulated multidimensional attribute information to specific file paths, providing accurate data sources and status information for subsequent log analysis.

[0180] Therefore, the construction of the parameter object list enables the system to simultaneously analyze log files from different service providers using parallel processing technology, further shortening the overall analysis time and making real-time monitoring feasible. By calling the location file, the analysis progress of each service provider can be accurately obtained, avoiding the problems of duplicate analysis or missing data, and ensuring the efficiency and completeness of the analysis process.

[0181] To more clearly understand the implementation process of log file collection, file analysis, and database storage described above, the following section will combine... Figure 4 The overall flowchart for log analysis and database entry shown below provides a further description of it.

[0182] Step 1: Create a process pool pool=multiprocessing.Pool(4) to execute tasks in parallel (usually consistent with the number of CPU cores on the server).

[0183] Step 2: Obtain the start time, the current time plus one day to indicate the end time, and the current time plus four days to indicate the start time (this is used to limit the time when updating data; real-time analysis is generally based on the data of the current day, and the end time is the early morning of the next day. The receipt may be delayed by 72 hours, so the start time is pushed back four days to supplement data for those who did not receive a receipt before the update).

[0184] Step 3: Iterate through the list of manufacturers;

[0185] Step 4: Query the position file name position_file in batches by manufacturer (used to record the starting position of the row in the previous analysis, file name);

[0186] Step 5, define the parameter object param;

[0187] Depending on the manufacturer, the manufacturer name, location file name, target receipt file name, start time, and end time are added as attributes to the parameter object, resulting in param={'report_file':report_file, 'position_file':position_file,'agent':agent,'start_day':start_day,'end_day':end_day}.

[0188] Step 6: Define a parameter object list paramList, and add the parameter object assembled in step S75 to the parameter list, paramList.append(param);

[0189] Step 7: Repeat step S74 until all manufacturers have been traversed, and the parameters corresponding to each manufacturer are added to paramList;

[0190] Step 8: Distribute the tasks to multiple processes in the process pool for parallel execution. The execution steps for a single task are as follows:

[0191] S81, determine if the current manufacturer's position_file exists;

[0192] If it does not exist, proceed to step S82; otherwise, proceed to step S87 in the above embodiment.

[0193] It should be noted that if the location file does not exist, it means that the current analysis task is the first analysis of the day, and the start position startPos is set to 0.

[0194] S82, set the current starting log line number startPos=0;

[0195] S83, read the total number of lines in the log file to be analyzed (total), and the actual end position (i.e., the actual end log line number) endPos=total (the total number of log lines in the current log file to be analyzed).

[0196] S84 sets the temporary end position (i.e., the initial end log line number) tmpPos=startPos+size, where size represents the number of lines analyzed in a single run (which can also be understood as the incremental step size), and the default value is 500.

[0197] S85, determine if endPos > tmpPos;

[0198] If so, set endPos=tmpPos;

[0199] S86, create a position file position_file, and write the start position (current starting log line number) and end position (current ending log line number) into the position file, a total of 2 lines, return to the start position, and jump to step S813.

[0200] S87. If a position_file exists, it indicates that this is not the first analysis. Read the file directly to obtain the start position lastStartPos and end position lastEndPos of the previous analysis task.

[0201] S88, set the new start position to the previous end position startPos=lastEndPos, the new end position to the current total number of lines endPos=total, and the new temporary end position tmpPos=startPos+pageSize;

[0202] S89, determine if startPos > endPos;

[0203] If so, set startPos=0 (this occurs during log rotation; in this scenario, log files are compressed daily, and after compression, they restart the writing mode for the next day, so endPos is very small or 0).

[0204] S810, determine if endPos > tmpPos;

[0205] If so, then endPos=tmpPos.

[0206] S811: Write the new start position and the new end position to the file and overwrite the original position. At the same time, return to the start position and jump to step S813.

[0207] S812, if startPos=endPos, then set startPos=0 and endPos=0, return to the starting position, and jump to step S813 (this happens because the log file is constantly increasing; if they are equal, it means that no new logs have been added during this period, and no analysis is needed).

[0208] After obtaining the start and end positions of the current analysis through the above steps, perform the following steps to conduct the analysis.

[0209] S813, if startPos=0 and endPos=0, it means that no new files have been added. Jump to step S820 and end the analysis.

[0210] Otherwise, proceed to step S814.

[0211] S814, determine if startPos <= endPos;

[0212] If yes, proceed to step S815; otherwise, proceed to step S819.

[0213] S815, Read the contents of line startPos from the log file to be analyzed;

[0214] S816: Parse the content of the current line, extract the mobile phone number, message ID, receipt time and receipt status code, and add them to the data object data;

[0215] Among them, the data object can be understood as the log object.

[0216] S817, Add the log object to the log object list;

[0217] S818, set startPos=startPos+1, jump to step S815, read the next line of content, until all the lines set in this step have been read, parsed and added to the log object list list.

[0218] S819, batch update the log object list data to the database;

[0219] S820, end the current analysis.

[0220] Based on the above analysis process, the delivery receipts from various vendors can be analyzed in real time at a 1-minute granularity. This significantly improves efficiency compared to the traditional T+1 offline analysis. This improvement brings a qualitative leap to the stability of SMS sending, such as notifications, which do not require user feedback. Previously, it was only possible to obtain the previous day's sending status the next day. If a vendor had a problem, users affected by that vendor's messages would not receive them. Now, because the sending status of a vendor can be known at the minute level, it can be checked every 5 minutes. If a vendor experiences a large number of delivery receipt failures, an alarm can be triggered, and that vendor can be manually or automatically disconnected, and traffic can be allocated to other vendors.

[0221] As an optional example, the above method also includes:

[0222] Get message receipts based on the receipt push interface address;

[0223] Message receipts are saved sequentially to at least one log file corresponding to at least one message service provider.

[0224] Combination Figure 3 As can be seen, when a message service provider receives a send request, it interacts with the operator to complete the message sending and obtain the receipt status. According to the technical solution in this application, these receipt statuses are pre-assigned to the receipt push interface address of each service provider and pushed to the log collection server in real time. For example, the receipt from SMS provider A will be pushed to the interface address http: / / sms.log.fyh.com / reportA. This address is already bound to provider A's SMS service during the design phase, ensuring that the receipt information can be collected accurately.

[0225] The collected message receipts need to be saved to the corresponding log file of the service provider for subsequent analysis and processing. In this embodiment, the receipt information of SMS provider A will be saved to the log file report_A.log, which is located in a specific directory of the file analysis server, such as / data / logs / repoert. In this way, the system can store receipt information separately for each service provider, facilitating subsequent real-time analysis and fault location. For example, if provider A reports a large number of sending failures in the receipts on a certain day, this information will be recorded in the report_A.log file until the real-time analysis task reads and processes it. The overall implementation process is as follows:

[0226] (1) First, after receiving the sending request, the SMS provider completes the SMS sending through the operator and obtains the receipt status. These receipt statuses are pushed to the log collection server in real time through their respective service interfaces;

[0227] (2) Upon receiving the receipt information, the log collection server is responsible for saving it to the corresponding log file. This saving process is continuous and real-time, ensuring that all receipt information can be accurately recorded, thus preparing the raw data for subsequent real-time analysis;

[0228] (3) After saving the receipt information, the system will enter the real-time analysis stage.

[0229] As described in the above embodiments, the system can quickly locate the current analysis position of each log file, read and process newly added receipt information. This process is executed in parallel by a pre-created process pool, and by continuously adjusting the start and end line numbers of the analysis, it ensures that all newly generated receipt information can be analyzed and stored in a timely manner.

[0230] To more clearly understand the log analysis methods described above, the following section will combine... Figure 3 The overall architecture diagram shown and Figure 4 The log analysis and database entry flowchart shown below provides a further description of the process.

[0231] In an optional embodiment, the technical solution of this application is mainly used for SMS services, for the analysis of receipt log files from various vendors. The analysis is changed from the original T+1 analysis of receipt files to real-time analysis (executed once per minute), reducing the lag in problem localization. Based on the real-time receipt status of SMS messages, the SMS service can switch to the problematic vendor, reducing SMS failures, as detailed below:

[0232] Step 1: The user submits a request to send an SMS message;

[0233] Step 2: After receiving the request, the server performs pre-processing and assembly before writing it to the message queue.

[0234] Step 3: After receiving the task, the consumer randomly selects an SMS provider to send the SMS message based on the weight.

[0235] Step 4: After receiving the request, the manufacturer submits it to the operator, who then sends the SMS message.

[0236] Step 5: After the operator completes the SMS sending, it will return a receipt based on the sending status.

[0237] Step 6: After receiving the receipt, the vendor pushes it to the log server;

[0238] Step 7: After receiving the request, the log server uses rsyslog to forward the logs to the specified directory on the file analysis server according to the vendor, thus completing the log collection.

[0239] Step 8: Begin log analysis, starting with assembling the parameters;

[0240] Step 9: Obtain the starting position; analyze only the content within the starting position range of the file each time.

[0241] Step 10: After parsing the content, update the database in batches to complete the current log analysis;

[0242] Step 11: For each manufacturer, repeat the actions in steps 8-10 above, once every minute, in parallel processing;

[0243] Step 12: Since the sending status code of the vendor can be obtained in real time, if a vendor has a problem, the server can make a timely judgment to remove the vendor, so as to ensure that the SMS service is not affected.

[0244] The log analysis and processing method and apparatus proposed in this application embodiment change the extraction of SMS receipt status from T+1 to real-time. Efficiency is improved by 100%. Based on the real-time receipt status of SMS messages, the SMS service can switch to problematic providers, reducing SMS failures.

[0245] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0246] According to another aspect of the embodiments of this application, as follows is also provided Figure 5 A log analysis device is shown, the device comprising:

[0247] The first acquisition unit 502 is used to acquire at least one log file consisting of message receipts provided by at least one message service provider, based on a pre-created list of parameter objects.

[0248] The second acquisition unit 504 is used to acquire the start position variable and the end position variable in the current analysis task through at least one thread executed in parallel. The start position variable is the start log line number determined according to the end log line number of the previous analysis task. The start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed. The at least one log file includes the log file to be analyzed. The end position variable is the end log line number of the current analysis task.

[0249] Analysis unit 506 is used to synchronously analyze the log file to be analyzed according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable, and obtain the target analysis result.

[0250] Optionally, the second acquisition unit 504 mentioned above includes:

[0251] The first processing module is configured to repeatedly perform the following processing on each of the at least one message service providers in the current analysis task: query the location file corresponding to the current message service provider to obtain the target query result; if the target query result indicates the existence of a current location file corresponding to the current message service provider, read the historical start line number and historical end line number from the previous analysis task from the current location file; and determine the current start log line number and the current end log line number in the current log file to be analyzed based on the historical start line number and the historical end line number.

[0252] Optionally, the first processing module mentioned above includes:

[0253] The first processing submodule is used to determine the historical end line number as the initial start line number of the log;

[0254] The second processing submodule is used to determine the initial end log line number by the sum of the initial start log line number and the incremental step size, and to determine the log line number of the last log in the current log file to be analyzed as the actual end log line number. The incremental step size is the total number of log lines in a single analysis task that is preset.

[0255] The third processing submodule is used to determine the current start log line number and the current end log line number based on the initial start log line number, the initial end log line number, and the actual end log line number.

[0256] Optionally, the second processing module described above further includes:

[0257] The fourth processing submodule is used to perform at least one of the following:

[0258] If the initial start log line number is greater than the actual end log line number, the current start log line number is set to 0; and if the actual end log line number is greater than the initial end log line number, the initial end log line number is set to the current end log line number.

[0259] If the initial start log line number is equal to the actual end log line number, both the current start log line number and the current end log line number will be set to 0.

[0260] If the initial start log line number is less than the actual end log line number, and the initial end log line number is less than the actual end log line number, then the initial start log line number will be determined as the current start log line number, and the initial end log line number will be determined as the current end log line number.

[0261] Optionally, the above-mentioned device further includes:

[0262] The first processing unit is used to determine the current starting log line number as 0 when the target query result indicates that the file at the current location does not exist;

[0263] The second processing unit is used to determine the initial end log line number by summing the initial start log line number and the incremental step size, and to determine the log line number of the last log in the current log file to be analyzed as the actual end log line number.

[0264] The third processing unit is used to determine the initial end log line number as the current end log line number when the actual end log line number is greater than the initial end log line number.

[0265] Optionally, the analysis unit 506 includes:

[0266] The reading module is used to read each log line from the actual starting log line number to the actual ending log line number in sequence when the actual starting log line number is less than the actual ending log line number, and obtain a set of log content. The actual ending log line number is equal to the sum of the actual starting log line number and the incremental step size.

[0267] The module adds a portion of the content of each log entry in the log content collection according to preset fields and adds the portion of the content to the log object list.

[0268] The update module is used to complete the current analysis task by updating the list of log objects to the target database.

[0269] Optionally, the first acquisition unit 502 mentioned above includes:

[0270] The first acquisition module is used to acquire multi-dimensional attribute information corresponding to each message service provider in at least one message service provider, wherein the multi-dimensional attribute information includes the service provider name, location file name, log file name and start time;

[0271] The encapsulation module is used to encapsulate multidimensional attribute information to obtain multiple parameter objects;

[0272] The second acquisition module is used to add multiple parameter objects to the parameter object list in sequence, and to obtain at least one log file and location file corresponding to each message service provider through the parameter object list.

[0273] It should be noted that the embodiments of the log analysis device described here can refer to the embodiments of the log analysis method described above, and will not be repeated here.

[0274] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described log analysis method is also provided. This electronic device may be... Figure 1 The target terminal or server is shown. This embodiment uses the electronic device as an example to illustrate the concept. Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps in any of the above method embodiments via the computer program.

[0275] Optionally, the aforementioned electronic device may be located in at least one of a plurality of network devices of the computer.

[0276] Optionally, the processor described above can be configured to perform the following steps via a computer program:

[0277] S1, based on a pre-created list of parameter objects, obtain at least one log file consisting of message receipts provided by at least one message service provider;

[0278] S2, by using at least one thread executed in parallel, obtain the start position variable and end position variable in the current analysis task, wherein the start position variable is the start log line number determined according to the end log line number of the previous analysis task, the start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed, the at least one log file includes the log file to be analyzed, and the end position variable is the end log line number of the current analysis task;

[0279] S3, according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable, performs synchronous analysis on the log file to be analyzed to obtain the target analysis results.

[0280] Alternatively, as those skilled in the art will understand, Figure 6 The structure shown is for illustrative purposes only. Figure 6 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 6 The different configurations shown.

[0281] The memory 602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the log analysis method and apparatus in this embodiment. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, thereby realizing the aforementioned log analysis method. The memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include memory remotely located relative to the processor 604, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 602 may be used, but is not limited to, to store parameter object lists, starting log line numbers, and target analysis results, etc. As an example, such as... Figure 6 As shown, the memory 802 may include, but is not limited to, the first acquisition unit 502, the second acquisition unit 504, and the analysis unit 506 in the log analysis device. Furthermore, it may include, but is not limited to, other module units in the log analysis device, which will not be elaborated upon in this example.

[0282] Optionally, the transmission device 606 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 606 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 606 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0283] In addition, the above-mentioned electronic device also includes: a display 608 for displaying target analysis results; and a connection bus 610 for connecting the various module components in the above-mentioned electronic device.

[0284] In other embodiments, the target terminal or server described above can be a node in a distributed system. This distributed system can be a blockchain system, formed by connecting multiple nodes through network communication. The nodes can form a point-to-point network, and any type of computing device, such as a server or target terminal, can become a node in the blockchain system by joining this point-to-point network.

[0285] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the log analysis method provided in various optional implementations of the above-described server verification processing, wherein the computer program is configured to execute the steps in any of the above-described method embodiments at runtime.

[0286] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0287] S1, based on a pre-created list of parameter objects, obtain at least one log file consisting of message receipts provided by at least one message service provider;

[0288] S2, by using at least one thread executed in parallel, obtain the start position variable and end position variable in the current analysis task, wherein the start position variable is the start log line number determined according to the end log line number of the previous analysis task, the start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed, the at least one log file includes the log file to be analyzed, and the end position variable is the end log line number of the current analysis task;

[0289] S3, according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable, performs synchronous analysis on the log file to be analyzed to obtain the target analysis results.

[0290] Optionally, in embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0291] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the target terminal. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0292] The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods in the various embodiments of this application.

[0293] In the above embodiments of this application, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed client can be implemented in other ways in the several embodiments provided in this application. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0294] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0295] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A log analysis method, characterized in that, include: Based on a pre-created list of parameter objects, obtain at least one log file consisting of message receipts provided by at least one message service provider; By using at least one thread executed in parallel, the start position variable and end position variable of the current analysis task are obtained. The start position variable is the start log line number determined based on the end log line number of the previous analysis task. The start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed. The at least one log file includes the log file to be analyzed. The end position variable is the end log line number of the current analysis task. The log file to be analyzed is synchronously analyzed according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable to obtain the target analysis result.

2. The method according to claim 1, characterized in that, The step of obtaining the start and end position variables of the current analysis task through at least one thread executed in parallel includes: In the current analysis task, the following process is repeatedly performed on each of the at least one message service provider: Query the location file corresponding to the current message service provider to obtain the target query results; If the target query result indicates the existence of a current location file corresponding to the current message service provider, read the historical start line number and historical end line number from the previous analysis task from the current location file; Based on the historical start line number and the historical end line number, determine the current start log line number and the current end log line number in the current log file to be analyzed.

3. The method according to claim 2, characterized in that, The step of determining the current start log line number and the current end log line number in the current log file to be analyzed based on the historical start line number and the historical end line number includes: The historical end line number is determined as the initial start log line number; The initial start log line number and the incremental step size are used to determine the initial end log line number, and the log line number of the last log in the current log file to be analyzed is used to determine the actual end log line number. The incremental step size is the total number of log lines in a single analysis task that is set in advance. Based on the initial start log line number, the initial end log line number, and the actual end log line number, the current start log line number and the current end log line number are determined.

4. The method according to claim 3, characterized in that, The determination of the current start log line number and the current end log line number based on the initial start log line number, the initial end log line number, and the actual end log line number includes at least one of the following: If the initial start log line number is greater than the actual end log line number, the current start log line number is set to 0; and if the actual end log line number is greater than the initial end log line number, the initial end log line number is set to the current end log line number. If the initial start log line number is equal to the actual end log line number, both the current start log line number and the current end log line number are set to 0. If the initial start log line number is less than the actual end log line number, and the initial end log line number is less than the actual end log line number, then the initial start log line number is determined as the current start log line number, and the initial end log line number is determined as the current end log line number.

5. The method according to claim 2, characterized in that, The method further includes: If the target query result indicates that the file at the current location does not exist, the current starting log line number is set to 0; The initial start log line number and the incremental step size are used to determine the initial end log line number, and the log line number of the last log in the current log file to be analyzed is used to determine the actual end log line number. If the actual end log line number is greater than the initial end log line number, the initial end log line number is determined as the current end log line number.

6. The method according to claim 1, characterized in that, The synchronous analysis of the log file to be analyzed, based on the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable, includes: If the actual starting log line number is less than the actual ending log line number, read each log line from the actual starting log line number to the actual ending log line number in sequence to obtain a log content set, wherein the actual ending log line number is equal to the sum of the actual starting log line number and the incremental step size; According to preset fields, extract a portion of the content of each log item in the log content collection, and add the portion of the content to the log object list; The current analysis task is completed by updating the log object list to the target database.

7. The method according to claim 1, characterized in that, The step of obtaining at least one log file composed of message receipts provided by at least one message service provider based on a pre-created list of parameter objects includes: Obtain multi-dimensional attribute information corresponding to each message service provider among the at least one message service provider, wherein the multi-dimensional attribute information includes service provider name, location file name, log file name, and start time; By encapsulating the multidimensional attribute information, multiple parameter objects are obtained; The plurality of parameter objects are added sequentially to the parameter object list, and the at least one log file and location file corresponding to each message service provider are obtained through the parameter object list.

8. A log analysis device, characterized in that, include: The first acquisition unit is used to acquire at least one log file consisting of message receipts provided by at least one message service provider, based on a pre-created list of parameter objects. The second acquisition unit is used to acquire the start position variable and end position variable in the current analysis task through at least one thread executed in parallel. The start position variable is the start log line number determined according to the end log line number of the previous analysis task. The start log line number represents the line number of the first log in the next analysis task in the log file to be analyzed. The at least one log file includes the log file to be analyzed. The end position variable is the end log line number of the current analysis task. The analysis unit is used to synchronously analyze the log file to be analyzed according to the actual starting log line number indicated by the starting position variable and the actual ending log line number indicated by the ending position variable, so as to obtain the target analysis result.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program can be executed by a terminal device or computer at runtime as described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to perform the method as described in any one of claims 1 to 7 via the computer program.

Images