Log information issuing method and device for data operation and medium

Abstract

The invention discloses a log information issuing method and device for data operation and a medium. The method comprises the steps that a data transmission path is captured through a deployed data flow tracker; creating a log record of the data according to the transmission path information; identifying target affiliation party information of the data in an affiliation party information base; and issuing the log record to the target affiliation party according to the contact way of the target affiliation party. By comprehensively tracking the data flow direction and recording detailed log information, the data traceability can be improved, the data ownership can be automatically identified, the log information can be timely issued, and key information can be accurately and timely transmitted to the data ownership.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and medium for sending log information of data operations. Background Art

[0002] With the rapid development of information technology, the amount of data is exploding. Enterprises and organizations have built a variety of data platforms to handle this massive data volume. Within these platforms, data integration involves acquiring data from multiple sources and consolidating it into a unified data store. Data governance aims to improve data quality and standardize data usage. Resource-based operations transform data into valuable resources for diverse business operations.

[0003] Currently, data operation logs for these complex data operations are often incomplete, failing to fully record the sender, receiver, and specific operation details. This makes it difficult to quickly locate and resolve data issues. Furthermore, distributing data operation log messages often requires manual processing, which is inefficient and prone to errors. This prevents key information from being conveyed to the data owner in a timely manner, leading to the inability to deliver comprehensive data operation log messages to relevant users. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and medium for sending log information of data operations, which are used to solve the problem that comprehensive data operation log messages cannot be sent to relevant users in a timely manner.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, an embodiment of the present application provides a method for sending log information of data operations, the method comprising: capturing the transmission path of data through a deployed data flow tracker; creating a log record of the data based on the transmission path information; identifying the target owner information of the data in the owner information database; and sending the log record to the target owner based on the contact information of the target owner.

[0007] In one example, the data transmission path is captured by the deployed data flow tracker, specifically including: according to the first data flow tracker, capturing the operation of extracting data from the data source to obtain the data source record of the data; according to the second data flow tracker, capturing the operation of the intermediate processing step to obtain the data processing step record; according to the third data flow tracker, capturing the operation of data storage to obtain the data target storage record; and generating the data transmission path according to the data source record, the processing step record and the target storage record.

[0008] In one example, creating a log record of the data based on the transmission path information specifically includes: identifying the transmission path information according to a structured log format to obtain structured log information; determining whether the structured log information has any gaps; when there are gaps, determining that the data operation of the gapped part was not successfully captured, marking the structured log information as gapped, and creating a log record of the data; when there are no gaps, creating a log record of the data based on the structured log information.

[0009] In one example, identifying the target attribution information of the data in the attribution information database specifically includes: matching the data source of the data in the attribution information database to obtain the user name to which the data source belongs; obtaining the user contact information of the data based on the user name; and obtaining the target attribution information of the data based on the user name and the user contact information.

[0010] In one example, after the log record is sent to the target party based on the contact information of the target party, the method also includes: generating a sending record of the log record; when the log record fails to be sent, generating sending failure information of the sending record; and sending the sending failure information to the management user.

[0011] In one example, after the log record is sent to the target party based on the contact information of the target party, the method also includes: counting the number of sent records and the number of sent failures within a preset period in a sent record database; obtaining a notification success rate based on the ratio between the number of sent failures and the number of sent records; when the notification success rate is less than a preset success threshold, grouping and counting the sent records that failed to be sent according to the cause of failure; and determining a repair plan for the log information sending system based on the number of sent records in each group.

[0012] In one example, the repair plan for the log information sending system is determined based on the number of sent records in each group, specifically including: sorting each group in descending order; determining the failure reasons of a preset number of groups with the highest order; when the failure reason includes an invalid email address, performing format verification on the emails in the attribution party's information database, deleting emails with incorrect formats, and updating emails with incorrect formats and invalid addresses through SMS verification; when the failure reason includes rejection by the mail server, and when the content of the sending template contains marketing sensitive words, performing sensitive word repair on the sending template.

[0013] In one example, sending the log record to the target party based on the contact information of the target party specifically includes: determining whether the target party has marked the notification channel preference; when marked, sending the log record to the target party based on the contact information of the notification channel preference.

[0014] On the other hand, an embodiment of the present application provides a device for sending log information of data operations, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned methods for sending log information of data operations.

[0015] On the other hand, an embodiment of the present application provides a non-volatile computer storage medium for sending log information of data operations, which stores computer-executable instructions, and the computer-executable instructions can execute any of the above-mentioned methods for sending log information of data operations.

[0016] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0017] By comprehensively tracking data flows and recording detailed log information, data traceability can be improved. When data issues arise, the data source, operation process, and recipient can be quickly and accurately located, significantly reducing troubleshooting time and improving data management efficiency. Furthermore, automatic identification of data owners and timely issuance of log messages ensures that data owners have real-time access to data operations, meeting data security and compliance management requirements and accurately and promptly communicating key information to data owners. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, some embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a method for sending log information of a data operation provided in an embodiment of the present application;

[0020] Figure 2 A structural diagram of a data operation log information sending device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] Some embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of a method for distributing log information for data operations provided in an embodiment of the present application. This method can be applied to various business areas, such as internet finance, e-commerce, instant messaging, gaming, and government affairs. Certain input parameters or intermediate results in this process can be manually adjusted to help improve accuracy.

[0024] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For ease of understanding and description, the following embodiments are described in detail using a server as an example.

[0025] It should be noted that the server can be a single device or a system composed of multiple devices, that is, a distributed server, and this application does not make any specific restrictions on this.

[0026] Figure 1 The process in includes the following steps:

[0027] S101: Capture the data transmission path through the deployed data flow tracker.

[0028] In some embodiments of the present application, a first data flow tracker is used to capture data extraction operations from a data source, generating a data source record. Then, a second data flow tracker is used to capture operations at intermediate processing steps, generating a data processing step record. Then, a third data flow tracker is used to capture data storage operations, generating a data destination storage record. Based on the data source record, processing step record, and destination storage record, a data transmission path is generated.

[0029] For example, data flow trackers are deployed at various data operation nodes in the data platform, including the data integration module, data governance module, and resource mobilization module. As data is transmitted and processed between these modules, the trackers capture the data's transmission path in real time, recording in detail the data source from which it originates, the intermediate processing steps it undergoes, and the target storage or business module to which it ultimately flows. For example, during data integration, the tracker records the entire process of data extracted from database A, processed through ETL (extract, clean, transform, load), and then flowing into data warehouse B.

[0030] S102: Creating a log record of the data according to the transmission path information.

[0031] In some embodiments of the present application, transmission path information is identified according to a structured log format to obtain structured log information.

[0032] Determine whether the structured log information has gaps.

[0033] When there is a gap, it is determined that the data operation of the gap portion is not successfully captured, a gap mark is performed on the structured log information, and a log record of the data is created.

[0034] When there are no gaps, a log record of the data is created according to the structured log information.

[0035] In other words, based on the transmission path information obtained by the data flow tracker, the log generator creates detailed log information. The log content includes information about the data sender, such as the data source name and IP address; information about the recipient, such as the system name and module name of the data receiving system; and operations on the data, such as data cleaning, conversion, and loading. This information is stored in a structured manner in the log database to facilitate subsequent query and analysis. For example, a log record may read "[timestamp], data was sent from data source [Database A, IP: 192.168.1.100], and after cleaning and format conversion, was received by data warehouse [Data Warehouse B]."

[0036] S103: Identify the target owner information of the data in the owner information database.

[0037] In some embodiments of the present application, identification is performed based on the source, purpose, and pre-set rules of the data through an attribution identification algorithm.

[0038] For example, in the attribution database, the data source is matched to obtain the user name to which the data source belongs. Then, based on the user name, the user's contact information is obtained. Then, based on the user name and the user's contact information, the target attribution information of the data is obtained.

[0039] It should be noted that the attribution party information is stored in the attribution party information database, including the attribution party's name, contact information (such as mobile phone number, email address), etc.

[0040] For example, the user name to which database A belongs is the data middle platform user analysis team.

[0041] S104: Send the log record to the target party according to the contact information of the target party.

[0042] The log information is sent to the home party via SMS gateway or email server based on the home party's contact information. For example, if the home party's contact information is email, the message delivery module formats the log information according to the email format and sends it to the home party's designated email address.

[0043] It should be noted that a notification channel preference field (e.g., "Priority Email" or "Only SMS") can be added to the home party's information database to select the preferred channel before sending notifications. For example, if the home party has marked "Only Email" but the system defaults to SMS, the policy needs to be revised to avoid invalid notifications.

[0044] For example, subject: Data Operation Notification - User Behavior Data Integration Completed

[0045] text:

[0046] Time: 2025-04-21 14:30:00

[0047] Data source: Database A (IP: 192.168.1.200)

[0048] Operation: After cleaning (filtering invalid logs) and format conversion (unifying the time format), the data has been loaded into user analysis data warehouse B.

[0049] In some embodiments of the present application, after the log record is sent to the target home party, a log record sending record is generated.

[0050] After each successful or failed message delivery, the delivery record generator generates a delivery record. This record includes the delivery time, delivery recipient (owner information), a summary of the delivered log information, the delivery status (success or failure), and the failure reason (if the delivery failed). These delivery records are also stored in the log database to facilitate subsequent query and statistics on message delivery status.

[0051] For example, the distribution record is as follows:

[0052] Issued: 2025-04-21 14:32:00

[0053] Issued to: User Analysis Team, Email = user_analysis@ecommerce.com

[0054] Log summary: User behavior data is integrated from database A to data warehouse B, including cleaning and conversion operations. Delivery status: Success.

[0055] When the log record fails to be delivered, delivery failure information of the delivery record is generated and delivered to the management user.

[0056] For example, the delivery failure information is as follows:

[0057] Issued: 2025-04-21 14:32:05

[0058] Issued to: User Analysis Team, Email = user_analysis@ecommerce.com

[0059] Log summary: User behavior data is integrated from database A to data warehouse B, including cleaning and conversion operations. Delivery status: Failed

[0060] Failure reason: SMTP connection timeout (server IP: 10.0.0.5 port 25 is unreachable).

[0061] In some embodiments of the present application, the effect of data operation notification is quantified by issuing record statistics.

[0062] Based on this, the number of delivery records and the number of delivery failures within a preset period are counted in the delivery record database.

[0063] Then, the notification success rate is obtained based on the ratio between the number of delivery failures and the number of delivery records.

[0064] When the notification success rate is less than the preset success threshold, the delivery records that failed to be delivered are grouped and counted according to the failure reasons.

[0065] Determine the repair plan for the log information delivery system based on the number of delivered records for each group.

[0066] Among them, the repair plan may include the following:

[0067] First, each group is sorted in descending order, and then the failure reasons of a preset number of groups at the top of the sort are determined.

[0068] When the reason for failure includes an invalid email address, perform format verification on the email addresses in the home party's information database, delete the email addresses with incorrect formats, and update the email addresses with incorrect formats and invalid addresses through SMS verification;

[0069] When the failure reason includes rejection by the mail server, or when the content of the sending template contains marketing sensitive words, the sending template will be repaired for sensitive words.

[0070] It effectively tracks the flow of data for data integration, governance, and resource utilization within the data platform, generating detailed logs based on this flow, recording key information such as the sender and receiver of the data, and the specific operations performed on the data. It automatically identifies the data's owner and accurately distributes log information based on that information via SMS or email, while also generating a distribution record to capture the details. This helps improve the transparency and traceability of data operations, ensuring data security and compliance.

[0071] It should be noted that although the embodiments of this application are based on Figure 1 Steps S101 to S104 are described in sequence, but this does not mean that steps S101 to S104 must be performed in a strict order. Figure 1 The order shown in FIG1 is to introduce and explain step S101 to step S104 in order to facilitate those skilled in the art to understand the technical solution of the embodiment of the present application. In other words, in the embodiment of the present application, the order between step S101 to step S104 can be appropriately adjusted according to actual needs.

[0072] pass Figure 1 This method improves data traceability by comprehensively tracking data flows and recording detailed log information. When data issues arise, the data source, operation process, and recipient can be quickly and accurately located, significantly reducing troubleshooting time and improving data management efficiency. Furthermore, automatic identification of data owners and timely delivery of log messages ensures that data owners have real-time visibility into data operations, meeting data security and compliance management requirements and accurately and promptly communicating key information to data owners.

[0073] In other words, by comprehensively recording data operation logs and automatically and accurately sending log messages to the data owner, work efficiency can be improved while also improving the transparency and traceability of data operations and strengthening data security and compliance management.

[0074] Based on the same idea, some embodiments of the present application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0075] Figure 2 A schematic diagram of a device for sending log information of data operations provided in an embodiment of the present application includes:

[0076] at least one processor; and,

[0077] a memory communicatively connected to the at least one processor; wherein,

[0078] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned methods for sending log information of data operations.

[0079] Some embodiments of the present application provide a non-volatile computer storage medium for sending log information of data operations, which stores computer-executable instructions. The computer-executable instructions can execute any of the above-mentioned methods for sending log information of data operations.

[0080] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0081] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0082] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0088] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0090] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the technical principles of the present application should fall within the scope of protection of the present application.

Claims

1. A method for sending log information of data operation, characterized in that: The method comprises: Capture the data transmission path through the deployed data flow tracker; Creating a log record of the data based on the transmission path information; Identifying target owner information of the data in an owner information database; The log record is sent to the target party according to the contact information of the target party.

2. The method according to claim 1, characterized in that The data transmission path captured by the deployed data flow tracker specifically includes: According to the first data flow tracker, an operation of extracting data from a data source is captured to obtain a data source record of the data; According to the second data flow tracker, operations of the intermediate processing steps are captured to obtain a data processing step record; According to the third data flow tracker, the data storage operation is captured to obtain the target storage record of the data; A transmission path for the data is generated according to the data source record, the processing step record and the target storage record.

3. The method according to claim 1, characterized in that The step of creating a log record of the data according to the transmission path information specifically includes: Identifying the transmission path information according to a structured log format to obtain structured log information; Determining whether the structured log information has any gaps; When there is a gap, determining that the data operation of the gap portion is not successfully captured, marking the gap in the structured log information, and creating a log record of the data; When there is no gap, a log record of the data is created according to the structured log information.

4. The method according to claim 1, wherein The identifying target attribution information of the data in the attribution information database specifically includes: In the attribution information database, the data source is matched to obtain the user name to which the data source belongs; According to the user name, obtain the user contact information of the data; The target attribution information of the data is obtained according to the user name and the user contact information.

5. The method according to claim 1, wherein After sending the log record to the target party according to the contact information of the target party, the method further includes: Generate a distribution record of the log record; When the log record fails to be issued, generating issuance failure information of the issuance record; The sending failure information is sent to the management user.

6. The method according to claim 5, characterized in that After sending the log record to the target party according to the contact information of the target party, the method further includes: In the delivery record database, count the number of delivery records and the number of delivery failures within a preset period; Obtaining a notification success rate based on a ratio between the number of failed delivery and the number of delivery records; When the notification success rate is less than a preset success threshold, grouping and counting the delivery records that failed to be delivered according to the failure reasons; Determine the repair plan for the log information delivery system based on the number of delivered records for each group.

7. The method according to claim 6, characterized in that Determining a repair plan for the log information delivery system based on the number of delivered records in each group specifically includes: Sort each group in descending order; Determine the failure reasons for a preset number of groups ranked first; When the reason for failure includes an invalid email address, perform format verification on the email addresses in the home party's information database, delete the email addresses with incorrect formats, and update the email addresses with incorrect formats and invalid addresses through SMS verification; When the failure reason includes rejection by the mail server, or when the content of the sending template contains marketing sensitive words, the sending template will be repaired for sensitive words.

8. The method according to claim 1, characterized in that The sending of the log record to the target party according to the contact information of the target party specifically includes: Determining whether the target party has marked a notification channel preference; When marking, the log record is sent to the target party according to the contact method preferred by the notification channel.

9. A device for sending log information of data operation, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for sending log information of a data operation as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium for sending log information of data operations, storing computer executable instructions, characterized in that: The computer executable instructions can execute the method for sending log information of a data operation as described in any one of claims 1 to 8.

Images