Data asset detection method and device
By monitoring the access operations of the target database and using the target rule database to trigger active detection, dynamically adjusting the detection rules, the problem of frequent detection affecting system performance is solved, and efficient and energy-saving data asset detection is achieved.
Patent Information
- Application Number
- CN202510222167.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-26
- Publication Date
- 2025-06-13
AI Technical Summary
While the prior art meets the need to timely grasp the changes in data assets, it is difficult to avoid the impact of frequent data asset detection on system performance.
By monitoring the access operations of the target database, the target rule database is used to trigger active detection of the target database, and dynamically adjust the rules according to the data asset changes record to reduce non-essential detection behavior.
It effectively reduces non-essential detection behavior, avoids the negative impact of frequent detection on system performance, and improves the utilization efficiency of computing resources.
Smart Images
Figure CN120144598A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communication technologies, and in particular, to a method and device for detecting data assets. Background Art
[0002] Data asset detection is a basic task in data management, which can help enterprises comprehensively understand, effectively utilize, and protect data assets, and at the same time provide strong support for digital transformation, business innovation, and compliance management. How to meet the need to promptly grasp the changes in data assets while avoiding frequent detection of data assets from affecting system performance is a key research direction in this field. Summary of the Invention
[0003] To overcome the problems existing in the related art, this application provides a method and device for detecting data assets.
[0004] According to the first aspect of the embodiments of this application, a method for detecting data assets is provided. The method includes:
[0005] Monitoring access operations on a target database;
[0006] When any rule in the target rule library matches the access operation, performing a detection operation on the target database to obtain a data asset detection result;
[0007] Updating the local data asset record according to the data asset detection result, and generating a data asset change record for the target database;
[0008] Wherein, the target rule library is initialized with any type of access operation and updated according to a rule change plan output by a target model with the data asset change record of the target database as input.
[0009] According to the second aspect of the embodiments of this application, a device for detecting data assets is provided. The device includes:
[0010] A monitoring module, configured to monitor access operations on a target database;
[0011] A detection module, configured to perform a detection operation on the target database to obtain a data asset detection result when any rule in the target rule library matches the access operation;
[0012] An update module, configured to update the local data asset record according to the data asset detection result, and generate a data asset change record for the target database;
[0013] Wherein, the target rule library is initialized with any type of access operation and updated according to a rule change plan output by a target model with the data asset change record of the target database as input.
[0014] According to a third aspect of the embodiments of the present application, an electronic device is provided, including:
[0015] a memory and one or more processors; the memory is coupled to the processor; wherein, computer program code is stored in the memory, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described above.
[0016] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, including computer instructions, and when the computer instructions run on an electronic device, the electronic device is caused to execute the method as described above.
[0017] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, and when the computer program product runs on a computer, the computer is caused to execute the method as described above.
[0018] The technical solutions provided by the embodiments of the present application may include the following beneficial effects:
[0019] The data asset detection method of the present application can monitor access operations to a target database; when an access operation matches any rule in the target rule library, a detection operation on the target database is executed to obtain a data asset detection result; finally, the local data asset record is updated according to the data asset detection result, and a data asset change record of the target database is generated; wherein, the target rule library is initialized with any type of access operation and is updated according to a rule change scheme output by a target model with the data asset change record of the target database as input.
[0020] It can be seen that this method triggers an active detection of the target database based on the target rule library, and the target rule library dynamically adjusts the rules according to the data asset change record during the operation of the system, effectively reducing unnecessary detection behaviors and avoiding the impact of frequent detection behaviors on system performance.
[0021] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of this application, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.
[0023] Figure 1 is a schematic diagram of a traditional data asset detection process;
[0024] Figure 2Flowchart of the data asset detection method provided by the embodiments of the present application;
[0025] Figure 3 Schematic diagram of the principle of the data asset detection method provided by the embodiments of the present application;
[0026] Figure 4 Schematic diagram of the detailed information of the access operation provided by the embodiments of the present application;
[0027] Figure 5 Functional block diagram of the data asset detection device provided by the embodiments of the present application;
[0028] Figure 6 Schematic diagram of the structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners
[0029] Next, with reference to the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the present application.
[0030] It should be noted that "at least one" in the present application means one or more, and "a plurality" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The terms "first", "second", "third", etc. (if any) in the specification, claims and drawings of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.
[0031] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific manner.
[0032] The data asset detection process is as Figure 1 shown. To achieve the detection of data assets, it is necessary to configure a data asset detection task, clarify the data scope and boundaries, and specify which data sources need to be detected, including databases, file systems, and cloud storage, etc. Secondly, it is necessary to clarify the detection mode of data assets, whether it is a one-time detection or a periodic detection, including the detection time period and when to start the detection.
[0033] Traditional data asset detection solutions are not aware of database changes. To ensure a comprehensive and timely grasp of data assets, periodic asset detection tasks are generally set, such as performing a full database scan starting at 0:00 every day. This detection method will trigger detection regardless of whether the data assets have changed, and the detection that does not update the data assets is a meaningless waste of resources. At the same time, it is difficult to accurately set the size of the detection cycle. If the cycle frequency is too low, data assets that have changed cannot be detected in a timely manner. If the cycle frequency is too high, a large number of ineffective scan operations will be brought, resulting in a significant waste of computing resources.
[0034] The principle of another data asset detection solution is to monitor the access operations of the database in real time, and trigger a detection task every time an access operation is monitored. However, not all access operations are necessary to perform a data asset detection. For example, after adding a table entry and then deleting the table entry, or after modifying a table entry and then changing back the value of the table entry. That is to say, if a data asset detection is triggered every time the database is accessed, a large number of meaningless detection tasks will also be brought, consuming computing resources and affecting system performance.
[0035] In view of the above problems, the present application provides a data asset detection method and device.
[0036] Next, the embodiments of the present application will be described in detail.
[0037] The embodiment of the present application provides a data asset detection method, as Figure 1 shown, the method may include the following steps:
[0038] Step 110, monitor the access operations of the target database;
[0039] Step 120, when the access operation matches any rule in the target rule library, perform a detection operation on the target database to obtain a data asset detection result;
[0040] Step 130, update the local data asset record according to the data asset detection result, and generate a data asset change record of the target database.
[0041] The data asset detection method of the present application is applied to the scenario where it is necessary to timely and accurately grasp the latest and most complete data assets after the initial data asset detection.
[0042] In this embodiment, the target rule library is used to record a series of rules. When the access operation matches any rule in the target rule library, a detection operation on the target database is triggered. Specifically, the target rule library is initialized to any access operation of the target database, that is, when any access operation of the target database is monitored, a detection operation on the target database will be triggered.
[0043] In particular, the target rule base will also be updated based on the rule change plan output by the target model with the data asset change records of the target database as the input. Through this update, the matching success probability of the rules in the target rule base is reduced, thereby reducing the detection frequency of the target database, saving computing resources, and avoiding the impact of frequent unnecessary detection operations on the system performance.
[0044] As a specific implementation manner, the update manner of the target rule base may be to delete some rules. At this time, the target model is used to determine the rules that can be deleted in the target rule base, hereinafter referred to as the rules to be deleted. At this time, the update process of the target rule base is as follows: input the data asset change records of the target database into the target model to obtain the rules to be deleted in the target rule base; perform a deletion operation on the rules to be deleted in the target rule base.
[0045] As a specific implementation manner, the update manner of the target rule base may be to add subsidiary rules to the rules. At this time, the target model is used to determine the subsidiary rules of the target rules (any rules) in the target rule base. That is, assuming that a certain rule in the target rule base is initially A0, and the target model determines that the subsidiary rule of this rule is A1. Before the rule update, when A0 is matched in the access operation, the detection of the target database is triggered. After the rule update, the detection of the target database can only be triggered when both A0 and A1 are matched in the access operation. At this time, the update process of the target rule base is as follows: input the data asset change records of the target database into the target model to obtain the subsidiary rules of the target rules in the target rule base; add subsidiary rules to the target rules in the target rule base.
[0046] Regarding the update timing of the target rule base, a target condition can be preset in advance. When the system reaches this target condition, the update operation of the target rule base is performed according to the above process. Specifically, at this time, the update process of the target rule base is as follows: when the target condition is reached, input the data asset change records of the target database into the target model to obtain the rule change plan of the target rule base; according to the rule change plan, perform an update operation on the target rule base.
[0047] The above target condition may specifically be that the data volume of the data asset change records of the target database reaches a preset quantity, or the record duration of the data asset change records reaches a preset duration, or a user update instruction is received. Specifically, the target condition can be set or adjusted according to actual requirements, and this embodiment does not limit this.
[0048] On this basis, in order to avoid updating the target rule base according to an unreliable rule update scheme, as a preferred implementation, after the target model outputs a rule update scheme, the user is first prompted to authorize the rule update scheme, and only after obtaining authorization, the target rule base is updated according to the rule update scheme output by the target model. At this time, specifically, the update process of the target rule base is as follows: input the data asset change record of the target database into the target model to obtain a rule change scheme for the target rule base; generate an authorization prompt for the rule change scheme; in response to the authorization operation for the rule change scheme, perform an update operation on the target rule base according to the rule change scheme.
[0049] As a specific implementation, this embodiment specifically monitors access operations on the target database in the following manner: use a data behavior probe to obtain access traffic data of the target database; determine access operations according to the access traffic data by means of packet parsing.
[0050] As a specific implementation, this embodiment specifically performs a detection operation on the target database in the following manner: send an active detection task to the data asset probe to enable the data asset probe to perform a detection operation on the target database.
[0051] As a specific implementation, this embodiment specifically obtains the target model in the following manner: use a random forest model to perform supervised learning on a training set to obtain the target model.
[0052] Next, taking an actual application as an example, the data asset detection method of the present application will be introduced.
[0053] As Figure 3 shown, deploy a data security management platform and a data behavior probe. The probe can collect access traffic data of the target database through the bypass mirroring method (or install a traffic collection proxy module on the database server), and identify detailed information of access operations such as sql statements of database requests in the traffic through packet parsing, as Figure 4 shown.
[0054] On the data security management platform, initialize the target rule base for any access operation. At this time, once access operations such as ALTER, DROP, Create, etc. are detected, an active detection task is triggered. The format of the target rule base is as follows:
[0055] Serial number Trigger rule Object category 1 The operation is CREATE Data table 2 The operation is DROP Data table 3 The operation is ALTER Data table 4 The operation is CREATE Database 5 The operation is DROP Database 6 The operation is ALTER Database
[0056] Specifically, if any rule in the target rule library is matched by the access operation in the user's access behavior to the target database, the data security management platform then sends an active detection task to the data asset probe. The data asset probe executes the active detection task and uploads the data asset detection results to the data security management platform. The data security management platform updates the local data asset records, forms the latest data asset inventory, and records each data asset change record.
[0057] The data asset change records are as follows:
[0058]
[0059] The purpose of data asset detection is to form a data asset ledger, and the purpose of each trigger of detection is to revise the changes of data assets. Suppose 10 data assets are obtained in the first scan, 11 are found in the second scan, and a new data asset Z is added. In the third scan, only 10 assets are found, and one data asset Z is missing. From an economic perspective, the second scan and the third scan are unnecessary.
[0060] Based on this, in this embodiment, the data security management platform analyzes the data asset change records to determine whether some rules in the target rule library need to be deleted through the target model. The process is as follows:
[0061] Sort out the data and read the historical records of the data asset change record table; load the initial target rule library. Use the pre-trained target model to predict the rules that can be deleted; conduct a secondary verification on the rules to be deleted predicted by the target model to ensure that important triggering conditions will not be misdeleted, such as verifying with business experts to ensure that the deletion operation will not affect the normal business process. Finally, after obtaining authorization, perform the deletion operation on the rules to be deleted in the target rule library.
[0062] The training process of the target model is as follows:
[0063] (1) Feature extraction. Extract features from the data for training the random forest model. The features include but are not limited to the following:
[0064] Change time difference: The time difference from the last change;
[0065] Change type frequency: The occurrence frequency of a certain change type within a certain period of time in the past (such as one day, one hour, etc.);
[0066] Count of multiple change types of the same asset occurring in a short time: The count of change types (such as new addition and deletion occurring simultaneously) of the same data asset within a certain time window.
[0067] (2) Model training. Divide the data into a training set and a test set, and use random forest to train on the training set.
[0068] (3) Model evaluation. Evaluate the model performance on the test set, and pay attention to the accuracy rate and error rate to ensure that the model can correctly identify which conditions can be deleted.
[0069] As can be seen from the above technical solutions, the data asset detection method provided by this application uses a data behavior probe to collect database access traffic and identify access operations on the database. When any rule in the target rule library is matched in the access operation, an active detection task is generated to obtain the latest and most complete data asset list in the most accurate and efficient manner. In addition, the target rule library is dynamically adjusted according to the data asset change record during the system operation, and the deletable rules are identified, achieving a further balance between cost and efficiency.
[0070] Based on the same inventive concept, this application also provides a data asset detection device, and its structural schematic diagram is as Figure 5 shown, specifically including:
[0071] A monitoring module 510, which is used to monitor access operations on the target database;
[0072] A detection module 520, which is used to perform a detection operation on the target database when any rule in the target rule library is matched in the access operation, and obtain a data asset detection result;
[0073] An update module 530, which is used to update the local data asset record according to the data asset detection result and generate a data asset change record of the target database;
[0074] Among them, the target rule library is initialized to any type of access operation and is updated according to the rule change plan output by a target model with the data asset change record of the target database as the input.
[0075] As a specific implementation manner, the monitoring module monitors access operations on the target database in the following specific way:
[0076] Use a data behavior probe to obtain access traffic data of the target database; determine the access operation according to the access traffic data by means of message parsing.
[0077] As a specific implementation manner, the detection module performs a detection operation on the target database in the following specific way:
[0078] Send an active detection task to the data asset probe to enable the data asset probe to perform a detection operation on the target database.
[0079] As a specific implementation manner, the device further includes:
[0080] A deletion module, configured to input the data asset change record of the target database into a target model to obtain a rule to be deleted in the target rule library; and perform a deletion operation on the rule to be deleted in the target rule library.
[0081] As a specific implementation manner, the device further includes:
[0082] An accessory module, configured to input the data asset change record of the target database into a target model to obtain an accessory rule of a target rule in the target rule library; and add the accessory rule to the target rule in the target rule library.
[0083] As a specific implementation manner, the device further includes:
[0084] A condition module, configured to, when a target condition is met, input the data asset change record of the target database into a target model to obtain a rule change plan for the target rule library; and perform an update operation on the target rule library according to the rule change plan.
[0085] As a specific implementation manner, the device further includes:
[0086] An authorization module, configured to input the data asset change record of the target database into a target model to obtain a rule change plan for the target rule library; generate an authorization prompt for the rule change plan; and in response to an authorization operation on the rule change plan, perform an update operation on the target rule library according to the rule change plan.
[0087] As a specific implementation manner, the device further includes:
[0088] A training module, configured to perform supervised learning on a training set by using a random forest model to obtain a target model.
[0089] An embodiment of the present application provides an electronic device, which may include: a memory and one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute each function or step of the foregoing method embodiment.
[0090] The structure of the electronic device may refer to Figure 6 the structure of the electronic device 100 shown in the figure.
[0091] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0092] An embodiment of this application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions run on an electronic device, the electronic device is enabled to execute each function or step of the above method embodiment.
[0093] The above computer-readable storage medium includes, but is not limited to, any one of the following: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disc, etc., various media that can store program codes.
[0094] An embodiment of this application also provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute each function or step of the above method embodiment.
[0095] Among them, the electronic device, computer-readable storage medium, and computer program product provided by the embodiments of this application are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.
[0096] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
[0097] In several embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the modules or units can be in electrical, mechanical or other forms.
[0098] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
[0099] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A data asset detection method, characterized in that: The method comprises: Monitor access operations to the target database; When the access operation matches any rule in the target rule base, a detection operation is performed on the target database to obtain a data asset detection result; Update the local data asset record according to the data asset detection result, and generate the data asset change record of the target database; The target rule base is initialized to any type of access operation and is updated according to the rule change plan output by the target model with the data asset change record of the target database as input.
2. The method according to claim 1, characterized in that: The method specifically monitors access operations to the target database in the following manner: Use data behavior probes to obtain access traffic data of the target database; The access operation is determined based on the access traffic data by means of message parsing.
3. The method according to claim 1, characterized in that The method specifically performs the detection operation on the target database in the following manner: An active detection task is sent to a data asset probe, so that the data asset probe performs a detection operation on the target database.
4. The method according to claim 1, characterized in that: The method further comprises: Inputting the data asset change records of the target database into the target model to obtain the rules to be deleted of the target rule base; A deletion operation is performed on the to-be-deleted rule in the target rule base.
5. The method according to claim 1, characterized in that The method further comprises: Inputting the data asset change record of the target database into the target model to obtain the subsidiary rules of the target rule in the target rule base; The subsidiary rule is added to the target rule in the target rule base.
6. The method according to claim 1, characterized in that The method further comprises: When the target condition is met, the data asset change record of the target database is input into the target model to obtain the rule change plan of the target rule base; According to the rule change plan, an update operation is performed on the target rule base.
7. The method according to claim 1, characterized in that The method further comprises: Inputting the data asset change records of the target database into the target model to obtain the rule change plan of the target rule base; generating an authorization prompt for the rule change plan; In response to an authorization operation on the rule change plan, an update operation is performed on the target rule base according to the rule change plan.
8. The method according to claim 1, characterized in that: The method further comprises: The random forest model is used to perform supervised learning on the training set to obtain the target model.
9. A data asset detection device, characterized in that: The device comprises: A monitoring module, used to monitor access operations to a target database; A detection module, used to perform a detection operation on the target database when the access operation matches any rule in the target rule base, and obtain a data asset detection result; An update module, used to update the local data asset record according to the data asset detection result, and generate a data asset change record of the target database; The target rule base is initialized to any type of access operation and is updated according to the rule change plan output by the target model with the data asset change record of the target database as input.
10. An electronic device, characterized in that: include: A memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-8.
11. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 8.
12. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.