A data asset identification method and device, storage medium and electronic device
By acquiring and analyzing the metadata information of data files and using model prediction processing to identify data assets, the problem of low efficiency in identifying data assets in massive data resources is solved, fast and accurate data asset identification is achieved, and the data asset processing of enterprises is improved efficiency.
Patent Information
- Application Number
- CN202410787301.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-06-18
AI Technical Summary
Staff who identify data assets in massive data resources face challenges. The existing technology cannot effectively and quickly identify data assets, resulting in high manual participation and low efficiency.
By obtaining the metadata information of the data file, extracting the number of usage sequences, determining whether the time series has random characteristics, if it does not have random characteristics, model prediction processing is performed, and future profit prediction is predicted using preset model parameters combinations to determine whether the data file satisfies the future. The income conditions are determined to be a data asset.
It realizes the rapid identification of data assets in massive data resources, reduces manual participation, improves work efficiency, deeply explores the relevant information value of data files, and provides enterprises with a solution to quickly identify data assets.
Smart Images

Figure CN118796770B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data asset identification method and device, a storage medium and an electronic device. Background Art
[0002] Data resources refer to a collection of various information and data that exist in digital form and can be collected, stored, processed, analyzed, and used in the digital economy era. It includes structured data, such as tables and fields in databases, and unstructured data, such as text, pictures, videos, audio, etc. Data resources exist widely in various information carriers, including but not limited to the Internet, enterprise information systems, IoT devices, social media platforms, etc.
[0003] Data resources can be measured or traded, and can directly or indirectly bring economic and social benefits. Such data resources can be called data assets. Among the data resources owned by an organization, not all data resources can be regarded as data assets. Data assets are those data resources that can create value for the organization.
[0004] After identifying data assets, they can be classified, graded, evaluated and priced so that they can be managed and applied. As a product of the development of new economic formats, data assets, including identification and measurement technologies, are new topics. How to identify data assets in massive data resources has become an urgent problem that staff need to solve. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a data asset identification method and device, a storage medium and an electronic device. By applying the method provided by the present invention, data assets can be quickly identified in massive data resources, assisting the enterprise's data asset processing work, and the entire process has less human involvement, reducing labor input and improving work efficiency.
[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0007] A data asset identification method, comprising:
[0008] Get metadata information of each data file;
[0009] Based on the metadata information of each of the data files, obtaining a usage count sequence of each of the data files, wherein the usage count sequence includes the usage count of the data file in each preset first time interval;
[0010] Determine a time sequence of each of the data files based on a usage count sequence of each of the data files;
[0011] For each of the data files, determine whether the time series of the data file has random characteristics. When the time series of the data file does not have random characteristics, determine the time series of the data file as a modeling sequence, and apply the modeling sequence and preset various model parameter combinations to perform model prediction processing on the data file to obtain the future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset.
[0012] In the above method, optionally, the step of obtaining metadata information of each data file includes:
[0013] Determine each file to be identified;
[0014] Obtaining description information of each file to be identified;
[0015] Based on each of the description information, the files to be identified that meet the preset priority identification rules are determined as data assets, and the files to be identified that do not meet the priority identification rules are determined as data files, and the description information of the data files is determined as metadata information.
[0016] The above method may optionally include determining, based on each of the description information, the files to be identified that meet a preset priority identification rule among the files to be identified as data assets, including:
[0017] Determine when to conduct data asset identification work and prioritize identification time steps;
[0018] Determine a priority identification time interval based on the work start time and the priority identification time step;
[0019] Based on each of the description information, determining a determination time for each of the files to be identified;
[0020] For each file to be identified, determine whether the judgment time of the file to be identified is within the priority identification time interval. If the judgment time of the file to be identified is within the priority identification time interval, determine that the file to be identified meets the priority identification rule, and determine the file to be identified as a data asset; if the judgment time of the file to be identified is not within the priority identification time interval, determine that the file to be identified does not meet the priority identification rule.
[0021] In the above method, optionally, determining whether to identify the data file as a data asset based on the time series of the data file includes:
[0022] Determining whether the time series is a white noise sequence;
[0023] When the time series is a white noise sequence, determining that the time series has a random characteristic;
[0024] When the time series is not a white noise sequence, it is determined that the time series does not have a random characteristic.
[0025] In the above method, optionally, applying the modeling sequence and the preset model parameter combinations to perform model prediction processing on the data file to obtain the future revenue prediction result of the data file includes:
[0026] Constructing a prediction model, and sorting the combinations of the model parameters to obtain a combination sequence;
[0027] Determine the first model parameter combination in the combination sequence as the target model parameter combination;
[0028] updating the model parameters of the prediction model based on the target model parameter combination, and processing the modeling sequence using the prediction model after the model parameters are updated, and outputting a predicted usage count sequence, wherein the predicted usage count sequence includes the predicted usage counts of the data file in each preset second time interval;
[0029] When the predicted usage times sequence contains a predicted usage times greater than 0, generating a future revenue prediction result indicating that the data file meets the future revenue condition;
[0030] When there is no predicted usage count greater than 0 in the predicted usage count sequence, determining whether the target model parameter combination is the last model parameter combination in the combination sequence;
[0031] When the target model parameter combination is not the last model parameter combination in the combination sequence, the next model parameter combination located at the target model parameter combination in the combination sequence is used as a new target model parameter combination, and the step of returning to execute the updating of the model parameters of the prediction model based on the target model parameter combination, and using the prediction model after the updated model parameters to process the modeling sequence, and outputting a prediction usage number sequence;
[0032] When the target model parameter combination is the last model parameter combination in the combination sequence, a future profit prediction result is generated, which indicates that the data file does not meet the future profit condition.
[0033] The above method may optionally further include:
[0034] Determine each data file that is not determined to be a data asset as a first target data file, and determine the data file that is determined to be a data asset through model prediction processing as a second target data file;
[0035] Based on the usage count sequence of each of the first target data files and the usage count sequence of each of the second target data files, determining a feature similarity sequence of each of the first target data files, wherein the feature similarity sequence includes feature similarities between the first target data file and each of the second target data files;
[0036] For each of the first target data files, when there is a feature similarity sequence of the first target data file that meets a preset threshold requirement, the first target data file is determined as a data asset.
[0037] A data asset identification device, comprising:
[0038] A first acquisition unit, used to acquire metadata information of each data file;
[0039] A second acquisition unit, configured to acquire a usage count sequence of each of the data files based on metadata information of each of the data files, wherein the usage count sequence includes a usage count of the data file in each preset first time interval;
[0040] A first determining unit, configured to determine a time sequence of each of the data files based on a sequence of usage times of each of the data files;
[0041] An identification unit is used to determine, for each of the data files, whether the time series of the data file has random characteristics. When the time series of the data file does not have random characteristics, the time series of the data file is determined as a modeling sequence, and the modeling sequence and a combination of preset model parameters are used to perform model prediction processing on the data file to obtain a future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset.
[0042] In the above device, optionally, the first acquisition unit includes:
[0043] A first determination subunit, used to determine each file to be identified;
[0044] A first acquisition subunit is used to acquire description information of each file to be identified;
[0045] The second determination subunit is used to determine, based on each of the description information, the files to be identified that meet the preset priority identification rules among the files to be identified as data assets, and determine the files to be identified that do not meet the priority identification rules as data files, and determine the description information of the data files as metadata information.
[0046] In the above device, optionally, the second determining subunit includes:
[0047] The first determination module is used to determine the time for carrying out data asset identification work and determine the priority identification time step;
[0048] A second determination module is used to determine a priority identification time interval based on the work start time and the priority identification time step;
[0049] A third determination module, used to determine the determination time of each of the to-be-identified files based on each of the description information;
[0050] The fourth determination module is used to determine, for each file to be identified, whether the determination time of the file to be identified is within the priority identification time interval; if the determination time of the file to be identified is within the priority identification time interval, it is determined that the file to be identified meets the priority identification rule, and the file to be identified is determined as a data asset; if the determination time of the file to be identified is not within the priority identification time interval, it is determined that the file to be identified does not meet the priority identification rule.
[0051] In the above device, optionally, the identification unit includes:
[0052] The first judgment subunit is used to judge whether the time series is a white noise sequence; when the time series is a white noise sequence, determine that the time series has random characteristics; when the time series is not a white noise sequence, determine that the time series does not have random characteristics.
[0053] In the above device, optionally, the identification unit includes:
[0054] A construction subunit is used to construct a prediction model, and to sort the combinations of the model parameters to obtain a combination sequence;
[0055] A third determining subunit, configured to determine the first model parameter combination in the combination sequence as a target model parameter combination;
[0056] A processing subunit, configured to update the model parameters of the prediction model based on the target model parameter combination, and use the prediction model after the model parameters are updated to process the modeling sequence, and output a predicted usage count sequence, wherein the predicted usage count sequence includes the predicted usage counts of the data file in each preset second time interval;
[0057] A first generating subunit is used for generating a future revenue prediction result indicating that the data file meets a future revenue condition when there is a predicted usage number greater than 0 in the predicted usage number sequence;
[0058] A second judging subunit is used to judge whether the target model parameter combination is the last model parameter combination in the combination sequence when there is no predicted usage count greater than 0 in the predicted usage count sequence; when the target model parameter combination is not the last model parameter combination in the combination sequence, use the next model parameter combination located at the target model parameter combination in the combination sequence as a new target model parameter combination, and return to the processing subunit to execute the steps of updating the model parameters of the prediction model based on the target model parameter combination, and using the prediction model after the updated model parameters to process the modeling sequence, and output the predicted usage count sequence;
[0059] The second generating subunit is used for generating a future profit prediction result indicating that the data file does not meet the future profit condition when the target model parameter combination is the last model parameter combination in the combination sequence.
[0060] The above device may optionally further include:
[0061] A second determination unit, configured to determine each data file that has not been determined as a data asset as a first target data file, and to determine a data file that has been determined as a data asset through model prediction processing as a second target data file;
[0062] A third determining unit, configured to determine a feature similarity sequence of each first target data file based on a usage count sequence of each first target data file and a usage count sequence of each second target data file, wherein the feature similarity sequence includes feature similarities between the first target data file and each second target data file;
[0063] The fourth determining unit is configured to determine, for each of the first target data files, the first target data file as a data asset when a feature similarity sequence of the first target data file satisfies a preset threshold requirement.
[0064] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the data asset identification method as described above.
[0065] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the data asset identification method as described above.
[0066] Compared with the prior art, the present invention has the following advantages:
[0067] The present invention provides a data asset identification method and device, a storage medium and an electronic device. After obtaining metadata information of a data file, the use frequency sequence of the data file is extracted from the metadata information, and then the time series of each data file is determined based on the use frequency sequence of each data file. For each data file, it is determined whether the time series of the data file has a random characteristic. When the time series of the data file does not have a random characteristic, the time series of the data file is determined as a modeling sequence, and the modeling sequence and the preset model parameter combinations are applied to perform model prediction processing on the data file to obtain the future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset. The model prediction processing process of the present invention abandons the conventional Akaike information criterion and Bayesian information criterion used to find the optimal model to determine the model parameters p and q, and proposes to use the prediction result as a feedback mechanism to determine the parameters p and q of the model. The present invention mines the metadata information of data files to obtain a usage frequency sequence for identifying whether a data file is a data asset, deeply mines the value of the relevant information of the data file, and identifies whether a data file is a data asset through modeling, thereby providing enterprises with a solution for quickly identifying whether a data file is a data asset, helping enterprises to process data assets, reduce the workload of staff, and improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0069] Figure 1 A flowchart of a data asset identification method provided by an embodiment of the present invention;
[0070] Figure 2A flow chart of a method for obtaining metadata information of each data file provided by an embodiment of the present invention;
[0071] Figure 3 A flowchart of using a priority recognition rule to recognize a file according to an embodiment of the present invention;
[0072] Figure 4 A flow chart of using the time series of a data file to identify whether a data file is a data asset provided by an embodiment of the present invention;
[0073] Figure 5 A flowchart of using a model to predict a file provided by an embodiment of the present invention;
[0074] Figure 6 A flowchart of feature-based supplementary identification provided by an embodiment of the present invention;
[0075] Figure 7 A flowchart of another data asset identification provided by an embodiment of the present invention;
[0076] Figure 8 A schematic diagram of the structure of a data identification device provided by an embodiment of the present invention;
[0077] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0079] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0080] Data assets are a brand new "new asset" with completely different value characteristics and value space from the current tangible and intangible assets. They have received unprecedented attention, and well-known companies in major cities are eager to carry out related work. However, as a product of the development of new technologies, the confirmation and measurement of data assets are new topics. How to identify data assets has become a problem that researchers need to solve.
[0081] In order to solve the above-mentioned problems, the present invention provides a data asset identification solution. The application of the present invention can quickly identify data assets, thereby facilitating the promotion of data asset financial processing.
[0082] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.
[0083] Reference Figure 1 , is a flow chart of a data asset identification method provided by an embodiment of the present invention, and is specifically described as follows:
[0084] S101: Obtain metadata information of each data file.
[0085] The data files of the present invention may be data files that have not been screened, or data files that have not been identified as data assets after preliminary identification processing.
[0086] The present invention can use the information acquisition module to acquire metadata information of the data file.
[0087] For data files that have not been screened, the metadata information of the data files can be obtained through an interface, or through an SQL statement. Furthermore, for entities that need to identify data assets (such as enterprises and systems), if the metadata information of the entity is insufficient, logs can be obtained, and then the required metadata information can be parsed through regular expressions or other tools. For example, the name of the data file used can be parsed from the SQL statement in the log file, and the running time of the statement can be parsed as the data file usage time. Finally, after parsing and cleaning, neat metadata information required for data asset identification is formed.
[0088] The metadata information includes data file name, data file creation time, data file modification time, data lineage (upstream and downstream dependencies), data file usage records, etc.
[0089] S102: Based on metadata information of each data file, obtain a usage count sequence of each data file, where the usage count sequence includes the usage count of the data file in each preset first time interval.
[0090] Preferably, the usage time in the data file usage record in the metadata information of each data file can be discretized, and then the time period sequence of each data file can be obtained, and there are multiple first time intervals in each time period sequence, and the time unit of the first time interval can be day, week, month, quarter, half a year or year; the time period sequence of each data file can be the same or different. Exemplarily, the time period sequence of all data files is the same, such as the time period sequence is {January 2024, February 2024, March 2024, May 2024, June 2024}.
[0091] Taking the time period sequence {January 2024, February 2024, March 2024, May 2024, June 2024} as an example, for each data file, the number of times the data file is used in each first time interval of the time series period is calculated, and then the usage number sequence of the data file can be obtained. For example, the usage number sequence is {0, 1, 2, 3, 0, 9}, which means that the data file is used 0 times in January 2024, 1 time in February 2024, 2 times in March 2024, 3 times in April 2024, 0 times in May 2024, and 9 times in June 2024.
[0092] S103: Determine the time series of each data file based on the usage frequency series of each data file.
[0093] The usage frequency sequence is a sequence based on the usage time records of the data file. For the specific statistical process, refer to Figure 1 The metadata information of the data file is processed using a data processing module to form neat time series data that can be directly used by subsequent models.
[0094] Preferably, when the usage frequency sequence of the data file is a stationary sequence, the usage frequency sequence of the data file can be used as a time series; when the usage frequency sequence of the data file is a non-stationary sequence, it is necessary to perform a differential operation on the usage frequency sequence of the data file to obtain the time series of the data file; specifically, the differential operation can be a d-order differential operation.
[0095] For example, M is used to represent a data file, and the usage frequency sequence of the data file M is represented by P, where P = {P1, P2, P3, ..., P t}, P tIndicates the number of times the data file M is used in the tth first time interval.
[0096] The process of testing whether the usage frequency sequence of data file M is a stable sequence is as follows:
[0097] The ADF test method (Augmented Dickey-Fuller Test, ADF) is used to test the stability of the usage frequency sequence P of the data file M. According to the ADF test results, if the sequence P can reject the null hypothesis of "the sequence has a unit root", it is stable, and W = P is used as the subsequent modeling sequence; if the sequence P cannot reject the null hypothesis of "the sequence has a unit root", it is non-stationary. At this time, the usage frequency sequence P can be used as X, that is, X = P, and X is transformed by d-order difference to obtain a stable time series.
[0098] For non-stationary situations, a stationarity test is performed after the difference is performed until the stationarity condition is met, and then the value of d is determined. For example, after first performing a first-order difference operation on P, if Δ d If P passes the ADF test, the second-order difference operation is stopped and the time series W = Δ d X=Δ d P is used as the subsequent modeling sequence; if the ADF test still cannot be passed after the first-order difference operation, the second-order difference is performed, and so on, until the ADF test is passed. The order of the difference when the ADF test is passed for the first time is d.
[0099] Exemplarily, W is used to represent the time series of data file M. When the usage frequency sequence P of data file M is a stationary sequence, W=P; when the usage frequency sequence of data file M is not a stationary sequence, W=Δ d X, where Δ d X=Δ d P, d is the order of the difference operation.
[0100] S104. For each data file, determine whether the time series of the data file has random characteristics. When the time series of the data file does not have random characteristics, determine the time series of the data file as a modeling sequence, and apply the modeling sequence and preset various model parameter combinations to perform model prediction processing on the data file to obtain the future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset.
[0101] It should be noted that each model parameter combination includes specific data of the model parameters p and q when performing model prediction processing.
[0102] The model prediction process is performed using the time series of each data file. In this process, the model parameters are continuously updated with the prediction results as a feedback mechanism. Finally, it can be determined whether the data file meets the future income conditions and the corresponding future income prediction results are generated. Furthermore, when the data file is likely to be used in the future, it means that the data file is likely to generate income in the future, indicating that the data file is valuable in the future. At this time, the future income prediction results that meet the future income conditions will be generated.
[0103] When it is predicted that the data file will have income in the future, or has the possibility of income in the future, it is determined that the data file meets the future income condition, and the corresponding future income prediction result is generated; when the future income prediction result indicates that the data file does not meet the future income condition, it indicates that the data file will not have income in the future. After the future income prediction result of the data file is generated, the data file characterized as the future income prediction result that meets the future income condition is determined as a data asset.
[0104] In the method provided by the embodiment of the present invention, the metadata information of each data file is obtained, and based on the metadata information of each data file, the usage number sequence of each data file is obtained, the usage number sequence includes the usage number of the data file in each preset first time interval, and the time series of each data file is determined based on the usage number sequence of each data file; for each data file, it is determined whether the time series of the data file has a random characteristic, when the time series of the data file does not have a random characteristic, the time series of the data file is determined as a modeling sequence, and the modeling sequence and the preset model parameter combinations are applied to the data file to perform model prediction processing to obtain the future income prediction result of the data file, and when the future income prediction result is characterized as meeting the future income condition, the data file is determined as a data asset. After obtaining the metadata information of the data file, the present invention extracts the usage number sequence of the data file from the metadata information, and then processes the usage number sequence to obtain the time series of the data file, and then applies the time series to perform model prediction processing on the data file, so as to obtain the future income prediction result that characterizes whether the data file meets the future income condition. By mining the metadata information of data files, we can obtain the usage frequency sequence used to identify whether a data file is a data asset, deeply mine the value of the relevant information of the data file, and identify whether a data file is a data asset through modeling. This provides enterprises with a solution for quickly identifying whether a data file is a data asset, helping enterprises to process data assets, reduce the workload of staff, and improve work efficiency.
[0105] After identifying data assets from a large amount of data using the solution provided by the present invention, the company can classify, grade, evaluate and price the data assets based on the present invention, which provides great convenience for the organization to manage data assets.
[0106] Figure 1 The identification scheme shown is to identify whether a data file is a data resource through model prediction processing. The present invention also provides another scheme for identifying data assets, specifically, as follows: obtaining metadata information of each data file; based on each metadata information, determining the data file that meets the preset priority identification rule among the data files as a data asset. The priority identification rule is used to identify whether the data file will have benefits in the future. The specific process can be referred to Figure 2 and Figure 3 The relevant description of this is that based on each metadata information, the data files that meet the preset priority identification rules in each data file are determined as data assets, which is equivalent to Figure 2 Based on the description information, the files to be identified that meet the preset priority identification specifications among the files to be identified are determined as the content of the data asset. For the specific process, refer to Figure 3 The metadata information here is equivalent to Figure 2 For more information, refer to Figure 2 and Figure 3 By setting priority identification rules, when a data file meets the priority identification rules, it is judged that the data file has benefits in the future, that is, the data file is judged to be a data asset. This solution can quickly determine whether a data file is a data asset, help enterprises manage data assets, and improve the work efficiency of enterprise staff.
[0107] for Figure 1 The process of obtaining metadata information of each data file in the scheme shown is described in detail. Figure 2 , is a flow chart of a method for obtaining metadata information of each data file provided by an embodiment of the present invention, and the specific process is as follows:
[0108] S201, determining each file to be identified.
[0109] The file to be identified is a file that needs to be identified as a data asset, and may be a data file to be identified.
[0110] S202: Obtain description information of each file to be identified.
[0111] When obtaining the description information of each file to be identified, the description information of the file to be identified can be obtained through an interface, or through an SQL statement.
[0112] The description information includes, but is not limited to, data file name, data file creation time, data file modification time, data lineage (upstream and downstream dependencies), data file usage records, etc.
[0113] S203. Based on the description information, determine the files to be identified that meet the preset priority identification rules among the files to be identified as data assets, determine the files to be identified that do not meet the priority identification rules as data files, and determine the description information of the data files as metadata information.
[0114] Preferably, the priority identification rules here are the same as the priority identification rules mentioned above, and are both used to identify whether a file will generate revenue in the future.
[0115] When the file to be identified meets the priority identification rule, it is determined that the file to be identified has benefits in the future. At this time, the file to be identified can be determined as a data asset. Otherwise, the file to be identified is determined as a data file, and the description information of the data file is determined as metadata information.
[0116] The priority recognition rules here can be used to perform preliminary recognition of the files to be recognized, and can also be understood as preliminary screening. Figure 2 The description content of determining the files to be identified that meet the preset priority identification rules among the files to be identified as data assets based on various descriptive information is applicable to the relevant description of determining the data files that meet the preset priority identification rules among the data files as data assets based on various metadata information above.
[0117] The embodiment of the present invention uses priority recognition rules to perform preliminary recognition on each file to be recognized, which can quickly screen out some files and reduce the pressure of subsequent recognition. The method of using priority recognition rules to recognize files is different from the method of subsequent recognition. A variety of different recognition methods are used to recognize files from multiple aspects, which can more comprehensively recognize files and avoid omissions.
[0118] Reference Figure 3 , which is a flowchart of using a priority recognition rule to recognize a file according to an embodiment of the present invention, is specifically described as follows:
[0119] S301. Determine the start time for data asset identification work and determine the priority identification time step.
[0120] The priority recognition time step can be set according to actual conditions.
[0121] S302: Determine a priority identification time interval based on the work start time and the priority identification time step.
[0122] The priority identification time interval is the time interval determined by backtracking based on the work start time and the priority identification time step. For example, if the work start time is April 10, 2024, the priority identification time step is 7 days, and the priority identification time interval is April 3 to 10, 2024.
[0123] S303: Determine the judgment time of each file to be identified based on each description information.
[0124] Preferably, the determination time can be the development time of the file to be identified or the modification time of the file to be identified. When the file to be identified only has the development time, the development time is used as the determination time; if the file to be identified has both the development time and the modification time, the latest modification time is used as the determination time.
[0125] S304. For each file to be identified, determine whether the determination time of the file to be identified is within the priority identification time interval; if the determination time of the file to be identified is within the priority identification time interval, execute S305; if the determination time of the file to be identified is not within the priority identification time interval, execute S306.
[0126] When the determination time of the file to be identified is within the priority identification time interval, it means that the file is developed for the use of the enterprise. At this time, it can be determined that the file will bring benefits to the enterprise in the future, so S305 is executed; otherwise, S306 is executed.
[0127] S305: Determine whether the file to be identified meets the priority identification rule, and determine the file to be identified as a data asset.
[0128] S306: Determine whether the file to be identified does not satisfy the priority identification rule.
[0129] Furthermore, the value of data is delayed, and data files will not be put into use immediately after production. The present invention uses the priority identification rule to propose a concept of "direct confirmation period", that is, the priority identification time interval is used for judgment, and the judgment time of the file can be directly confirmed as a data asset within this period. According to the situation of each subject, the duration of the period is set to L (the period here can be understood as the priority identification time step mentioned above), and the time node for carrying out data asset identification work is T0 (i.e., the judgment time mentioned above). The data files developed and completed in the L days from T0 are considered to be expected to bring benefits to the enterprise before they are invested in development. In practice, there is a lack of sufficient basis to refute this expectation, and then directly confirmed as data assets. When determining whether a data file belongs to the direct confirmation period, the larger of the creation time and modification time of the data file in the information acquisition link is taken and compared with the "direct confirmation period". Preferably, the files determined as data assets can be represented as a set FD, which contains multiple files determined as data assets, and the files not determined as data assets are represented as a set FS, which contains multiple files not determined as data assets, and the subsequent identification process can be executed for the set FS.
[0130] The present invention uses priority identification rules to identify files that are expected to be developed in order to generate revenue for the enterprise, and can quickly identify some data assets to reduce subsequent identification pressure.
[0131] Reference Figure 4 , which is a flow chart of using the time series of data files to identify whether a data file is a data asset according to an embodiment of the present invention. The flow chart can be applied to each data file, and is specifically described as follows:
[0132] S401, judging whether the time series of the data file has random characteristics; when the time series of the data file does not have random characteristics, executing S402; when the time series of the data file has random characteristics, executing S411.
[0133] It should be noted that when determining whether the time series of a data file has random characteristics, it can be determined whether the time series is a white noise sequence. When the time series is a white noise sequence, it is determined that the time series has random characteristics. When the time series is not a white noise sequence, it is determined that the time series does not have random characteristics. The present invention also supports the use of other methods to determine whether a time series has random characteristics. By determining whether a time series is a white noise sequence, the present invention can quickly determine whether a time series has random characteristics, speed up the recognition speed of data files, and improve the efficiency of the entire recognition process.
[0134] It should be noted that the white noise sequence is a random sequence and therefore has random characteristics. When a time series has random characteristics, it means that the data file corresponding to the time series may be used randomly in the future, and the possibility of future returns cannot be ruled out. Therefore, the white noise sequence meets the future return conditions. Therefore, when the time series is determined to be a white noise sequence, a future return prediction result will be generated that characterizes the data file meeting the future return conditions. That is, when the time series is a white noise sequence, the data file is identified as a data asset.
[0135] Furthermore, in the solution provided in the embodiment of the present invention, when the time series has random characteristics, the data files corresponding to the time series may be used randomly in the future, and it is impossible to accurately judge whether the data files will be used in the future. In addition to determining the data files corresponding to the time series as data assets as mentioned above, the data files corresponding to the time series can also be directly eliminated, thereby eliminating the influence of randomness on the identification process and improving the accuracy of identifying data assets.
[0136] The process of determining whether the time series of a data file is a white noise series is as follows:
[0137] Since time series data may contain noise, before using time series data for modeling, the Ljung Box test can be used to determine whether the time series is a white noise series.
[0138] The formula used for the test statistic is as follows:
[0139]
[0140] The LB statistic also approximately follows a chi-square distribution with g degrees of freedom. Where t is the number of observation periods of the sequence, that is, the number of the first time intervals mentioned above, g is the specified maximum delay order, and ρ k is the estimated value of the autocorrelation coefficient of the k-th lag.
[0141] ρ k It can be calculated specifically by the following formula:
[0142]
[0143] in, W1 to W t The mean value of W t is the tth value in the time series W.
[0144] The test first assumes that the sequence is white noise, and obtains its significance P value based on the above statistics. If the null hypothesis cannot be rejected, the sequence is considered to be a white noise sequence. If the null hypothesis is rejected, it is considered to be a non-white noise sequence.
[0145] When W is a non-white noise sequence, W is used for subsequent modeling, that is, the time series of the data file is determined as the modeling sequence; when W is a white noise sequence, the sequence is random, the data file will be used randomly, and the data file can be identified as a data asset.
[0146] S402: Determine the time series as a modeling sequence.
[0147] S403: construct a prediction model, sort the combinations of model parameters, and obtain a combination sequence.
[0148] The model parameter combination includes the model parameters of the prediction model, which can be used to subsequently update the model parameters of the prediction model.
[0149] There are many ways to sort the various model parameter combinations. They can be sorted randomly or according to different application stages. For example, the model parameter combinations belonging to the early stage are sorted first, followed by the model parameter combinations belonging to the preliminary prediction stage, and then the model parameter combinations belonging to the deep prediction stage.
[0150] When building a forecasting model, the Auto Regressive Integrated Moving Average Model (ARIMA) is the most common method for time series forecasting. It is also a classic model that has been tested in practice. It has been applied in many disciplines such as statistics and econometrics. This model can be used as a forecasting model. The specific form of the forecasting model is as follows: t =μ+φ1W t-1 +φ2W t-2 +...+φ p W t-p +ε t +θ1ε t-1 +...+θ q ε t-q ;
[0151] Among them, μ is the constant term, φ is the autoregressive coefficient, and p is its order; θ is the moving average coefficient, q is the lag order of the error term; ε t is a white noise sequence.
[0152] S404: Determine the first model parameter combination in the combination sequence as the target model parameter combination.
[0153] S405. Update the model parameters of the prediction model based on the target model parameter combination, and use the prediction model after the updated model parameters to process the modeling sequence, and output a predicted usage number sequence, wherein the predicted usage number sequence includes the predicted usage times of the data file in each preset second time interval.
[0154] Preferably, the model parameters of the prediction model are p and q, and the predicted usage frequency sequence output by the prediction model is the usage prediction result of the data file. Each second time interval is a preset time interval, and each time interval can be set according to actual needs.
[0155] S406, determine whether there is a predicted usage count greater than 0 in the predicted usage count sequence; when there is a predicted usage count greater than 0 in the predicted usage count sequence, execute S407; when there is no predicted usage count greater than 0 in the predicted usage count sequence, execute S408.
[0156] S407 , generating a future income prediction result representing that the data file satisfies the future income condition, and executing S411 .
[0157] S408, determine whether the target model parameter combination is the last model parameter combination in the combination sequence; when the target model parameter combination is not the last model parameter combination in the combination sequence, execute S409; when the target model parameter combination is the last model parameter combination in the combination sequence, execute S410.
[0158] S409: Use the next model parameter combination after the target model parameter combination in the combination sequence as a new target model parameter combination, and return to execute S405.
[0159] S410: Generate a future income prediction result indicating that the data file does not meet the future income condition.
[0160] S411. Identify the data file as a data asset.
[0161] The prediction model of the present invention is purpose-oriented when predicting a data file. When there is no predicted usage count greater than 0 in the output predicted usage count sequence, the model parameters of the prediction model are updated based on the model parameter combination that has not been applied, and then the data file is predicted again until there is a predicted usage count greater than 0 in the output predicted usage count sequence, and a future profit prediction result is generated that characterizes that the data file meets the future profit condition, or until all model parameter combinations are traversed and there is still no predicted usage count greater than 0 in the output predicted usage count sequence, a future profit prediction result is generated that characterizes that the data file does not meet the future profit condition, thereby realizing the identification of whether the data file is a data asset using the prediction model.
[0162] Furthermore, each model parameter combination is a model parameter combination at different stages of the model, and each model parameter combination is different. For example, there are model parameter combinations when the model is initially established, model parameter combinations when the initial retrieval stage is performed, and model parameter combinations when the deep retrieval stage is performed.
[0163] In the present invention, when determining the parameters p and q of the model, the present invention is purpose-oriented and focuses on the identification of data assets. In fact, it is not necessary to find the optimal model to achieve identification, so a p and q selection method with prediction results as a feedback mechanism is proposed. It can be divided into the following steps:
[0164] 1. First, conduct simple modeling, select AR model (i.e. select p=1, q=0) and MA model (i.e. select p=0, q=1) for trial. When the fitting result of any one of the two models is recognized as a valid model in terms of the performance of indicators such as the coefficient of determination and the F test, and the result of future period prediction through the valid model is greater than 0, it means that the data file is predicted to be used in the future, and the data file can be identified as a data asset. After achieving the purpose of this identification, stop the subsequent modeling, and no more work is required to conduct grid search for other combinations of p and q. If both model selections fail to achieve the purpose of identifying as a data asset, then conduct a subsequent search.
[0165] 2. Secondly, conduct a preliminary search. Studies have shown that in most cases, p≤2 and q≤2 can meet the modeling needs. Later, adjustments can be made based on empirical values in the field of data asset identification. After excluding simple modeling steps within this range, other combinations of p and q are modeled. When the fitting results of any model are recognized as valid in terms of the performance of indicators such as the coefficient of determination and the F test, and when the results of future predictions are made using this valid model, there is a situation where the result is greater than 0, which means that the data file is predicted to be used in the future, and the data file can be identified as a data asset. After achieving the purpose of this identification, subsequent modeling is stopped, and no further work is required to conduct grid searches for other combinations of p and q. If all model selections fail to achieve the purpose of identification as a data asset, a subsequent in-depth search will be conducted.
[0166] 3. Once again, conduct a deep search. The upper limit value q' of q and the upper limit value p' of p can be determined based on expert judgment. Later, adjustments can be made based on empirical values in the field of data asset identification. After excluding the aforementioned modeled situations within this range, other combinations of p and q are modeled. When the fitting results of any model are determined to be valid in terms of the performance of indicators such as the coefficient of determination and the F test, and the result of future predictions using this valid model is greater than 0, it means that the data file is predicted to be used in the future, and the data file can be identified as a data asset. If it is still not effectively predicted that the data file will be used in the future, then the data file will not be identified as a data asset.
[0167] In the process of determining p and q, the logic of modeling and prediction has been mentioned. After establishing an effective model, use this model to analyze the usage frequency series P t Predict the future period. Preset the number of future periods f (the future period here can be understood as the second time interval). When setting the future period, it can be set according to expert judgment or industry experience. According to the model, the prediction result set {P t+1 ,P t+2 ,…,P t+f} (which can be understood as the predicted usage count sequence above). When any value in the prediction result set is greater than 0, it means that the data file is predicted to be used in the future, and the data file can be identified as a data asset.
[0168] Figure 4 The process shown can be seen as the process of using the model for prediction and recognition, refer to Figure 5 , which is a flowchart of using a model to predict a file provided in an embodiment of the present invention, and is specifically described as follows.
[0169] This example illustrates the process of predicting data files by the prediction model at different stages. Different combinations of model parameters are applied at different stages, and the values of the model combination parameters at different stages are restricted by the value range of the model parameters.
[0170] After obtaining the time series of the data file, the AR model and the MA model are used for simple modeling. The p and q parameters in the AR model and the MA model are set based on the combination of each model parameter. Then, the model with the set parameters is used to predict the time series of the data file, and the first predicted usage sequence is output. Based on the first predicted usage sequence, it is determined whether the data file is expected to be used. When the predicted number of uses in the first predicted usage sequence is greater than 0, it is determined that the data file is expected to be used, and the data file can be determined as a data asset. When the predicted number of uses in the first predicted usage sequence does not exceed 0, the ARIMA model is used for preliminary prediction. At this time, p≤2, q≤2 in the ARIMA model. Each first model parameter combination is determined based on the value range of p and q, and the values of p and q are different in different first model parameter combinations. Furthermore, the determined first model parameter combinations do not include model parameter combinations that have been used for modeling. In other words, the obtained first model parameter combinations are all model parameter combinations that are still used for modeling. For example, before determining the first model parameter combination based on the range of p≤2, q≤2, model parameter combinations such as (0,1) and (1,0) have been used for modeling. Therefore, when determining the first model parameter combinations based on p≤2, q≤2, it is necessary to eliminate the model parameter combinations such as (0,1) and (1,0) that have been modeled to obtain the first model parameters. The obtained first model parameter combinations are (1,1), (1,2), (2,1), (2,2), etc., which are not explained in detail here. Based on each first model parameter combination, the ARIMA model is used to process the time series of the data file and then a second predicted usage sequence corresponding to each first model parameter combination is output. Based on each second predicted usage sequence, it is determined whether the data file is expected to be used. When there is a predicted usage number greater than 0 in each second predicted usage sequence, it is determined that the data file is expected to be used, and the data file can be determined as a data asset. When there is no predicted usage number greater than 0 in each second predicted usage sequence, the parameters in the ARIMA model are changed to make p≤p′, q≤q′. Based on the value range of p and q at this time, each second model parameter combination is determined. Each second model parameter combination does not contain a model parameter combination that has been used for modeling. In other words, each second model parameter combination obtained is a model parameter combination that is still used for modeling. Based on each second model parameter combination, the ARIMA model is used to make a deep prediction. The ARIMA model processes the time series of the data file and outputs a third predicted usage sequence corresponding to each second model parameter combination. Based on each third predicted usage sequence, it is determined whether the data file is expected to be used. When there is a predicted number of uses greater than 0 in each third predicted usage sequence, it is determined that the data file is expected to be used, and the data file can be determined as a data asset. When there is no predicted number of uses greater than 0 in each third predicted usage sequence, the data file is not determined as a data asset.
[0171] It should be noted that when p and q in the model take different values, the model will output different prediction usage sequences after processing the time series of the same data file.
[0172] In the method provided by the present invention, by using the established model to identify the data file, it is possible to quickly and effectively identify whether the data file is a data asset, thereby reducing the workload of the staff.
[0173] Further, after Figure 4 After the identification process shown, the remaining data files that have not been identified as data assets may still contain data files that will generate revenue in the future. In order to avoid omissions and misjudgments of data assets, the present invention provides a process for supplementary identification based on features to further identify the remaining data files. For specific processes, refer to Figure 6 The specific instructions are as follows:
[0174] S501. Determine each data file that has not been determined as a data asset as a first target data file, and determine a data file that is determined as a data asset through model prediction processing as a second target data file.
[0175] S502: Determine a feature similarity sequence for each first target data file based on the usage count sequence of each first target data file and the usage count sequence of each second target data file, wherein the feature similarity sequence includes feature similarities between the first target data file and each second target data file.
[0176] The usage count sequence of the second target data file can be regarded as a feature sequence of the second target data file, and the feature sequence includes the features of the second target data file as a data asset.
[0177] The usage frequency sequence of the first target data file can be regarded as a feature sequence of the first target data file. By performing feature mining on the feature sequence, it is determined whether there is a data file in the feature sequence that can be used as a feature of a data asset.
[0178] For each first target data file, a similarity operation is performed on the usage count sequence of the first target data file and the usage count sequence of each second target data file to obtain the feature similarity between the first target data file and each second target data file, and a feature similarity sequence of the first target data file is obtained based on each feature similarity.
[0179] S503: For each first target data file, when there is a feature similarity sequence of the first target data file that meets a preset threshold requirement, the first target data file is determined as a data asset.
[0180] It should be noted that the threshold requirement can be set according to actual needs, and the threshold requirement includes a preset threshold. Exemplarily, when the preset threshold is an upper limit value, if any feature similarity in the feature similarity sequence of the first target data file is less than the preset threshold, then it is determined that the feature similarity sequence of the first target data file meets the threshold requirement, otherwise, it is determined that the feature similarity sequence of the first target data file does not meet the threshold requirement; when the preset threshold is a lower limit value, if any feature similarity in the feature similarity sequence of the first target data file is greater than the preset threshold, then it is determined that the feature similarity of the first target data file meets the threshold requirement, otherwise, it is determined that the feature similarity sequence of the first target data file does not meet the threshold requirement.
[0181] In order to specifically illustrate the process of supplementary identification based on features, the present invention provides an example for illustration, as follows:
[0182] After the modeling and identification process, the data files are divided into two categories. One category is the data files identified as data assets. Assume that there is a set FT, FT = {FT1, FT2, ..., FT u}, the data files in this set are all third target data files; one type is data files that are not identified as data assets, assuming there are v of them, represented as a set FF, FF = {FF1, FF2, ..., FF v}, the data files in this set are all second target data files.
[0183] Correspondingly, in the data file set FT, any data file has a corresponding usage count sequence P, which is re-expressed as PT here. u The usage frequency sequence is expressed as PT u , the data file FT u The time period involved (i.e. the first time interval mentioned above) is denoted as tu, then the sequence in, Indicates the number of times the data file FTu is used in the time period tu. In the data file set FF, any data file has a corresponding usage count sequence P, which is re-expressed as PF here. v The usage frequency sequence is expressed as PF v , the time period involved in the data file FFv is represented as tv, then the sequence in, Indicates data file FT v The number of times tv is used in the time period.
[0184] Here, the usage count sequence PT corresponding to the data files in the set FT is used as the feature sequence that meets the requirements for identification as data assets. 1To PT u There are a total of u feature sequences that meet the requirements for identification as data assets. Then, the usage frequency sequences PF of the data files in the set FF are similarly calculated one by one with the usage frequency sequences PT corresponding to the data files in the set FT.
[0185] The process of similarity calculation is as follows:
[0186] When calculating the similarity between two sequences, there are two situations: the sequences are of equal length and the sequences are of unequal length. v and the characteristic sequence PT u Taking similarity calculation as an example, the length of the former is tv and the length of the latter is tu. The specific calculation is as follows:
[0187] When the sequences are of equal length, i.e., tv = tu, PF v and PT u The cosine similarity method can be used to calculate the similarity. The two sequences are regarded as two vectors, and the similarity is cos(η)=PF v *PT v *|PF v |*|PT v |; When cos(η)>ξ, PF v Corresponding data file FF v Identified as a data asset; when ξ=1, it means PF v and PT u To be exactly the same, however, for the same sequence in the modeling process, there will be the same model judgment results, so there will be no PT u Identified as being used in the future, PF v It is identified as not being used in the future, that is, there will be no sequence PF when ξ=1 v No data files will be identified for future use, and no data files will be newly identified as data assets.
[0188] Preferably, when the sequences are of equal length, in addition to the cosine similarity method, existing distance formulas such as Euclidean distance, Manhattan distance, and Chebyshev distance can also be used as a substitute. When using a distance formula as a substitute, when the distance is less than ξ, PF v The corresponding data file FFv is identified as a data asset.
[0189] When the sequences are of unequal length, i.e. tv≠tu, PF v and PT u The DTW method can be used to calculate the distance. According to the principle of dynamic programming, the time series can be "distorted" to make necessary dislocations in the time series and calculate the most appropriate distance. The shortest distance calculated is expressed as MD. When MD < ξ, PF vCorresponding data file FF v Here, special attention should be paid to the case where tv>tu. and PF can only be used when the distance between the misaligned and aligned positions is less than ξ. v Corresponding data file FF v as data assets.
[0190] Furthermore, for the setting of the similarity threshold, the similarity threshold ξ can be determined based on expert judgment or industry experience. When the similarity between any one of the usage frequency sequences PF and any feature sequence in each usage frequency sequence PT meets the threshold ξ requirement, the corresponding data file to be calculated is identified as a data asset. v , and use its usage frequency sequence PF v The similarity calculations are performed on the u feature sequences corresponding to the data file set FT. When the similarity calculation result with a feature sequence meets the threshold ξ requirement, the data file FF corresponding to the calculation is v Identify as a data asset.
[0191] When the threshold requires that the similarity be judged as completely similar, no data files will be newly identified as data assets, which means that the recognition results of the priority recognition rules and models are still used, and the step of supplementary recognition based on features is essentially skipped. Different thresholds can be set for different application scenarios of data asset recognition. After all, in the data asset confirmation scenario, if it is not confirmed this time, it can still be confirmed later, while in the data asset impairment scenario, once the impairment is confirmed, it cannot be reversed.
[0192] The present invention further identifies the remaining data files that have not been identified as data assets by providing a feature-based supplementary identification process, thereby avoiding missing data assets.
[0193] Reference Figure 7 , is another data asset identification flow chart provided by an embodiment of the present invention, and is specifically described as follows:
[0194] Figure 7 The data asset identification process shown includes a data asset identification path system of "priority identification" → "modeling prediction" → "supplementary identification", which identifies data assets from multiple aspects and comprehensively identifies data assets to avoid missing data assets.
[0195] like Figure 7As described, first obtain the information of the data file to be identified, determine whether there is directly available metadata information in the obtained information, if not, parse the required metadata information through regular expressions or other tools to obtain neat metadata information, use the obtained metadata information for priority identification, determine whether the data file meets the priority identification rules, if yes, determine that the data file is a data asset, if not, perform time series data processing based on the metadata information, then establish an ARIMA model, and use the ARIMA model for prediction. The specific process of using the model for prediction can refer to the above description and will not be repeated here. If it is predicted that the data file will be used in the future, the data file will be determined as a data asset. Otherwise, a supplementary identification process is performed based on the usage frequency sequence of metadata information statistics, and the usage frequency sequence is used to calculate the similarity. When the calculated similarity meets the threshold requirement, the data file is identified as a data asset, otherwise, the data file is identified as not a data asset.
[0196] Currently, when identifying data assets, there is a method of confirming data assets based on the point in time of value conversion. However, there is a time misalignment between the point in time of value conversion and the actual situation, which leads to differences in the scope of data assets at the time of confirmation. From the perspective of the concept of data assets, it is the expected ability to generate revenue, rather than the fact that revenue has already been generated, which leads to inaccurate identification results.
[0197] The present invention approaches this issue from the perspective of judging whether data resources are expected to bring economic benefits to the enterprise in the future. The core is to judge whether data files can be used in the future to determine whether they can be expected to bring benefits to the company. A practical data asset identification method and system are proposed, which overall builds a data asset identification path system of "priority identification" → "modeling prediction" → "supplementary identification", and innovatively provides a hyper-parameter selection method in terms of "modeling prediction" with the goal of achieving business goals. The data asset identification method and system of the present invention can serve the scenarios of data asset confirmation and impairment, realize the confirmation of data assets to assist in the initial measurement of data assets, process the impairment of data assets to assist in the subsequent measurement of data assets, and promote the financial processing of data assets through efficient and convenient data asset identification.
[0198] This invention proposes a feasible data asset identification method, and provides a clear technical path and detailed description to support systematic implementation, filling the technical gap in the data asset identification business field. This method and system can serve the scenarios of data asset confirmation and impairment, efficiently and conveniently implement data asset identification, and thus accelerate the financial processing of data assets, helping to activate and grow the data factor market.
[0199] The present invention proposes a data asset identification system from "priority identification" → "modeling prediction" → "supplementary identification". The rule-based "priority identification" can ensure that the data files that are recently formed and have a short time period and are not suitable for modeling can be identified as data assets. The model-based "modeling prediction" can efficiently and conveniently realize the identification and prediction of large batches of data files. The feature-based "supplementary identification" can additionally identify data files with similar features as data assets. Through these three processes, data assets can be more fully identified, and the confirmation and impairment of data assets can be carried out more carefully.
[0200] In the process of constructing the ARIMA model, the present invention is purpose-oriented and focuses on the identification of data assets. It abandons the conventional Akaike information criterion and Bayesian information criterion to find the optimal model to determine p and q, and proposes to determine p and q with the prediction results as a feedback mechanism. At the same time, in order to reduce the exploration workload of model construction, a modeling path of "simple modeling" → "preliminary search" → "deep search" is proposed. When asset identification can be achieved through simple modeling, there is no need to promote subsequent grid search for modeling. When asset identification can be achieved through preliminary search, there is no need to promote subsequent deep search for modeling. In the process of feature-based supplementary identification, the present invention proposes feasible calculation methods for the cases of equal length and unequal length sequences. For the case of unequal length sequences tv>tu, combined with business meanings and needs, the requirements for time series misalignment are further restricted to prevent the usage frequency sequence of the data file to be confirmed from being identified as a data asset after the distance <ξ is aligned with the feature sequence in the early stage.
[0201] and Figure 1 Corresponding to the method shown in the figure, the present invention also provides a data recognition device, which is used to support Figure 1 Specific implementation of the method shown.
[0202] Reference Figure 8 , is a schematic diagram of the structure of a data identification device provided by an embodiment of the present invention, and is specifically described as follows:
[0203] A first acquisition unit 601 is used to acquire metadata information of each data file;
[0204] A second acquisition unit 602 is used to acquire a usage count sequence of each of the data files based on metadata information of each of the data files, wherein the usage count sequence includes a usage count of the data file in each preset first time interval;
[0205] A first determining unit 603 is used to determine a time sequence of each of the data files based on a usage count sequence of each of the data files;
[0206] The identification unit 604 is used to determine, for each of the data files, whether the time series of the data file has random characteristics. When the time series of the data file does not have random characteristics, the time series of the data file is determined as a modeling sequence, and the modeling sequence and the preset model parameter combinations are used to perform model prediction processing on the data file to obtain a future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset.
[0207] After obtaining the metadata information of the data file, the present invention extracts the usage frequency sequence of the data file from the metadata information, processes the usage frequency sequence to obtain a time series, and then applies the time series to identify whether the data file is a data asset, and obtains a future income prediction result that characterizes whether the data file meets the future income conditions. By performing data mining on the metadata information of the data file, the usage frequency sequence used to identify whether the data file is a data asset is obtained, the value of the relevant information of the data file is deeply mined, and the data file is identified through modeling whether it is a data asset, providing a solution for enterprises to quickly identify whether the data file is a data asset, helping enterprises to process data assets, reducing the workload of staff, and improving work efficiency.
[0208] In another embodiment provided by the present invention, the first acquisition unit of the above-mentioned device includes:
[0209] A first determination subunit, used to determine each file to be identified;
[0210] A first acquisition subunit is used to acquire description information of each file to be identified;
[0211] The second determination subunit is used to determine, based on each of the description information, the files to be identified that meet the preset priority identification rules among the files to be identified as data assets, and determine the files to be identified that do not meet the priority identification rules as data files, and determine the description information of the data files as metadata information.
[0212] In another embodiment provided by the present invention, the second determining subunit of the above-mentioned device includes:
[0213] The first determination module is used to determine the time for carrying out data asset identification work and determine the priority identification time step;
[0214] A second determination module is used to determine a priority identification time interval based on the work start time and the priority identification time step;
[0215] A third determination module, used to determine the determination time of each of the to-be-identified files based on each of the description information;
[0216] The fourth determination module is used to determine, for each of the files to be identified, whether the determination time of the file to be identified is within the priority identification time interval; if the determination time of the file to be identified is within the priority identification time interval, it is determined that the file to be identified satisfies the priority identification rule, and the file to be identified is determined as a data asset; if the determination time of the file to be identified is not within the priority identification time interval, it is determined that the file to be identified does not satisfy the priority identification rule.
[0217] In another embodiment provided by the present invention, the identification unit of the above-mentioned device includes:
[0218] The first judgment subunit is used to judge whether the time series is a white noise sequence; when the time series is a white noise sequence, determine that the time series has random characteristics; when the time series is not a white noise sequence, determine that the time series does not have random characteristics.
[0219] In another embodiment provided by the present invention, the identification unit of the above-mentioned device includes:
[0220] A construction subunit is used to construct a prediction model, and to sort the combinations of the model parameters to obtain a combination sequence;
[0221] A third determining subunit, configured to determine the first model parameter combination in the combination sequence as a target model parameter combination;
[0222] A processing subunit, configured to update the model parameters of the prediction model based on the target model parameter combination, and use the prediction model after the model parameters are updated to process the modeling sequence, and output a predicted usage count sequence, wherein the predicted usage count sequence includes the predicted usage counts of the data file in each preset second time interval;
[0223] A first generating subunit is used for generating a future revenue prediction result indicating that the data file meets a future revenue condition when there is a predicted usage number greater than 0 in the predicted usage number sequence;
[0224] A second judging subunit is used to judge whether the target model parameter combination is the last model parameter combination in the combination sequence when there is no predicted usage count greater than 0 in the predicted usage count sequence; when the target model parameter combination is not the last model parameter combination in the combination sequence, use the next model parameter combination located at the target model parameter combination in the combination sequence as a new target model parameter combination, and return to the processing subunit to execute the steps of updating the model parameters of the prediction model based on the target model parameter combination, and using the prediction model after the updated model parameters to process the modeling sequence, and output the predicted usage count sequence;
[0225] The second generating subunit is used for generating a future profit prediction result indicating that the data file does not meet the future profit condition when the target model parameter combination is the last model parameter combination in the combination sequence.
[0226] In another embodiment provided by the present invention, the above-mentioned device further includes:
[0227] A second determination unit, configured to determine each data file that has not been determined as a data asset as a first target data file, and to determine a data file that has been determined as a data asset through model prediction processing as a second target data file;
[0228] a third determining unit, configured to determine a feature similarity sequence of each first target data file based on a usage count sequence of each first target data file and a usage count sequence of each second target data file, wherein the feature similarity sequence includes feature similarities between the first target data file and each second target data file;
[0229] The fourth determining unit is configured to determine, for each of the first target data files, the first target data file as a data asset when a feature similarity sequence of the first target data file satisfies a preset threshold requirement.
[0230] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data asset identification method.
[0231] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Fig. 9 As shown, it specifically includes a memory 801 and one or more instructions 802, wherein the one or more instructions 802 are stored in the memory 801 and are configured to be executed by one or more processors 803 to execute the one or more instructions 802 to perform the above-mentioned data asset identification method.
[0232] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0233] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.
[0234] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0235] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0236] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data asset identification method, characterized in that: include: Get metadata information of each data file; Based on the metadata information of each of the data files, obtaining a usage count sequence of each of the data files, wherein the usage count sequence includes the usage count of the data file in each preset first time interval; Determine a time sequence of each of the data files based on a usage count sequence of each of the data files; For each of the data files, determine whether the time series of the data file has random characteristics. When the time series of the data file does not have random characteristics, determine the time series of the data file as a modeling sequence, and apply the modeling sequence and preset various model parameter combinations to perform model prediction processing on the data file to obtain the future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset.
2. The method according to claim 1, characterized in that The step of obtaining metadata information of each data file includes: Determine each file to be identified; Obtaining description information of each file to be identified; Based on each of the description information, the files to be identified that meet the preset priority identification rules are determined as data assets, and the files to be identified that do not meet the priority identification rules are determined as data files, and the description information of the data files is determined as metadata information.
3. The method according to claim 2, characterized in that The step of determining, based on each of the description information, the files to be identified that meet a preset priority identification rule among the files to be identified as data assets includes: Determine when to conduct data asset identification work and prioritize identification time steps; Determine a priority identification time interval based on the work start time and the priority identification time step; Based on each of the description information, determining a determination time for each of the files to be identified; For each file to be identified, determine whether the judgment time of the file to be identified is within the priority identification time interval. If the judgment time of the file to be identified is within the priority identification time interval, determine that the file to be identified meets the priority identification rule, and determine the file to be identified as a data asset; if the judgment time of the file to be identified is not within the priority identification time interval, determine that the file to be identified does not meet the priority identification rule.
4. The method according to claim 1, characterized in that: The determining whether the time series of the data file has a random characteristic comprises: Determining whether the time series is a white noise sequence; When the time series is a white noise sequence, determining that the time series has a random characteristic; When the time series is not a white noise sequence, it is determined that the time series does not have a random characteristic.
5. The method according to claim 1, characterized in that The applying the modeling sequence and the preset model parameter combinations to perform model prediction processing on the data file to obtain the future income prediction result of the data file includes: Constructing a prediction model, and sorting the combinations of the model parameters to obtain a combination sequence; Determine the first model parameter combination in the combination sequence as the target model parameter combination; updating the model parameters of the prediction model based on the target model parameter combination, and processing the modeling sequence using the prediction model after the model parameters are updated, and outputting a predicted usage count sequence, wherein the predicted usage count sequence includes the predicted usage counts of the data file in each preset second time interval; When the predicted usage times sequence contains a predicted usage times greater than 0, generating a future revenue prediction result indicating that the data file meets the future revenue condition; When there is no predicted usage count greater than 0 in the predicted usage count sequence, determining whether the target model parameter combination is the last model parameter combination in the combination sequence; When the target model parameter combination is not the last model parameter combination in the combination sequence, the next model parameter combination located at the target model parameter combination in the combination sequence is used as a new target model parameter combination, and the step of returning to execute the updating of the model parameters of the prediction model based on the target model parameter combination, and using the prediction model after the updated model parameters to process the modeling sequence, and outputting a prediction usage number sequence; When the target model parameter combination is the last model parameter combination in the combination sequence, a future profit prediction result is generated, which indicates that the data file does not meet the future profit condition.
6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Determine each data file that is not determined to be a data asset as a first target data file, and determine the data file that is determined to be a data asset through model prediction processing as a second target data file; Based on the usage count sequence of each of the first target data files and the usage count sequence of each of the second target data files, determining a feature similarity sequence of each of the first target data files, wherein the feature similarity sequence includes feature similarities between the first target data file and each of the second target data files; For each of the first target data files, when there is a feature similarity sequence of the first target data file that meets a preset threshold requirement, the first target data file is determined as a data asset.
7. A data asset identification device, characterized in that: include: A first acquisition unit, used to acquire metadata information of each data file; A second acquisition unit, configured to acquire a usage count sequence of each of the data files based on metadata information of each of the data files, wherein the usage count sequence includes a usage count of the data file in each preset first time interval; A first determining unit, configured to determine a time sequence of each of the data files based on a sequence of usage times of each of the data files; An identification unit is used to determine, for each of the data files, whether the time series of the data file has random characteristics. When the time series of the data file does not have random characteristics, the time series of the data file is determined as a modeling sequence, and the modeling sequence and a combination of preset model parameters are used to perform model prediction processing on the data file to obtain a future income prediction result of the data file. When the future income prediction result is characterized as satisfying the future income condition, the data file is determined as a data asset.
8. The device according to claim 7, characterized in that The first acquisition unit includes: A first determination subunit, used to determine each file to be identified; A first acquisition subunit is used to acquire description information of each file to be identified; The second determination subunit is used to determine, based on each of the description information, the files to be identified that meet the preset priority identification rules among the files to be identified as data assets, and determine the files to be identified that do not meet the priority identification rules as data files, and determine the description information of the data files as metadata information.
9. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the data asset identification method according to any one of claims 1 to 6.
10. An electronic device, characterized in that: It comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the data asset identification method as described in any one of claims 1-6.
Citation Information
Patent Citations
Training sample generation method, machine learning model training method and related devices
CN112364077A
Method and device for predicting activity delivery information and electronic equipment
CN115293821A
