Storage management system based on intellectual property data

Abstract

The invention belongs to the technical field of data storage management, and discloses an intellectual property data-based storage management system, which comprises the following steps of: firstly, acquiring comprehensive data, preprocessing, and constructing a comprehensive feature set; secondly, according to the comprehensive feature set, repeated data are detected, and repeated data items are identified; then, the influence of the repeated data items is evaluated and processed; and finally, in the data item processing process, data among multiple nodes are kept consistent, and abnormal states are monitored in real time, so that quick response and problem solving are facilitated, the intellectual property data management efficiency and quality are improved, the robustness and reliability of the system are enhanced, and more reliable and convenient services are provided for users.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage management, and more specifically, to a storage management system based on intellectual property data. Background Art

[0002] Intellectual property data includes important information such as patents, trademarks, and copyrights. Due to the diversity and complexity of intellectual property data, traditional data management systems face many challenges when processing this information, such as data duplication and inconsistency. In particular, when multiple nodes maintain the same data at the same time, it becomes more difficult to maintain data consistency and integrity. In addition, intellectual property data often contains a large amount of text content. How to effectively extract features and perform similarity matching to identify duplicate or related data entries is also a problem that needs to be solved urgently.

[0003] The current intellectual property data storage and management systems on the market mainly solve the problems of data management and duplicate data processing in the following ways: using advanced database technology and indexing strategies to improve search efficiency and reduce data redundancy, but this method has limited processing capabilities for unstructured text data; using machine learning technologies such as natural language processing (NLP) to automatically identify and process duplicate data, but the computational cost is high and model training requires a large amount of high-quality sample data. Although these methods have alleviated the problem to a certain extent, there are still problems such as low processing efficiency and poor flexibility. In particular, in terms of cross-node data consistency maintenance, existing solutions usually lack effective mechanisms to ensure real-time synchronization and exception handling.

[0004] In view of this, the present invention proposes a storage management system based on intellectual property data to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a storage management system based on intellectual property data, comprising:

[0006] Data processing module: used to obtain comprehensive data, perform preprocessing, and build a comprehensive feature set;

[0007] Data detection module: used to detect duplicate data and identify duplicate data entries based on a comprehensive feature set;

[0008] Impact Assessment Module: used to assess the impact of duplicate data entries and handle them;

[0009] Data management module: used to maintain data consistency among multiple nodes during data entry processing and to monitor abnormal conditions in real time.

[0010] Furthermore, the comprehensive data includes basic information, legal information, technical information and application areas;

[0011] Basic information includes title, abstract, keywords and categories;

[0012] Legal information includes application number, application date, grant number, grant date, right holder, agent, and legal status;

[0013] Technical information includes technical field, invention content and claims.

[0014] Furthermore, the preprocessing method includes data cleaning, data standardization and data conversion;

[0015] The method of constructing the comprehensive feature set is:

[0016] Dividing the text data in the comprehensive data into long text data and short text data according to a preset text length threshold;

[0017] Convert all long text data into long text vectors and concatenate them into text vector features in sequence;

[0018] Extract and convert numerical data, short text data, and categorical data into numerical features from comprehensive data;

[0019] The text vector features and numerical features are concatenated into a comprehensive feature set.

[0020] Furthermore, the method of converting all long text data into long text vectors includes:

[0021] Perform word segmentation, stop word removal, stem extraction and lemmatization on long text data to form a vocabulary data set for each long text data;

[0022] Set a H_u value, and H_u>3. In the vocabulary data set, take each word, every two consecutive words, every three consecutive words, and every H_u consecutive words as an initial feature, and integrate them into a single vocabulary feature set, two vocabulary feature sets, three vocabulary feature sets, and H_u vocabulary feature sets respectively;

[0023] Calculate the frequency of occurrence of each initial feature in the document, set a k value, select the first k initial features with the highest frequency in each feature set as text features, and integrate all text features to form a text feature dataset;

[0024] The text feature dataset is used as input data to train the Word2vec model so that each text feature is mapped to a vector space of fixed length. The term vector of each text feature in the text feature dataset is obtained through the output of the model training.

[0025] All term vectors in the text feature data set are processed by weighted average method to obtain the long text vector of each long text data.

[0026] Furthermore, the method of detecting duplicate data and identifying duplicate data entries based on the comprehensive feature set includes:

[0027] The data entry is a row of features in a comprehensive feature set, representing an independent intellectual property record;

[0028] The comprehensive feature set was standardized, the distance between each pair of data entries was calculated using Euclidean distance, and suspected duplicate data entries were preliminarily screened;

[0029] Suspected duplicate data entries were screened a second time, and the final duplicate data entries were identified and marked.

[0030] Furthermore, the method of obtaining suspected duplicate data entries through preliminary screening includes:

[0031] Step S61: setting a threshold value e, defining the neighborhood of a data point as all data points whose distance is less than or equal to e;

[0032] Among them, a data point represents a data entry in the comprehensive feature set;

[0033] Step S62: Setting the minimum number of samples s_m;

[0034] For each data point, if the number of data points in its neighborhood is greater than or equal to s_m, then mark the data point as a core point;

[0035] Step S63: Select an unvisited core point as the starting point for access, add the core point and all the data points in its neighborhood to a cluster as the current cluster, recursively access the neighborhood of all data points in the current cluster until no new points can be added to the current cluster, and repeat step S63 until all core points have been visited;

[0036] Step S64: for each formed cluster, check the data entries therein, and mark the data entries in the same cluster as suspected duplicate data entries.

[0037] Furthermore, the method of performing secondary screening on the suspected duplicate data entries and identifying and marking the finally determined duplicate data entries includes:

[0038] Step S71: define hypotheses H0 and H1, where H0 indicates that two data entries are duplicated, and H1 indicates that two data entries are not duplicated;

[0039] Set the initial prior probability P(H0) to be equal to P(H1);

[0040] Among them, P(H0) represents the probability that any two data items in the same cluster are duplicates, and P(H1) represents the probability that any two data items in the same cluster are not duplicates;

[0041] Step S72: for any pair of data items in the same cluster, calculate the absolute difference xX of the numerical features between the two data items, and use cosine similarity to calculate the feature difference vX of the text vector features between the two data items, and calculate the likelihood functions of the absolute difference and the feature difference respectively;

[0042] Likelihood function of feature differences:

[0043]

[0044] Where c = [1, 0], H c =[H0,H1],μ c Is vX based on H c The mean vector under c Is vX based on H c The covariance matrix under , d is the dimension of the vector;

[0045] Likelihood function of absolute difference:

[0046] Among them, xμ x Is xX based on H c The mean vector under σx is xX based on H c Variance under ;

[0047] Step S73: Combine the likelihood functions of the text vector features and the time features to form a joint probability distribution P(vX,xX,|H c )=P(vX|H c )·P(xX|H c );

[0048] According to the joint probability distribution, calculate the posterior probability

[0049] Where P(vX,xX) is the marginal probability of feature difference and absolute difference, P(vX,xX)=P(vX,xX|H0)×P(H0)+P(vX,xX|H1)×P(H1);

[0050] Step S74: Set the repetition threshold xc. If P(H c |vX,xX)>xc, the two data entries are judged to be duplicates, otherwise, they are judged not to be duplicates;

[0051] Step S75: record the data items determined to be duplicated as xn_0, record the number of non-duplicate data items as xn_1, and update the prior probabilities P(H0) and P(H1) respectively;

[0052] Among them, the method of updating P(H0) is to use the ratio of xn_0 to the sum of xn_0 and xn_1 as the new prior probability P(H0);

[0053] The method to update P(H1) is to use the ratio of xn_1 to the sum of xn_0 and xn_1 as the new prior probability P(H1);

[0054] Step S76: Repeat the iterations of step S72 to step S75, and calculate the difference between the prior probabilities updated between two adjacent iterations. If the difference is less than a preset difference threshold, it is determined that the prior probability has converged.

[0055] Step S77: Use the converged prior probability calculation to obtain the repeated data entries in the same cluster.

[0056] Furthermore, the method of evaluating the impact of duplicate data entries and handling them includes:

[0057] Step S81: Identify all data entry pairs in the same cluster and mark them as deletable data entry pairs;

[0058] Step S82: Integrate the pairs of deletable data entries in all clusters, set a risk quantity threshold, and if the total number of pairs of deletable data entries is less than or equal to the risk quantity threshold, determine that the current deletion operation is risk-free, and remove one data entry in each pair of deletable data entries from the database;

[0059] If the total number of deletable data item pairs is greater than the risk quantity threshold, the current deletion operation is determined to contain high risk, and an assessment report containing comprehensive data of the deletable data items is generated;

[0060] Step S83: Introduce a user feedback mechanism to send the evaluation report to the administrator, allowing the administrator to confirm or reject the deletion operation with high risk.

[0061] Furthermore, in the process of processing data entries, the method of maintaining consistency of data among multiple nodes and performing real-time monitoring of abnormal states includes:

[0062] Using message queue technology, when a data entry is deleted, a deletion event is generated and published;

[0063] Among them, the deletion event contains the unique identifier and basic information of the deleted data entry;

[0064] The background service of each storage node is notified through the message queue. After receiving the deletion event, the background service executes the deletion operation in the local database.

[0065] If the deletion operation is successful, the background service sends a successful processing signal to the message queue, and the message queue removes the deletion event;

[0066] If the deletion operation fails, the background service will automatically retry until the upper limit of the number of retries is reached. If the deletion is not successful, an exception signal will be sent to the message queue.

[0067] Furthermore, the method of real-time monitoring of abnormal conditions includes:

[0068] Real-time monitoring of data entry deletion operation status indicators, including deletion success rate, deletion failure rate, number of failures, and processing time;

[0069] Record log information during each data entry deletion process, including success and failure logs;

[0070] When the message queue receives an abnormal signal, it integrates status indicators and log information to generate an abnormal report;

[0071] Send alarm notifications and exception reports to administrators.

[0072] The technical effects and advantages of a storage management system based on intellectual property data of the present invention are as follows:

[0073] The present invention constructs a comprehensive feature set including text vectors, numerical features and time features by performing standardized processing, data cleaning, data standardization and conversion on comprehensive data, thereby improving the efficiency and accuracy of subsequent processing steps; accurately identifies duplicate data entries through preliminary screening and secondary screening, and reduces the workload of manual review; effectively avoids the risk of accidentally deleting important data by counting the number of duplicate data, evaluating the impact of duplicate data, and taking corresponding processing measures according to the degree of risk; ensures the consistency of cross-node data operations by using message queue technology, and can ensure data integrity and consistency even in the case of unstable network; provides real-time status monitoring and detailed log records, and can generate reports and notify administrators in time once an abnormal situation is found, which helps to quickly respond to and solve problems. In summary, the present invention not only improves the efficiency and quality of intellectual property data management, but also enhances the robustness and reliability of the system, and provides users with more reliable and convenient services. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A schematic diagram of a storage management system based on intellectual property data of the present invention;

[0075] Figure 2The figure is a schematic diagram of a storage management method based on intellectual property data of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Example 1

[0078] See also Figure 1 As shown, the storage management system based on intellectual property data described in this embodiment includes:

[0079] Data processing module: used to obtain comprehensive data, perform preprocessing, and build a comprehensive feature set;

[0080] Data detection module: used to detect duplicate data and identify duplicate data entries based on a comprehensive feature set;

[0081] Impact Assessment Module: used to assess the impact of duplicate data entries and handle them;

[0082] Data management module: used to maintain data consistency among multiple nodes during data entry processing and to monitor abnormal status in real time;

[0083] Furthermore, the comprehensive data includes basic information, legal information, technical information and application fields (application fields can help narrow the search scope and improve detection efficiency);

[0084] Basic information includes title (title is usually unique and can be used to quickly exclude most irrelevant items), abstract (abstract provides more details and helps to further confirm similarity), keywords (keywords can help identify items with similar content) and categories (categories can narrow the search scope and improve detection efficiency);

[0085] Legal information includes application number (application number is unique and can be used directly to exclude duplicate entries), application date, grant number (grant number is also unique and can be used directly to exclude duplicate entries), grant date, right holder (right holder information can help identify multiple entries of the same right holder), agent (agent information can assist in identifying multiple entries of the same agent) and legal status (legal status can assist in confirming the validity of an entry);

[0086] Technical information includes technical fields (technical fields can help narrow the search scope and improve detection efficiency), invention content (the invention content provides rich text information, which helps to confirm similarity) and claims (the claims provide specific legal protection scope, which helps to confirm similarity);

[0087] Furthermore, the preprocessing method includes data cleaning, data standardization and data conversion;

[0088] The method of constructing the comprehensive feature set includes:

[0089] A text length threshold is set, and text data in the comprehensive data that is greater than the text length threshold is classified as long text data, and text data that is less than or equal to the text length threshold is classified as short text data;

[0090] Convert long text data into long text vectors and concatenate them into text vector features in sequence;

[0091] Extract and transform (including text encoding, category encoding and numerical normalization) numerical data, short text data and category data into numerical features from comprehensive data;

[0092] Combine text vector features and numerical features into a comprehensive feature set;

[0093] The method of converting long text data into a long text vector includes:

[0094] Perform word segmentation, stop word removal, stem extraction and lemmatization on long text data to form a vocabulary data set for each long text data;

[0095] Set a H_u value, and H_u>3 (H_u determines the maximum length of the continuous phrase to be considered). In the vocabulary data set, each word, every two consecutive words, every three consecutive words, and every H_u consecutive words are used as an initial feature, and are respectively integrated into a single vocabulary feature set (each word is used as an initial feature), two vocabulary feature sets (every two consecutive words are used as an initial feature), three vocabulary feature sets (every three consecutive words are used as an initial feature), and H_u vocabulary feature sets (every H_u consecutive words are used as an initial feature);

[0096] Calculate the frequency of occurrence of each initial feature in the document, set a k value (for example, 100, 500, 1000, because the number of lexical features of long text data is too large, in order to reduce the amount of calculation and reduce the system computing burden, it is necessary to extract some of the key features as text features, the number should not be too small or too large, too small will cause distortion of the long text data, too large will cause excessive calculation), select the first k initial features with the highest frequency in each feature set as text features, and integrate all text features into a text feature data set (integrate the text features in a single lexical feature set, two lexical feature sets, three lexical feature sets and H_u lexical feature sets together to generate a text feature data set);,

[0097] It should be noted that this feature set with different numbers of words is of great significance in text processing and feature extraction. A single word is the most basic feature unit and can capture the key words in a document. However, a single word often cannot fully express complex semantics. By considering multiple consecutive words, richer semantic information can be captured. A single word may appear frequently in multiple documents, resulting in low feature discrimination, while a combination of multiple consecutive words is usually more unique and helps to improve feature discrimination. For example, "machine learning" can better distinguish documents on a specific topic than "machine" or "learning" alone. Similarity calculation based on a single word may not be accurate enough, because a single word often cannot fully express the meaning of a sentence. A combination of multiple consecutive words can better represent the local structure and semantics of a sentence, thereby improving the accuracy of similarity calculation. For example, when calculating the similarity between two documents, using bigram or trigram can better capture the subtle differences between documents.

[0098] The text feature dataset is used as input data to train the Word2vec model so that each text feature is mapped to a vector space of fixed length. The term vector of each text feature in the text feature dataset is obtained through the output of the model training.

[0099] The term vectors in each text feature data set are processed by weighted average method to obtain the long text vector of each long text data;

[0100] Furthermore, the method of detecting duplicate data and identifying duplicate data entries based on the comprehensive feature set includes:

[0101] The data entry is a row of features in the comprehensive feature set, representing an independent intellectual property record. The comprehensive feature set is a set of multiple features, each row represents a data entry (i.e., an independent intellectual property record), and each independent intellectual property record contains a series of attributes describing the intellectual property, i.e., comprehensive data (for example, if there is a data set containing 4 patents, and the intellectual property data of each patent has 5 data types, such as [title, right holder, application number, applicant, application date], then in this example, each row in the comprehensive feature set represents a patent entry, row 1: Patent 1, row 2: Patent 2..., and each column represents a data type, then column 1: Title, column 2: Right holder...);

[0102] Each data entry can be represented as an n-dimensional vector, where n is the total dimensionality of all features (for example, if a data entry has a 50-dimensional text vector feature, 3 numerical features, and 1 time feature, then the data entry can be represented as a 54-dimensional vector);

[0103] The comprehensive feature set is standardized, and the distance between each pair of data entries is calculated using the Euclidean distance to preliminarily screen possible duplicate data entries;

[0104] Euclidean distance formula

[0105] Among them, x i and x j is the feature vector of the two data entries, N is the number of features, x ik and x jk They are x i and x j The value of the g-th feature;

[0106] Based on the results of the initial screening, the suspected duplicate data entries are screened again, and the final duplicate data entries are identified and marked;

[0107] The method of obtaining suspected duplicate data entries by preliminary screening includes:

[0108] Step S61: setting a threshold value e, defining the neighborhood of a data point as all data points whose distance is less than or equal to e;

[0109] Among them, a data point represents a data entry in the comprehensive feature set, that is, a row of features, which includes text vector features, numerical features and time features, which together constitute a point in a high-dimensional space;

[0110] Step S62: Set the minimum number of samples s_m, and define a core point as having at least s_m data points in its neighborhood (usually the value of s_m can be set to 5 or 10, depending on the size and density of the comprehensive feature set);

[0111] For each data point, if the number of data points in its neighborhood is greater than or equal to s_m, then mark the data point as a core point;

[0112] Step S63: Select an unvisited core point as the starting point for access, add the core point and all the data points in its neighborhood to a cluster as the current cluster, recursively access the neighborhood of all data points in the current cluster until no new points can be added to the current cluster, and repeat step S63 until all core points have been visited;

[0113] Step S64: for each formed cluster, check the data entries therein, and mark the data entries in the same cluster as possible duplicate data entries (for example, two clusters contain 3 and 4 data entries respectively, and the 3 entries of the first cluster are marked as suspected duplicates. Similarly, the 4 entries of the second cluster are also marked as suspected duplicates. The suspected duplicate data entries in the two clusters are not interoperable);

[0114] The method of performing secondary screening on the suspected duplicate data entries and identifying and marking the finally confirmed duplicate data entries includes:

[0115] Step S71: define hypotheses H0 and H1, where H0 indicates that two data entries are duplicated, and H1 indicates that two data entries are not duplicated;

[0116] Set the initial prior probability P(H0) to be equal to P(H1). Since the total probability is 1, P(H0)+P(H1)=1, P(H0)=P(H1)=0.5;

[0117] Among them, P(H0) represents the probability that any two data items in the same cluster are duplicates, and P(H1) represents the probability that any two data items in the same cluster are not duplicates;

[0118] It should be noted that since it is unknown how many duplicate data entries there are in a cluster, Xu Ya first sets an initial probability;

[0119] Step S72: for any pair of data items in the same cluster, calculate the absolute difference xX of the numerical features between the two data items, and use cosine similarity to calculate the feature difference vX of the text vector features between the two data items, and calculate the likelihood function of the absolute difference and the feature difference respectively (what needs to be paid attention to is the difference between the two data items, so the feature difference between the two data items is input into the likelihood function as a new feature);

[0120] Likelihood function of feature differences:

[0121]

[0122] Among them, c = [1,0], used to represent H0 and H1, that is, when c = 0, H c represents H0, when c=1, H x Indicates H1, H c =[H0,H1],μ x Is vX based on H x The mean vector under c Is vX based on H c The covariance matrix under , d is the vector dimension of vX;

[0123] Likelihood function of absolute difference:

[0124] Among them, xμ c Is xX based on H x The mean vector under σx is xX based on H x Variance under ;

[0125] It should be noted that these likelihood functions describe the probability of observing feature differences and absolute differences under given assumptions;

[0126] Step S73: Combine the likelihood functions of the text vector features and the time features to form a joint probability distribution P(vX,xX,|H x )=P(vX|H x )·P(xX|H x );

[0127] According to the joint probability distribution, calculate the posterior probability

[0128] Where P(vX,xX) is the marginal probability of feature difference and absolute difference, P(vX,xX)=P(vX,xX|H0)×P(H0)+P(vX,xX|H1)×P(H1);

[0129] Step S74: Set the repetition threshold xc. If P(H c |vX,xX)>xc, the two data entries are judged to be duplicates, otherwise, they are judged not to be duplicates;

[0130] Step S75: record the data items determined to be duplicated as xn_0, record the number of non-duplicate data items as xn_1, and update the prior probabilities P(H0) and P(H1) respectively;

[0131] Among them, the method of updating P(H0) is to use the ratio of xn_0 to the sum of xn_0 and xn_1 as the new prior probability P(H0);

[0132] The method to update P(H1) is to use the ratio of xn_1 to the sum of xn_0 and xn_1 as the new prior probability P(H1);

[0133] Step S76: Repeat the iterations of step S72 to step S75, and calculate the difference between the prior probabilities updated between two adjacent iterations. If the difference is less than a preset difference threshold, it is determined that the prior probability has converged.

[0134] Step S77: using the converged prior probability calculation to obtain the duplicate data entries in the same cluster;

[0135] Furthermore, the method of evaluating the impact of duplicate data entries and handling them includes:

[0136] Step S81: Identify all data entry pairs in the same cluster and mark them as deletable data entry pairs;

[0137] Step S82: Integrate the pairs of deletable data entries in all clusters, set a risk quantity threshold, and if the total number of pairs of deletable data entries is less than or equal to the risk quantity threshold, determine that the current deletion operation is risk-free, and remove one data entry in each pair of deletable data entries from the database;

[0138] Before deleting a data entry, the data entry to be deleted is automatically backed up, and the timestamp and version number of the backup are recorded so as to enable rollback when necessary;

[0139] If the total number of deletable data item pairs is greater than the risk quantity threshold, the current deletion operation is determined to contain high risk, and an assessment report containing comprehensive data of the deletable data items is generated;

[0140] Step S83: Introduce a user feedback mechanism to send the evaluation report to the administrator, allowing the administrator to confirm or reject the deletion operation with high risk;

[0141] Provide a visual interface to display the pending report for the administrator to review and decide whether to confirm the deletion operation (the administrator can manually adjust the deletion scope to ensure the accuracy of the operation);

[0142] Record the administrator's confirmation operations to ensure traceability;

[0143] Furthermore, in the process of processing data entries, the method of maintaining consistency of data among multiple nodes and performing real-time monitoring of abnormal states includes:

[0144] Before executing the deletion operation, generate backup files and backup logs;

[0145] The backup file contains the data entries to be deleted and their timestamps so that rollback can be performed when necessary;

[0146] The backup log includes the backup file location and version number to ensure data recoverability;

[0147] Using message queue technology, when a data entry is deleted, a deletion event is generated and published;

[0148] Among them, the deletion event contains the unique identifier and basic information of the deleted data entry;

[0149] The background service of each storage node is notified through the message queue. After receiving the deletion event, the background service executes the deletion operation in the local database.

[0150] If the deletion operation is successful, the background service sends a successful processing signal to the message queue, and the message queue removes the deletion event;

[0151] If the deletion operation fails, the background service will automatically retry until the retry limit is reached. If the deletion is not successful, an abnormal signal will be sent to the message queue.

[0152] The method of performing real-time monitoring and exception handling includes:

[0153] Real-time monitoring of data entry deletion operation status indicators, including deletion success rate, deletion failure rate, number of failures, and processing time;

[0154] Record log information during each data entry deletion process, including success and failure logs;

[0155] When the message queue receives an abnormal signal, it integrates status indicators and log information to generate an abnormal report, and sends an alarm notification and abnormal report to the administrator.

[0156] In this embodiment, by standardizing, cleaning, standardizing and converting comprehensive data, a comprehensive feature set including text vectors, numerical features and time features is constructed, thereby improving the efficiency and accuracy of subsequent processing steps; combining preliminary screening and secondary screening, duplicate data entries are accurately identified, reducing the workload of manual review; by counting the number of duplicate data, the impact of duplicate data is evaluated, and corresponding processing measures are taken according to the degree of risk, effectively avoiding the risk of accidentally deleting important data; using message queue technology to ensure the consistency of cross-node data operations, the integrity and consistency of data can be guaranteed even in the case of unstable network; providing real-time status monitoring and detailed log records, once an abnormal situation is found, a report can be generated in time and the administrator can be notified, which is conducive to rapid response and problem solving. In summary, the present invention not only improves the efficiency and quality of intellectual property data management, but also enhances the robustness and reliability of the system, and provides users with more reliable and convenient services.

[0157] Example 2

[0158] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, which provides a storage management method based on intellectual property data, including:

[0159] S1, obtain comprehensive data, perform preprocessing, and construct a comprehensive feature set;

[0160] S2. Detect duplicate data and identify duplicate data entries based on the comprehensive feature set;

[0161] S3, evaluate the impact of duplicate data entries and deal with them;

[0162] S4. During the data entry processing, the data between multiple nodes is kept consistent and abnormal status is monitored in real time.

[0163] Example 3

[0164] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the storage management method based on intellectual property data provided above is implemented.

[0165] Since the electronic device introduced in this embodiment is an electronic device used to implement a storage management method based on intellectual property data in the embodiment of the present application, based on the storage management method based on intellectual property data introduced in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements an electronic device used in a storage management method based on intellectual property data in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0166] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0167] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A storage management system based on intellectual property data, characterized in that: include: Data processing module: used to obtain comprehensive data, perform preprocessing, and build a comprehensive feature set; Data detection module: used to detect duplicate data and identify duplicate data entries based on a comprehensive feature set; Impact Assessment Module: used to assess the impact of duplicate data entries and handle them; Data management module: used to maintain data consistency among multiple nodes during data entry processing and to monitor abnormal conditions in real time.

2. A storage management system based on intellectual property data according to claim 1, characterized in that: The comprehensive data includes basic information, legal information, technical information and application areas; Basic information includes title, abstract, keywords and categories; Legal information includes application number, application date, grant number, grant date, right holder, agent, and legal status; Technical information includes technical field, invention content and claims.

3. A storage management system based on intellectual property data according to claim 1, characterized in that: The preprocessing method includes data cleaning, data standardization and data conversion; The method of constructing the comprehensive feature set is: Dividing the text data in the comprehensive data into long text data and short text data according to a preset text length threshold; Convert all long text data into long text vectors and concatenate them into text vector features in sequence; Extract and convert numerical data, short text data, and categorical data into numerical features from comprehensive data; The text vector features and numerical features are concatenated into a comprehensive feature set.

4. A storage management system based on intellectual property data according to claim 3, characterized in that: The method of converting all long text data into long text vectors includes: Perform word segmentation, stop word removal, stem extraction and lemmatization on long text data to form a vocabulary data set for each long text data; Set a H_u value, and H_u>3. In the vocabulary data set, take each word, every two consecutive words, every three consecutive words, and every H_u consecutive words as an initial feature, and integrate them into a single vocabulary feature set, two vocabulary feature sets, three vocabulary feature sets, and H_u vocabulary feature sets respectively; Calculate the frequency of occurrence of each initial feature in the document, set a k value, select the first k initial features with the highest frequency in each feature set as text features, and integrate all text features to form a text feature dataset; The text feature dataset is used as input data to train the Word2vec model so that each text feature is mapped to a vector space of fixed length. The term vector of each text feature in the text feature dataset is obtained through the output of the model training. All term vectors in the text feature data set are processed by weighted average method to obtain the long text vector of each long text data.

5. A storage management system based on intellectual property data according to claim 1, characterized in that: The method of detecting duplicate data and identifying duplicate data entries based on the comprehensive feature set includes: The data entry is a row of features in a comprehensive feature set, representing an independent intellectual property record; The comprehensive feature set was standardized, the distance between each pair of data entries was calculated using Euclidean distance, and suspected duplicate data entries were preliminarily screened; Suspected duplicate data entries were screened a second time, and the final duplicate data entries were identified and marked.

6. A storage management system based on intellectual property data according to claim 5, characterized in that: The method of obtaining suspected duplicate data entries by preliminary screening includes: Step S61: setting a threshold value e, defining the neighborhood of a data point as all data points whose distance is less than or equal to e; Among them, a data point represents a data entry in the comprehensive feature set; Step S62: Setting the minimum number of samples s_m; For each data point, if the number of data points in its neighborhood is greater than or equal to s_m, then mark the data point as a core point: Step S63: Select an unvisited core point as the starting point for access, add the core point and all data points in its neighborhood to a cluster as the current cluster, recursively access the neighborhood of all data points in the current cluster until no new points can be added to the current cluster, and repeat step S63 until all core points have been visited: Step S64: for each formed cluster, check the data entries therein, and mark the data entries in the same cluster as suspected duplicate data entries.

7. A storage management system based on intellectual property data according to claim 5, characterized in that: The method of performing secondary screening on the suspected duplicate data entries and identifying and marking the finally confirmed duplicate data entries includes: Step S71: define hypotheses H0 and H1, where H0 indicates that two data entries are duplicated, and H1 indicates that two data entries are not duplicated; Set the initial prior probability P(H0) to be equal to P(H1); Among them, P(H0) represents the probability that any two data items in the same cluster are duplicates, and P(H1) represents the probability that any two data items in the same cluster are not duplicates; Step S72: for any pair of data items in the same cluster, calculate the absolute difference xX of the numerical features between the two data items, and use cosine similarity to calculate the feature difference vX of the text vector features between the two data items, and calculate the likelihood functions of the absolute difference and the feature difference respectively; Likelihood function of feature differences: Where c = [1, 0], H c =[H0,H1],μ c Is vX based on H c The mean vector under c Is vX based on H c The covariance matrix under , d is the dimension of the vector; Likelihood function of absolute difference: Among them, xμ c Is xX based on H c The mean vector under H, σx is xX based on H c Variance under ; Step S73: Combine the likelihood functions of the text vector features and the time features to form a joint probability distribution P(vX, xX, |H c )=P(vX|H c )·P(xX|H c ); According to the joint probability distribution, calculate the posterior probability Where P(vX,xX) is the marginal probability of feature difference and absolute difference, P(vX,xX)=P(vX,xX|H0)×P(H0)+P(vX,xX|H1)×P(H1); Step S74: Set the repetition threshold xc. If P(H c |vX,xX)>xc, the two data entries are judged to be duplicates, otherwise, they are judged not to be duplicates; Step S75: record the data items determined to be duplicated as xn_0, record the number of non-duplicate data items as xn_1, and update the prior probabilities P(H0) and P(H1) respectively; Among them, the method of updating P(H0) is to use the ratio of xn_0 to the sum of xn_0 and xn_1 as the new prior probability P(H0); The method to update P(H1) is to use the ratio of xn_1 to the sum of xn_0 and xn_1 as the new prior probability P(H1); Step S76: Repeat the iterations of step S72 to step S75, and calculate the difference between the prior probabilities updated between two adjacent iterations. If the difference is less than a preset difference threshold, it is determined that the prior probability has converged. Step S77: Use the converged prior probability calculation to obtain the repeated data entries in the same cluster.

8. A storage management system based on intellectual property data according to claim 1, characterized in that: The method of evaluating the impact of duplicate data entries and handling them includes: Step S81: Identify all data entry pairs in the same cluster and mark them as deletable data entry pairs; Step S82: Integrate the pairs of deletable data entries in all clusters, set a risk quantity threshold, and if the total number of pairs of deletable data entries is less than or equal to the risk quantity threshold, determine that the current deletion operation is risk-free, and remove one data entry in each pair of deletable data entries from the database; If the total number of deletable data item pairs is greater than the risk quantity threshold, the current deletion operation is determined to contain high risk, and an assessment report containing comprehensive data of the deletable data items is generated; Step S83: Introduce a user feedback mechanism to send the evaluation report to the administrator, allowing the administrator to confirm or reject the deletion operation with high risk.

9. A storage management system based on intellectual property data according to claim 1, characterized in that: The method of maintaining data consistency among multiple nodes and performing real-time monitoring of abnormal status during data entry processing includes: Using message queue technology, when a data entry is deleted, a deletion event is generated and published; Among them, the deletion event contains the unique identifier and basic information of the deleted data entry; The background service of each storage node is notified through the message queue. After receiving the deletion event, the background service executes the deletion operation in the local database. If the deletion operation is successful, the background service sends a successful processing signal to the message queue, and the message queue removes the deletion event; If the deletion operation fails, the background service will automatically retry until the upper limit of the number of retries is reached. If the deletion is not successful, an exception signal will be sent to the message queue.

10. A storage management system based on intellectual property data according to claim 9, characterized in that: The method of real-time monitoring of abnormal status includes: Real-time monitoring of data entry deletion operation status indicators, including deletion success rate, deletion failure rate, number of failures, and processing time; Record log information during each data entry deletion process, including success and failure logs; When the message queue receives an abnormal signal, it integrates status indicators and log information to generate an abnormal report; Send alarm notifications and exception reports to administrators.

Images