Data discovery method, system and device based on weight distribution and entity labeling and medium

Abstract

The invention provides a data discovery method, system and device based on weight distribution and entity annotation and a medium, and belongs to the technical field of data search processing. Dynamically adjusting the weight of each data item by adopting a machine learning algorithm according to the actual characteristics of the data; a self-adaptive weight distribution model is configured, and actual features in the data are labeled in a multi-layer entity labeling mode; constructing a reverse index retrieval model, extracting features of each data item according to the self-adaptive weight distribution model, distributing corresponding weights, performing score calculation, performing sorting according to scores, and feeding back to users; a closed loop is formed by collecting and analyzing user search data, dynamically adjusting adaptive weight distribution, a search and sorting algorithm and an index performance optimization strategy, iteratively executing evaluation and optimization and combining a user feedback mechanism. The method can accurately reflect the importance of different features to search results. And diversified search requirements of the user can be better met.

Description

Technical Field

[0001] The present invention belongs to the technical field of data search and processing, and in particular relates to a data discovery method, system, device and medium based on weight allocation and entity labeling. Background Art

[0002] Directory and data searches in existing blockchain systems require the use of keywords or natural language processing technology. However, due to the lack of weight distinction in the tag system, the search results are not accurately positioned and sorted reasonably, making it difficult to meet users' needs for efficient and accurate information acquisition.

[0003] Moreover, in existing data processing technologies, data is usually only simply cleaned and organized, lacking in-depth analysis of data features and weight allocation, resulting in low accuracy and efficiency of data processing. At the same time, existing data annotation methods often only focus on the annotation of basic entities, ignoring the complex relationships in the data and failing to fully reflect the inherent structure of the data. In addition, existing data processing methods lack iterative optimization and loop feedback mechanisms, and cannot be dynamically adjusted according to user behavior and data changes, limiting the improvement of data processing effects. Summary of the invention

[0004] The present invention provides a data discovery method based on weight allocation and entity labeling. The method constructs a comprehensive model of data attributes, entity relationships and their importance, combines historical data with user feedback for iterative optimization, and realizes accurate search and sorting of on-chain directories and data.

[0005] Methods include: Perform preprocessing operations on the data; According to the actual characteristics of the data, the machine learning algorithm is used to dynamically adjust the weight of each data item; Configure an adaptive weight distribution model and use a multi-layer entity annotation method to annotate the actual features in the data; Construct an inverted index retrieval model, extract features from each data item according to the adaptive weight allocation model, assign corresponding weights, and perform score calculations. According to the scores, the data items are sorted and fed back to the user. By collecting and analyzing user search data, dynamically adjusting adaptive weight distribution, search and sorting algorithms, and index performance optimization strategies, and iteratively performing evaluation and optimization, a closed loop is formed in combination with a user feedback mechanism.

[0006] It should be further explained that the step uses a machine learning algorithm to dynamically adjust the weight of each data item, uses the Pearson correlation coefficient to calculate the characteristics of the impact of data attributes and entity relationships on search results, and calculates the mean and standard deviation of data attributes and entity relationships respectively; Divide the covariance by the product of the standard deviation of the data attribute and the entity relationship to obtain the Pearson correlation coefficient; The Pearson correlation coefficient calculation formula is:

[0007] is the covariance of X and Y; express The standard deviation of express The standard deviation of X represents the data attribute, Represents entity relationships; The covariance is calculated as:

[0008] in, is the i-th observation value of the data attribute, is the i-th observation value of the entity relationship, is the mean of the data attributes, is the mean of entity relationships, and n is the number of observations.

[0009] It should be further explained that, in the method, the threshold of Pearson correlation coefficient is set; The calculated Pearson correlation coefficient is compared with the Pearson correlation coefficient threshold; When the calculated Pearson correlation coefficient is greater than or equal to the Pearson correlation coefficient threshold, defining the data attribute for calculating the Pearson correlation coefficient as a feature; The effectiveness of the features was verified by cross-validation method.

[0010] It is further necessary to explain that, in the method, a machine learning algorithm is used to train the features to learn the weight of each feature on the search results; The overall score is calculated based on the following formula:

[0011] in, is the standardized feature value in the original feature set, is the weight corresponding to the eigenvalue; is the normalized value of the additional feature, is the weight corresponding to the value; i is an index variable used to traverse all features in the original feature set; n is the total number of features in the original feature set; The comprehensive score textscore is calculated based on the standardized values ​​of all features and their corresponding weights; the comprehensive score textscore is used to measure the degree of match between a search result and the user's query.

[0012] It should be further explained that the step of optimizing the data according to the results of the adaptive weight allocation and multi-layer entity labeling also includes: Through the Transformer algorithm, after entity recognition, it is labeled and a labeled data set is constructed; Using the BERT model, we perform a hierarchical labeling process and use the output as features to feed a fully connected neural network. The location information of the entity in the text is input into the relationship classifier as an additional feature to assist in determining the hierarchical relationship; Label the hierarchical relationship tasks, use the cross entropy loss function to train the relationship classifier, and select the dam optimizer to update the model weights in real time; All entities are identified, the hierarchical relationship scores between each pair of entities are calculated, and a hierarchical relationship graph between entities is constructed based on the scoring results and the preset scoring result threshold.

[0013] It is further necessary to explain that, in the method, the results of entity and hierarchical relationship annotation are integrated into the data graph to form a structured representation with multi-layer entity annotations; Based on entities and hierarchical relationships, the nodes and edges of the data graph are constructed; and the identified entities and their identifiers, as well as the hierarchical relationship annotation results, are configured into the data graph; Visualize the data map.

[0014] It should be further explained that the steps of constructing the inverted index retrieval model also include: Receive the query statement input by the user and parse the query condition information; Match records in the inverted index retrieval model based on query condition information; Collect all matching records and build an initial result set; Based on the adaptive weight assignment model, features are extracted for each data item and weights are assigned. The hierarchical relationship features between entities are extracted using the results of multi-layer entity annotation. Linearly combine the extracted features and weights to calculate the score of each data item; The scoring formula is: Score = Σ( * Fi), where is the weight of the feature, Fi is the value of the feature; The calculated scores are normalized so that they are within a preset score range. A score result set is constructed based on the calculated scores, and the scores are sorted. The sorted score result set is returned to the user.

[0015] The present application also provides a data discovery system based on weight allocation and entity annotation, the system comprising: A data preprocessing module is used to perform preprocessing operations on data; The weight adjustment module is used to dynamically adjust the weight of each data item using a machine learning algorithm based on the actual characteristics of the data; The feature annotation module is used to configure the adaptive weight distribution model and annotate the actual features in the data using a multi-layer entity annotation method; The index construction scoring module is used to build an inverted index retrieval model. According to the adaptive weight allocation model, it extracts features from each data item, assigns corresponding weights, and performs score calculations. According to the scores, it sorts and provides feedback to users. The adjustment and optimization feedback module is used to dynamically adjust the adaptive weight distribution, search and sorting algorithms, and index performance optimization strategies by collecting and analyzing user search data, and iteratively perform evaluation and optimization, forming a closed loop in combination with the user feedback mechanism.

[0016] According to another embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a data discovery method based on weight allocation and entity labeling when executing the program.

[0017] According to another embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data discovery method based on weight allocation and entity labeling are implemented.

[0018] It can be seen from the above technical solutions that the present invention has the following advantages: This application mainly uses machine learning algorithms to automatically calculate and assign weights to data attributes and entity relationships, which can accurately reflect the importance of different features to search results. This adaptive weight allocation method is more flexible and accurate than the traditional fixed weight allocation, and can better meet the diverse search needs of users. Deep learning technology is used to perform multi-layer entity annotation on the on-chain data, identify and annotate key entities and their hierarchical relationships. This annotation method helps the system to deeply understand the data content, so as to more accurately match the user's query intent during the search process and improve the accuracy of search results.

[0019] In the data discovery method based on weight allocation and entity annotation provided by this application, a search algorithm is configured to support rapid retrieval based on multiple query methods such as keywords, attributes, entities, etc. Through optimization means such as indexing technology and caching mechanism, the query efficiency can be significantly improved. In addition, the search results can be intelligently sorted according to the adaptive weight allocation model and multi-layer entity annotation results. The data that users are interested in is displayed first, reducing the time for users to browse irrelevant information and improving search efficiency. Due to the improvement of the accuracy of search results and search efficiency, users can get a better experience when using the search system built by this patented technology. Through the user feedback mechanism and iterative optimization strategy, the system can continuously improve according to user needs and feedback. This user participation method helps to enhance the interaction and connection between users and the system, and further improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 Flowchart of the data discovery method based on weight assignment and entity labeling; Figure 2 A flow chart of an embodiment of a data discovery method based on weight allocation and entity labeling; Figure 3 It is a flowchart based on iterative optimization and feedback mechanism; Figure 4 Schematic diagram of a data discovery system based on weight assignment and entity labeling; Figure 5 Schematic diagram of an electronic device. DETAILED DESCRIPTION

[0022] The data discovery method based on weight allocation and entity labeling provided in this application is mainly to solve the problems of low data processing accuracy and efficiency, incomplete data labeling, and lack of iterative optimization and loop feedback mechanism in the prior art.

[0023] The data discovery method based on weight allocation and entity labeling provided in this application involves performing preprocessing steps such as cleaning and denoising on the original data set, which can eliminate outliers, missing values ​​and irrelevant information in the data, thereby providing a data basis for subsequent data analysis or processing processes.

[0024] The present application provides an adaptive weight assignment model that can dynamically assign weights to each data item based on the inherent characteristics of the data. In the process of weight assignment, multiple dimensions such as the importance and relevance of the data are used to ensure that the data items receive more attention and processing. On a data set that has completed adaptive weight assignment, multi-layer entity annotation technology is used to identify and annotate entities in the data. The annotation process presents a hierarchical feature, gradually deepening from basic entities to complex relationships, thereby obtaining the intrinsic structure and association of the data. Based on the results of adaptive weight assignment and multi-layer entity annotation, the original data is further optimized.

[0025] In order to continuously meet users' needs for data search, iterative optimization and loop feedback strategies are used to regularly collect and analyze user search behavior data, such as click-through rate, dwell time, etc., as key indicators for evaluating search and sorting effects. Based on these evaluation results, the system automatically adjusts the weight distribution model, optimizes the search algorithm and annotation strategy, thereby forming a closed-loop feedback mechanism. This can continuously improve the accuracy and efficiency of on-chain data search and sorting.

[0026] The specific steps of the data discovery method based on weight allocation and entity annotation are described in detail below. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0027] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 FIG. 1 is a flowchart of a data discovery method based on weight allocation and entity labeling in a specific embodiment, and the method includes: S101: Preprocessing the data.

[0030] The data involved in this embodiment include transaction records, smart contract data, metadata, etc. on the blockchain network. It can also analyze and sample business data and personnel change data generated by public social institutions such as enterprises, institutions, and schools, and uniformly incorporate the analysis data into the source database to facilitate analysis of the analyzed data resources.

[0031] This embodiment can use the blockchain probe tool to capture the on-chain data and store it in the database.

[0032] To ensure the integrity of the data and meet the attribute recommendation requirements, the captured data can be cleaned to remove duplicate, erroneous or invalid data to ensure the accuracy and completeness of the data.

[0033] This embodiment also involves data standardization, and the cleaned data can be encoded and stored according to a unified format and standard to facilitate subsequent processing and analysis.

[0034] For example, in the financial field, customer transaction records, credit records and other data are collected and analyzed, and blockchain technology is used to collect and verify transaction data for risk assessment, credit scoring and fraud detection, thereby improving the transparency and security of financial transactions.

[0035] For this embodiment, user behavior data, such as clicks, purchases, ratings, etc., are collected from various data sources. Missing values, duplicate values, outliers, etc. are processed to ensure data quality. Data from multiple data sources are merged into a unified data set. Data is converted into a form suitable for subsequent analysis through smooth aggregation, data generalization, normalization, etc. After the above processing, the size of the data set can be reduced while maintaining the integrity of the data to improve processing efficiency.

[0036] S102: According to the actual characteristics of the data, a machine learning algorithm is used to dynamically adjust the weight of each data item.

[0037] In some embodiments, meaningful features are extracted from the preprocessed data using AHP or entropy weight method, and the weight of each data item is dynamically adjusted according to the importance of the feature.

[0038] Specifically, you can identify and extract key features relevant to the recommendation system. Choose an appropriate machine learning algorithm to calculate the weights. Train the model to learn the relative importance of features. Dynamically adjust the weights of each data item based on the model output.

[0039] In this way, it can automatically adapt to changes in data characteristics and improve the accuracy and relevance of recommendation results.

[0040] S103: configuring an adaptive weight distribution model, and using a multi-layer entity labeling method to label actual features in the data.

[0041] In some embodiments, an adaptive weight allocation model that can be dynamically adjusted according to data characteristics can be configured. The entities in the data are hierarchically labeled using techniques such as multi-level learning to rank to capture the actual characteristics of the data.

[0042] It can be seen that in this embodiment, a suitable adaptive weight distribution model can be selected and configured. The data is labeled using multi-layer entity labeling technology to extract richer feature information. The labeled data is input into the weight distribution model for processing. In this way, the actual features of the data are captured by multi-layer entity labeling technology. The weight distribution is made more reasonable and the performance of the recommendation system is improved.

[0043] S104: Construct an inverted index retrieval model, extract features from each data item according to the adaptive weight allocation model, assign corresponding weights, and perform score calculations. According to the scores, the data items are sorted and fed back to the user.

[0044] In some embodiments, an inverted index retrieval model is constructed and preprocessed data is loaded. For each query request, an adaptive weight allocation model is used to extract relevant features and assign weights. The score of each data item is calculated based on the weight and feature. The data items with higher scores are sorted and fed back to the user. In this way, the retrieval efficiency is improved, and the user can find relevant information faster. Through weight allocation and score calculation, the recommendation results are more in line with the user's interests and needs.

[0045] S105: By collecting and analyzing user search data, dynamically adjusting the adaptive weight allocation, search and sorting algorithms, and index performance optimization strategies, and iteratively performing evaluation and optimization, combined with the user feedback mechanism to form a closed loop.

[0046] In some embodiments, user search data collection: continuously collect user search behavior data. Dynamically adjust adaptive weight allocation, search and sorting algorithms, and index performance optimization strategies based on user search data. Regularly evaluate system performance and perform iterative optimization based on evaluation results. Establish a user feedback mechanism to collect and process user opinions and suggestions in a timely manner.

[0047] As a preferred implementation of step S105, a user search data collection mechanism can be defined, and relevant data can be collected in real time. Use machine learning algorithms to analyze user search data to discover potential trends and patterns. Dynamically adjust various components of the system based on the analysis results, such as weight distribution models, search and sorting algorithms, etc. Regularly evaluate the performance of the system, including indicators such as recommendation accuracy, retrieval efficiency, and user satisfaction. Formulate optimization strategies based on the evaluation results, and iterate the optimization process. Establish a user feedback mechanism to promptly handle user feedback and suggestions to continuously improve system performance.

[0048] It can be seen that the method can continuously learn and adapt to changes in user behavior, thereby improving system performance and user experience.

[0049] In an embodiment of the present invention, based on step S102, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0050] In this embodiment, features that have a greater impact on search results are selected from data attributes and entity relationships, such as timestamp, transaction type, amount, importance of both parties to the transaction, etc.; this can play a key role in electronic data forensics and intellectual property protection in the fields of financial transactions and blockchain, formulating appropriate transaction strategies and risk management measures, understanding the market's trading activity, capital flow and investors' risk preferences, and evaluating the quality and potential value of transactions.

[0051] It should be noted that the data attributes in this embodiment are characteristics or features associated with data items, which can describe the nature, state or behavior of data items. In the fields of financial transactions, blockchain, etc., data attributes may include timestamps (recording the time when the transaction occurred), transaction types (such as buying, selling, transfers, etc.), amounts (the amount of funds involved in the transaction), the importance of both parties to the transaction (such as the credit ratings and market positions of both parties to the transaction), etc. Attributes are key information that describes transaction behavior.

[0052] The entity relationship in this embodiment refers to the connection or association between different data items or entities. In the field of financial transactions, entity relationships may be manifested as the connection between the two parties to a transaction (such as the relationship between a buyer and a seller), the connection between a transaction and a specific event or condition (such as the relationship between a transaction and a specific market event), etc. Entity relationships reveal the inherent connection between data items, which helps to deeply understand the structure and behavior patterns of data, so as to formulate more appropriate trading strategies and risk management measures.

[0053] Optionally, this embodiment uses the Pearson correlation coefficient to calculate the features between data attributes and entity relationships that have a greater impact on search results, and calculates the mean of the data attributes and entity relationships respectively. and standard deviation .

[0054] Specifically, the Pearson correlation coefficient is calculated based on the following method: The calculation method is to divide the covariance by the product of the standard deviation of the data attribute and the entity relationship to obtain the Pearson correlation coefficient.

[0055] The formula is:

[0056] Here, covariance is a statistic that measures the extent to which two variables vary together.

[0057] The covariance calculation formula is:

[0058] in, and are the i-th observation values ​​of variables X and Y, respectively. is the covariance of X and Y; express The standard deviation of express The standard deviation of Represents entity relationships; and are the means of variables X and Y respectively, and n is the number of observations.

[0059] In this embodiment, a threshold of the Pearson correlation coefficient can be set according to actual needs. This threshold can be set according to factors such as the characteristics of the data and the accuracy requirements of the search results. The calculated Pearson correlation coefficient is compared with the set threshold. If the Pearson correlation coefficient of a data attribute is greater than or equal to the threshold, it is considered that the data attribute has a greater impact on the search results and is selected as a feature.

[0060] This embodiment also verifies the validity of the selected features by cross-validation or other methods. Cross-validation is a method of dividing a data set into a training set and a test set, then training a model on the training set and evaluating the model performance on the test set. Through cross-validation, the impact of the selected features on the model performance can be evaluated.

[0061] Considering the characteristics that the larger the data scale, the more stable and reliable the calculation result of the Pearson correlation coefficient, it is necessary to collect more data to calculate the Pearson correlation coefficient when possible. The Pearson correlation coefficient has excellent processing capabilities when processing large amounts of complex data on the data platform. The Pearson correlation coefficient is used to calculate the data attributes and entity relationships to select features that have a greater impact on the search results. In practical applications, it is necessary to flexibly adjust and optimize according to the characteristics of the data and actual needs.

[0062] In the weight calculation process of this embodiment, a machine learning algorithm can be used to train the selected features to learn the weight of each feature's importance to the search results. The weight calculation process considers multiple dimensions such as the timeliness, rarity, and user attention of the data.

[0063] The linear regression model is used to calculate the weight of the importance of the search results. After selecting the weight and standardizing the data, the linear model is obtained:

[0064] in, is the weight of the feature, is the standardized feature value, and score is the final score of the search result.

[0065] This embodiment takes into account multiple factors such as the timeliness, rarity, and user attention of search results, sets additional features for these dimensions, and learns their weights during the training process.

[0066] In summary, the weight calculation mathematical model can be expressed as:

[0067] in, is the standardized feature value in the original feature set, is the corresponding weight; is the standardized value of additional features such as timeliness, rarity, and user attention. are the corresponding weights. By training the model, these weights can be learned and used to sort the search results.

[0068] In this embodiment, the model can be tuned by cross-validation, grid search and other methods, the data set can be divided into multiple subsets, and the performance of the model on different data subsets can be evaluated through multiple training and testing to determine the optimal model parameters. And within the specified parameter range, the optimal parameter combination can be found through exhaustive search to improve the accuracy and stability of weight distribution.

[0069] For the cross-validation method, the data set can be divided into multiple subsets, model training and weight calculation can be performed in some subsets, and the performance of the model can be tested on the remaining subsets. Define the search space of weights within a reasonable range, use an exhaustive search method, try all possible weight combinations in this search space, and use cross-validation for each weight combination to evaluate its performance.

[0070] The evaluation method mainly focuses on selecting appropriate performance indicators, such as accuracy or recall, to evaluate the performance of the model, recording the average performance indicator of each weight combination in cross-validation, and selecting the best performing weight combination as the optimal weight based on the results of the performance evaluation.

[0071] In this embodiment, after dynamically adjusting the weight of each data item using a machine learning algorithm, an adaptive weight distribution model is configured, and a multi-layer entity annotation method is used to annotate the actual features in the data. As for the annotation method, the Transformer algorithm can be used to perform entity recognition on the chain data to identify key entities such as organizations, individuals, projects, etc.

[0072] This embodiment further analyzes the hierarchical relationship between entities based on entity recognition. Hierarchical relationship annotation is a very useful task in natural language processing and specific fields such as blockchain data analysis. It is based on entity recognition and can provide more in-depth information, such as the department structure within the organization, the relationship between projects and participants, etc. by parsing and annotating the hierarchical or organizational structure relationship between entities.

[0073] like Figure 2 As shown, the specific process of using the Transformer algorithm to perform entity recognition on the on-chain data is described in detail below.

[0074] Step S1031: By using the Transformer algorithm, the key entities obtained from the on-chain data after entity recognition are annotated to construct an annotated data set, in which each sample contains entities and the hierarchical relationships between them. Through relevant regulations or document requirements, fixed online available rules are generated for hierarchical annotation. If there are no fixed or clear institutional rules, manual annotation can be performed.

[0075] Step S1032: Use the BERT model to complete the hierarchical labeling task, and input the output results as features into the relationship classifier. The relationship classifier can predict the hierarchical relationship between entities based on the features provided by the BERT model.

[0076] Step S1033: Generate an embedding representation for each entity using the BERT model by taking the average or maximum value of all token embeddings in the entity. The location information of the entity in the text is input into the relationship classifier as an additional feature to assist in determining the hierarchical relationship.

[0077] Step S1034: For the hierarchical relationship labeling task, the relationship classifier is trained using the cross entropy loss function. In this embodiment, the dam optimizer can be selected to update the model weights in real time. Indicators such as precision, recall rate, and F1 score are used to evaluate the performance of the model.

[0078] Step S1035: For a given text, first identify all entities, then calculate the hierarchical relationship score between each pair of entities, and construct a hierarchical relationship graph between entities based on the score results and a preset threshold.

[0079] In the process of executing step S1031, noise relations may be removed, similar entities may be merged, etc. The hierarchical relationship graph is visualized so as to intuitively understand the hierarchical structure between entities. In addition, explanatory information may be provided, such as the confidence of each relationship prediction, etc.

[0080] As for the method of integrating annotation results, the results of entity recognition and hierarchical relationship annotation can be integrated into the data graph to form a structured representation of multi-layer entity annotation.

[0081] In this embodiment, Neo4j, a data graph construction tool, is used to construct nodes and edges of a data graph based on entities and their hierarchical relationships. The identified entities and their identifiers, as well as the hierarchical relationship annotation results, are imported into the data graph. For example, redundant information is removed, key features are enhanced, etc., to improve the usability and value of the data graph.

[0082] In this embodiment, a data map visualization tool (Neo4j Browser) can be used to visualize the data map, and a quick search function based on multiple query methods such as keywords, attributes, and entities is provided so that users can easily query and browse the information in the data map.

[0083] Furthermore, as a refinement and extension of the specific implementation of step S104 in the above embodiment, in order to fully illustrate the specific implementation process in this embodiment, a specific example of step S104 is provided, which specifically includes the following methods.

[0084] In step S104, a search algorithm may also be designed, and the search algorithm supports fast retrieval based on multiple query methods such as keywords, attributes, entities, etc. The search algorithm may adopt a reverse index fast retrieval model.

[0085] Among them, the inverted index search of the inverted index fast retrieval model can be expressed as: Results = Index.search (Query), where Index is the inverted index, Query is the query condition entered by the user, and Results is the set of matching records.

[0086] Specifically, an inverted index is created for all preprocessed keywords, attributes, entities, etc. This is used to quickly find all documents or records containing a keyword or attribute. The constructed inverted index is stored in memory or an efficient storage system for fast access.

[0087] The specific implementation method is given below.

[0088] S1041: receiving a query statement input by a user, parsing query conditions such as keywords, attributes or entities, etc. Converting the parsed query conditions into a form suitable for searching in an inverted index.

[0089] S1042: Use the converted query condition to search for matching records in the inverted index.

[0090] S1043: Collect all matching records to form an initial result set.

[0091] The sorting algorithm of this embodiment is designed based on the adaptive weight distribution model and the multi-layer entity labeling results to intelligently sort the search results. The sorting process takes into account multiple factors such as weight distribution results, entity importance, and user preferences.

[0092] Here, a score can be introduced to measure the matching degree of the sorting process.

[0093] Score calculation formula: Score = Σ( * Fi), where Fi is the value of the feature (obtained from multi-layer entity annotation results and other data attribute extraction). The specific implementation method includes the following steps: S2041: Extract features for each data item and assign corresponding weights according to the adaptive weight assignment model. Extract hierarchical relationship features between entities using the result of multi-layer entity annotation.

[0094] S2042: Perform a linear combination of the extracted features and weights to calculate the score of each data item. The score formula can be expressed as: Score = Σ( * Fi), where Fi is the value of the feature. The calculated score is normalized so that it is within a suitable range.

[0095] S2043: Result sorting: sort the result set according to the calculated scores and return the sorted results to the user.

[0096] Furthermore, this embodiment also optimizes the performance of the search and sorting algorithms, such as using indexing technology and caching mechanisms to improve query efficiency.

[0097] The specific methods include: S3041: Compress the inverted index to reduce storage space usage. The index is stored in a distributed system to improve access concurrency and fault tolerance.

[0098] S3042: Cache the results of frequent queries to reduce the time of repeated calculations. Cache the sorted results to improve the query response speed.

[0099] S3043: Use multi-core processors or distributed computing resources to perform parallel computing on search and sorting algorithms. Improve and optimize search and sorting algorithms based on specific application scenarios and data characteristics.

[0100] S3044: Expand memory capacity and improve data access speed. Use high-speed storage devices such as SSD to increase the reading and writing speed of indexes and data.

[0101] In order to further improve the reliability of the data search recommendation result in step S105, the following provides an implementable method. In one embodiment, Figure 3 As shown, user experience and system value can be improved based on iterative optimization and feedback mechanism.

[0102] Specifically, the system regularly collects and analyzes data indicators such as user search behavior, click-through rate, and dwell time. By regularly collecting information such as user search keywords, search frequency, and search time, and through statistical analysis of these data, we can understand users' search preferences, demand trends, and hot topics. Combined with users' historical search records, we analyze users' search paths and search habits to identify users' possible search intentions and potential needs. We use machine learning algorithms to model user search behavior, predict users' future search trends, and provide data support for system optimization.

[0103] In this embodiment, the system will record the time users stay on the search results page, and analyze the user's attraction and attention to the search results content by comparing the stay time of different search results, different content types, and different user groups. Search results with longer stay time often mean that users have a higher interest and satisfaction with the content, which can be used as an important reference for optimizing search results. Through in-depth analysis of the stay time data, the user's behavior patterns on the search results page can be discovered, providing a basis for optimizing page layout and improving user experience.

[0104] This embodiment evaluates the search and ranking effects based on collected data indicators such as user satisfaction, search efficiency, click-through rate, and dwell time, and identifies existing problems and improvement points.

[0105] This embodiment sets different search and sorting strategies, conducts group tests on users, and compares the differences in satisfaction, search efficiency, click-through rate, dwell time and other indicators among different groups of users. It collects direct feedback from users on search results and sorting to guide the improvement and optimization of strategies. It adjusts strategies such as the adaptive weight allocation model, search and sorting algorithm, etc. according to the evaluation results to improve system performance.

[0106] The following is a specific implementation method to illustrate the specific process of dynamically adjusting the adaptive weight allocation, search and sorting algorithms, and index performance optimization strategy, iteratively performing evaluation and optimization, and combining the user feedback mechanism to form a closed loop.

[0107] S1051: Use key indicators such as user satisfaction, search efficiency, click-through rate, dwell time, etc. obtained during the performance evaluation phase, as well as specific opinions and suggestions from user feedback, as the basis for strategy adjustment.

[0108] S1052: Based on user feedback and search behavior analysis, re-evaluate which data attributes and entity relationships have a greater impact on search results, and may need to add or delete certain features.

[0109] This embodiment adjusts the weights of various features in the machine learning algorithm according to changes in user satisfaction and search efficiency, especially those that affect dimensions such as user attention, data timeliness, and rarity.

[0110] For adaptive weight allocation model tuning, new data sets are used or the amount of data is increased, and cross-validation and grid search are re-performed to find better model parameters.

[0111] According to the hot spots and trends of user search behavior, the search algorithm is optimized to enable it to more quickly locate the information that users are interested in. Then, according to changes in user satisfaction and click-through rate, the weights of various factors in the sorting algorithm are adjusted, such as adaptive weight distribution results, entity importance, user preferences, etc., to provide search result sorting that better meets user expectations.

[0112] In this embodiment, according to the growth of data volume and changes in user search behavior, the index technology is regularly updated to improve search efficiency. According to changes in user access patterns and hot data, the cache strategy is adjusted to reduce the number of database accesses and improve system response speed.

[0113] S1053: Based on the evaluation results and the content of the policy adjustment, formulate a detailed adjustment plan, including the adjustment schedule, specific steps, required resources, etc. According to the adjustment plan, gradually implement various adjustment measures to ensure the smooth progress of the adjustment. During the adjustment process, continuously monitor the changes in system performance and data indicators, and collect user feedback in a timely manner to evaluate and further optimize the adjustment effect.

[0114] S1054: After the implementation of the strategy adjustment, the effect evaluation is conducted again by comparing the user satisfaction, search efficiency, click-through rate, dwell time and other indicators before and after the adjustment, as well as the specific opinions and suggestions of the user feedback, to evaluate the effect of the strategy adjustment. According to the evaluation results, the necessary fine-tuning or further optimization of the strategy adjustment is carried out to ensure the continuous improvement of system performance.

[0115] The feedback mechanism in this embodiment can introduce a user feedback mechanism, allowing users to rate or mark search results. The system dynamically adjusts the weight distribution model according to user feedback to achieve adaptive optimization. This application can keep the user feedback channel unblocked, collect user opinions and suggestions in a timely manner, and form a closed-loop feedback mechanism. Through continuous iterative optimization, the user experience and system value are improved.

[0116] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0117] The following is an embodiment of a data discovery system based on weight allocation and entity labeling provided in an embodiment of the present disclosure. This system and the data discovery method based on weight allocation and entity labeling in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the data discovery system based on weight allocation and entity labeling, please refer to the embodiment of the data discovery method based on weight allocation and entity labeling mentioned above.

[0118] like Figure 4 As shown, the system includes: A data preprocessing module is used to perform preprocessing operations on data; The weight adjustment module is used to dynamically adjust the weight of each data item using a machine learning algorithm based on the actual characteristics of the data; The feature annotation module is used to configure the adaptive weight distribution model and annotate the actual features in the data using a multi-layer entity annotation method; The index construction scoring module is used to build an inverted index retrieval model. According to the adaptive weight allocation model, it extracts features from each data item, assigns corresponding weights, and performs score calculations. According to the scores, it sorts and provides feedback to users. The adjustment and optimization feedback module is used to dynamically adjust the adaptive weight distribution, search and sorting algorithms, and index performance optimization strategies by collecting and analyzing user search data, and iteratively perform evaluation and optimization, forming a closed loop in combination with the user feedback mechanism.

[0119] The data discovery system based on weight allocation and entity labeling involved in this application is mainly aimed at the problems of unclear data features and unclear data weights in a large amount of enterprise data and public data today. It uses adaptive weight allocation and a multi-layer entity labeling model to perform batch and systematic optimization of data, thereby completing the optimization of search functions and intelligentizing the recommendation structure, which facilitates users to retrieve system data and managers to manage related data systems.

[0120] This application can also improve the model optimization method during the continuous use of users, have strong adaptability to data processing, continuously adjust the data processing plan, and through the user feedback mechanism and iterative optimization strategy, the system can continuously improve according to user needs and feedback. This user participation method helps to enhance the interaction and connection between users and the system, and further improve the user experience.

[0121] like Figure 5 As shown, the present application also provides an electronic device, including a display module 103, a memory 102, a processor 101, and a computer program stored in the memory and executable on the processor 101, wherein the processor 101 implements the steps of a method for parsing and loading a GIM model of a power transmission project when executing the program.

[0122] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile systems, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing systems. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0123] In the embodiment of the present application, the processor 101 can be implemented by using at least one of an application-specific integrated circuit (ASIC), a programmable logic system (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. In some cases, such an implementation can be implemented in a controller. For software implementation, implementations such as processes or functions can be implemented with separate software modules that allow execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0124] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0125] The memory 102 may be used to store software programs and various data. The memory 102 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0126] The present application also provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the data discovery method based on weight allocation and entity labeling are implemented.

[0127] The storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data discovery method based on weight allocation and entity annotation, characterized in that: include: Perform preprocessing operations on the data; According to the actual characteristics of the data, the machine learning algorithm is used to dynamically adjust the weight of each data item; Configure an adaptive weight distribution model and use a multi-layer entity annotation method to annotate the actual features in the data; Construct an inverted index retrieval model, extract features from each data item according to the adaptive weight allocation model, assign corresponding weights, and perform score calculations. According to the scores, the data items are sorted and fed back to the user. By collecting and analyzing user search data, dynamically adjusting adaptive weight distribution, search and sorting algorithms, and index performance optimization strategies, and iteratively performing evaluation and optimization, a closed loop is formed in combination with a user feedback mechanism.

2. The data discovery method based on weight allocation and entity labeling according to claim 1, characterized in that: The step adopts a machine learning algorithm to dynamically adjust the weight of each data item, adopts the Pearson correlation coefficient to calculate the characteristics of the influence of data attributes and entity relationships on search results, and calculates the mean and standard deviation of data attributes and entity relationships respectively; Divide the covariance by the product of the standard deviation of the data attribute and the entity relationship to obtain the Pearson correlation coefficient; The Pearson correlation coefficient calculation formula is: is the covariance of X and Y; express The standard deviation of express The standard deviation of X represents the data attribute, Represents entity relationships; The covariance is calculated as: in, is the i-th observation value of the data attribute, is the i-th observation value of the entity relationship, is the mean of the data attributes, is the mean of entity relationships, and n is the number of observations.

3. The data discovery method based on weight allocation and entity labeling according to claim 2, characterized in that: In the method, the Pearson correlation coefficient threshold is set; The calculated Pearson correlation coefficient is compared with the Pearson correlation coefficient threshold; When the calculated Pearson correlation coefficient is greater than or equal to the Pearson correlation coefficient threshold, defining the data attribute for calculating the Pearson correlation coefficient as a feature; The effectiveness of the features was verified by cross-validation method.

4. The data discovery method based on weight allocation and entity labeling according to claim 3, characterized in that: In the method, a machine learning algorithm is used to train the features to learn the weight of each feature on the search results; The overall score is calculated based on the following formula: in, is the standardized feature value in the original feature set, is the weight corresponding to the eigenvalue; is the normalized value of the additional feature, is the weight corresponding to the value; i is an index variable used to traverse all features in the original feature set; n is the total number of features in the original feature set; The comprehensive score textscore is calculated based on the standardized values ​​of all features and their corresponding weights; the comprehensive score textscore is used to measure the degree of match between a search result and the user's query.

5. The data discovery method based on weight allocation and entity labeling according to claim 1, characterized in that: According to the results of adaptive weight allocation and multi-layer entity labeling, the data is optimized and processed, and the following steps are also included: Through the Transformer algorithm, after entity recognition, it is labeled and a labeled data set is constructed; Using the BERT model, we perform a hierarchical labeling process and use the output as features to feed a fully connected neural network. The location information of the entity in the text is input into the relationship classifier as an additional feature to assist in determining the hierarchical relationship; Label the hierarchical relationship tasks, use the cross entropy loss function to train the relationship classifier, and select the dam optimizer to update the model weights in real time; All entities are identified, the hierarchical relationship scores between each pair of entities are calculated, and a hierarchical relationship graph between entities is constructed based on the scoring results and the preset scoring result threshold.

6. The data discovery method based on weight allocation and entity labeling according to claim 5, characterized in that: In the method, the results of entity and hierarchical relationship annotation are integrated into the data graph to form a structured representation with multi-layer entity annotations; Construct nodes and edges of data graphs based on entity and hierarchical relationships; And configure the identified entities and their identifiers, hierarchical relationship annotation results into the data graph; Visualize the data map.

7. The data discovery method based on weight allocation and entity labeling according to claim 1, characterized in that: The steps to build an inverted index retrieval model also include: Receive the query statement input by the user and parse the query condition information; Match records in the inverted index retrieval model based on query condition information; Collect all matching records and build an initial result set; Based on the adaptive weight assignment model, features are extracted for each data item and weights are assigned. The hierarchical relationship features between entities are extracted using the results of multi-layer entity annotation. Linearly combine the extracted features and weights to calculate the score of each data item; The scoring formula is: Score = Σ( * Fi), where is the weight of the feature, Fi is the value of the feature; The calculated scores are normalized so that they are within a preset score range. A score result set is constructed based on the calculated scores, and the scores are sorted. The sorted score result set is returned to the user.

8. A data discovery system based on weight assignment and entity annotation, characterized in that: The system is used to implement the data discovery method based on weight allocation and entity labeling as described in any one of claims 1 to 7; the system includes: A data preprocessing module is used to perform preprocessing operations on data; The weight adjustment module is used to dynamically adjust the weight of each data item using a machine learning algorithm based on the actual characteristics of the data; The feature annotation module is used to configure the adaptive weight distribution model and annotate the actual features in the data using a multi-layer entity annotation method; The index construction scoring module is used to build an inverted index retrieval model. According to the adaptive weight allocation model, it extracts features from each data item, assigns corresponding weights, and performs score calculations. According to the scores, it sorts and provides feedback to users. The adjustment and optimization feedback module is used to dynamically adjust the adaptive weight distribution, search and sorting algorithms, and index performance optimization strategies by collecting and analyzing user search data, and iteratively perform evaluation and optimization, forming a closed loop in combination with the user feedback mechanism.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data discovery method based on weight allocation and entity labeling as described in any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data discovery method based on weight allocation and entity labeling as described in any one of claims 1 to 7 are implemented.

Images