Data management screening method

Abstract

The invention relates to a data management screening method, which comprises the following steps of: performing data preprocessing based on data acquired in an application system, and obtaining candidate data; based on current consultation information of a user and the candidate data, respectively extracting three types of data features, wherein the data features comprise semantic similarity features, behavior association degree features and scene matching degree features; and constructing a matching degree calculation model based on the data features, and calculating the matching degree between the current consultation information of the user and the candidate data. According to the method, three types of features of semantic similarity, behavior association degree and scene matching degree are fused, so that semantic matching can be combined, and user historical retrieval and behavior scene matching related replies can also be combined, and the reply accuracy is further improved; besides, by continuously optimizing model parameters, the problems that existing model parameters are fixed and the adaptability is reduced can be solved, and it is ensured that the reply quality is gradually improved in the long-term use process of the automatic reply system.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically a data management and filtering method. Background Technology

[0002] With the widespread use of internet applications, users' demand for consultations through application systems is increasing. For example, in the field of technology transfer, users often inquire about patent transfer paths, rights distribution, and industry-academia-research collaboration. To improve response efficiency, existing technologies mostly employ automated response systems, which achieve rapid replies by matching historical consultation responses or standard templates.

[0003] However, the application of automatic reply systems still has the following shortcomings: most methods in automatic replies are based solely on semantic similarity for matching, ignoring the correlation between user's historical search, browsing and other behavioral data and the consultation scenario; and the templates and reply information of automatic replies are mostly fixed templates set manually, which cannot be dynamically optimized according to new consultation data and user feedback, resulting in poor reply accuracy. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] This invention provides a data management and filtering method that solves the problems mentioned in the background section.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: a data management and filtering method, comprising the following steps:

[0008] Step 1: Based on the multi-user search data, browsing data, consultation data, and corresponding manual consultation response data and response templates collected in the application system, perform data preprocessing to obtain candidate data;

[0009] Step 2: Extract three types of data features based on the user's current consultation information and the candidate data. The data features include semantic similarity features, behavioral relevance features, and scene matching features.

[0010] Step 3: Construct a matching degree calculation model based on the data features, and calculate the matching degree between the user's current consultation information and the candidate data;

[0011] Step 4: Sort the candidate data according to their matching degree, and use a number of candidate data with high matching degree to output automatic response;

[0012] Step 5: Based on the user satisfaction labels of several automatic reply contents, construct an update model and update the weighted summation model.

[0013] Preferably, the semantic similarity feature extraction process is as follows:

[0014] Step 21: Construct a target domain thesaurus, which contains keywords for that domain;

[0015] Step 22: Enter the user's current inquiry information and candidate data Convert them into TF-IDF vectors respectively and Each element of each vector represents the weight of the corresponding keyword in the text;

[0016] Step 23: Calculate semantic similarity features using the cosine similarity formula. The calculation formula is as follows:

[0017]

[0018] In the formula, vector and dot product, and They are vectors and of Norm.

[0019] In a further preferred embodiment, the extraction process of the behavioral correlation features is as follows:

[0020] Step 24: Define the user's historical behavior set Each element in the user's historical behavior set is a keyword from the user's historical search or browsing content;

[0021] Step 25: Statistical analysis of candidate data Includes user history behavior collection Number of keywords ;

[0022] Step 26: Calculate the behavioral correlation feature using the following formula. :

[0023] .

[0024] In a further preferred embodiment, the extraction process of the scene matching degree feature is as follows:

[0025] Step 27: Construct scenario dimensions based on the consulting field, consulting time, and user type;

[0026] Step 28: Set the matching identifier for a single scene dimension as follows If the user's current inquiry information and candidate data Matching in the corresponding scenario dimension If it does not match

[0027] Step 29: Calculate the scene matching feature using the following formula. :

[0028] .

[0029] In a further preferred embodiment, the calculation formula for the matching degree calculation model is as follows:

[0030]

[0031] In the formula, The user's current inquiry information and candidate data The degree of matching between them For the label Data characteristics For the label The weight parameters of the data features, This is a bias term.

[0032] In a further preferred embodiment, the calculation formula for the updated model is as follows:

[0033]

[0034]

[0035] In the formula, For the number of iterations, For learning rate, loss function For weight parameters The partial derivatives, For loss function For bias terms The partial derivatives, and These are the weight parameters before and after the update, respectively. and These are the bias terms before and after the update, respectively.

[0036] In a further preferred embodiment, the loss function is... The calculation formula is as follows:

[0037]

[0038] In the formula, The loss function is used to predict the deviation between the actual matching degree and the matching degree. The number of user satisfaction ratings is indicated. For the predicted matching degree, This represents the actual degree of match.

[0039] In a further preferred embodiment, the data preprocessing includes the removal of invalid and noisy data and the standardization of the remaining data. The standardization process is based on a unified text encoding format, removes garbled characters and meaningless special symbols from the data text, and standardizes the description text of similar keywords in each domain.

[0040] (III) Beneficial Effects

[0041] Compared with existing technologies, the present invention provides a data management and filtering method, which has the following beneficial effects:

[0042] In this invention, by integrating three types of features—semantic similarity, behavioral relevance, and scene matching—it is possible not only to combine semantic matching but also to combine user history retrieval and behavioral scene matching with relevant responses, thereby improving the accuracy of responses.

[0043] In addition, by continuously optimizing the model parameters, the problems of fixed existing model parameters and decreased adaptability can be solved, ensuring that the quality of responses gradually improves during long-term use of the automatic response system. Attached Figure Description

[0044] Figure 1 A flowchart illustrating the data management and screening method based on the implementation plan; Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 A data management and filtering method includes the following steps:

[0047] Step 1: Based on the multi-user search data, browsing data, consultation data, and corresponding manual consultation response data and response templates collected in the application system, data preprocessing is performed to obtain candidate data. The data preprocessing includes the removal of invalid and noisy data, as well as the standardization of the remaining data. Invalid data may include highly similar duplicate consultation data, data without actual semantics, and consultation information outside the target domain. Noisy data may include typos and redundant modifiers in the consultation text. Data standardization may include removing garbled characters and meaningless special symbols from the data text, and organizing the processed data into a unified text encoding format, ensuring consistent textual representation of similar keywords within each domain. For example, words such as "transformation," "transfer," and "landing" are all unified as "transformation" in the field of technology transfer.

[0048] Step 2: Extract three types of data features based on the new user's current consultation information and candidate data. The data features include semantic similarity features, behavioral relevance features, and scene matching features.

[0049] In this embodiment, when extracting semantic similarity features, a target domain thesaurus containing keywords of that domain can be constructed. Then, the current consultation information of new users can be... and candidate data Convert them into TF-IDF vectors respectively and This involves assigning a weight to each keyword in the text to each element of the vector. Finally, the semantic similarity features can be calculated using the cosine similarity formula. The calculation formula is as follows:

[0050]

[0051] In the formula, vector and The dot product is used to reflect the consistency of vector direction; and They are vectors and of Norm, used to reflect the length of a vector; semantic similarity features Used to measure the user's current inquiry With candidate data The semantic consistency level, with a value range of . Furthermore, the closer the value is to 1, the better the semantic match.

[0052] TF-IDF vector and The calculation formula is as follows:

[0053]

[0054] In the formula, For words In the document In Weight; For words In the document The frequency of the word in the document is used to reflect its importance in a single document; For words The inverse document frequency across the entire document collection is used to reflect the distinctiveness of the word within the domain.

[0055] In this embodiment, when extracting behavioral correlation features, the user's historical behavior set can be defined first. Each element in the user's historical behavior set represents a keyword from the user's historical searches or browsing content. Candidate data can then be compiled. Includes user history behavior collection Number of keywords Finally, the behavioral correlation feature can be calculated using the following formula. :

[0056]

[0057] In the formula, These are keywords related to historical behavior, belonging to the user's historical behavior set. Individual keywords in the content, and activities extracted by users searching and browsing content. Behavioral relevance features. It can measure users' historical behavior and candidate data. The correlation reflects the degree of matching between user interests and candidate responses, with a value range of [value range missing]. Furthermore, the closer the value is to 1, the stronger the correlation.

[0058] In this embodiment, when extracting scene matching features, scene dimensions are first constructed based on the consultation domain, consultation time, and user type. Then, the matching identifier for each individual scene dimension is... If the user's current inquiry information and candidate data Matching in the corresponding scenario dimension If it does not match Finally, the scene matching degree feature can be calculated using the following formula. :

[0059]

[0060] In the formula, Including consulting field matching identifiers Consultation time matching identifier User type matching identifier Scene matching features Used to measure the user's current inquiry information and candidate data Consistency of consultation scenarios, with a value range of [missing information]. Furthermore, the closer the value is to 1, the better the scenario matches.

[0061] Step 3: Construct a matching degree calculation model based on data features, and calculate the matching degree between the user's current consultation information and candidate data. The calculation formula for the matching degree calculation model is as follows:

[0062]

[0063] In the formula, For the label Data characteristics; For the label The weight parameters of the data features correspond to the importance weights of semantics, behavior, and scene, respectively. The sum of the three is ≤1, and the initial value can be set according to experience and updated with the model. This is a bias term used to adjust the basic matching degree, preventing valid candidate data from being missed due to low feature values. Its value range can be set to... The initial value can be set based on experience and then updated as the model is updated. The user's current inquiry information and candidate data The degree of matching between them, which can be fused , , The overall matching score is used as the core criterion for selecting candidate data.

[0064] Step 4: Sort the candidate data based on their matching degree, and use a number of candidate data with high matching degree to output automatic response.

[0065] Step 5: Based on the user satisfaction labels of several automatic reply messages, construct and update the model, and update the weighted summation model. User satisfaction labels for automatic reply messages can come from user ratings or manual annotations by backend staff after querying past chat logs. The calculation formula for updating the model is as follows:

[0066]

[0067]

[0068] In the formula, The iteration number represents the number of iterations for updating model parameters. The learning rate controls the step size for parameter updates, preventing excessively large step sizes from causing model oscillations or excessively small step sizes from causing low training efficiency. Its value range can be set to... And it can be adjusted according to the loss convergence speed; loss function For weight parameters The partial derivatives, For loss function For bias terms The partial derivatives of are used to reflect the direction and magnitude of the weight update. If the value is positive, the weight decreases; if the value is negative, the weight increases. and These are the weight parameters before and after the update, respectively. and These are the bias terms before and after the update, respectively.

[0069] loss function The calculation formula is as follows:

[0070]

[0071] In the formula, The number of labeled samples for user satisfaction, i.e. the total number of labeled samples used for model training; The matching degree predicted by the model. The true matching degree of the samples is derived from manual annotation or user satisfaction conversion. The loss function is the deviation between the predicted and actual matching scores. It measures the discrepancy between the model's predicted and actual matching scores, and its value ranges from [value range missing]. The smaller the value, the more accurate the model prediction, and the goal of model updating is to minimize... .

[0072] In this embodiment, the triggering conditions for updating the matching degree calculation model include the following:

[0073] When the number of newly labeled samples exceeds a preset threshold, a batch update can be triggered to optimize model parameters by incorporating the newly added samples.

[0074] When the number of samples with user satisfaction ratings below a certain threshold for automatic replies reaches a certain number, an incremental update is triggered to optimize the matching logic for low satisfaction replies.

[0075] A full update is triggered after a certain number of days since the last update, integrating all historical data and feedback to ensure the long-term adaptability of the model.

[0076] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention; the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0077] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data management and filtering method, characterized in that, Includes the following steps: Step 1: Based on the multi-user search data, browsing data, consultation data, and corresponding manual consultation response data and response templates collected in the application system, perform data preprocessing to obtain candidate data; Step 2: Extract three types of data features based on the user's current consultation information and the candidate data. The data features include semantic similarity features, behavioral relevance features, and scene matching features. Step 3: Construct a matching degree calculation model based on the data features, and calculate the matching degree between the user's current consultation information and the candidate data; Step 4: Sort the candidate data according to their matching degree, and use a number of candidate data with high matching degree to output automatic response; Step 5: Based on the user satisfaction labels of several automatic reply contents, construct an update model and update the weighted summation model.

2. The data management and filtering method according to claim 1, characterized in that: The extraction process of the semantic similarity features is as follows: Step 21: Construct a target domain thesaurus, which contains keywords for that domain; Step 22: Record the user's current inquiry information. and candidate data Convert them into TF-IDF vectors respectively and Each element of each vector represents the weight of the corresponding keyword in the text; Step 23: Calculate semantic similarity features using the cosine similarity formula. The calculation formula is as follows: In the formula, vector and dot product, and They are vectors and of Norm.

3. The data management and filtering method according to claim 2, characterized in that: The extraction process of the behavioral correlation features is as follows: Step 24, Define the user's historical behavior set. Each element in the user's historical behavior set is a keyword from the user's historical search or browsing content; Step 25: Statistical analysis of candidate data Includes user history behavior collection Number of keywords Step 26: Calculate the behavioral correlation feature using the following formula. : .

4. The data management and filtering method according to claim 3, characterized in that: The extraction process of the scene matching feature is as follows: Step 27, construct scene dimensions according to consultation field, consultation time, and user type; Step 28, set the matching identifier of a single scene dimension as... If the user's current inquiry information and candidate data Matching in the corresponding scenario dimension If it does not match Step 29: Calculate the scene matching feature using the following formula. : .

5. A data management and filtering method according to any one of claims 1-4, characterized in that: The calculation formula for the matching degree calculation model is as follows: In the formula, The user's current inquiry information and candidate data The degree of matching between them For the label Data characteristics For the label The weight parameters of the data features, This is a bias term.

6. The data management and filtering method according to claim 5, characterized in that: The calculation formula for the updated model is as follows: In the formula, For the number of iterations, For learning rate, loss function For weight parameters The partial derivatives, loss function For bias terms The partial derivatives, and These are the weight parameters before and after the update, respectively. and These are the bias terms before and after the update, respectively.

7. The data management and filtering method according to claim 6, characterized in that: The loss function The calculation formula is as follows: In the formula, The loss function is used to predict the deviation between the actual matching degree and the matching degree. The number of user satisfaction ratings is indicated. For the predicted matching degree, This represents the actual degree of match.

8. The data management and filtering method according to claim 1, characterized in that: The data preprocessing includes the removal of invalid and noisy data, as well as the standardization of the remaining data.

9. The data management and filtering method according to claim 8, characterized in that: The standardization process is based on a unified text encoding format, removes garbled characters and meaningless special symbols from the data text, and standardizes the textual descriptions of similar keywords in each domain.

Images