Intelligent processing method and device for information source discovery

Abstract

The invention discloses an intelligent processing method and device for information source discovery. The method comprises the following steps: acquiring user demand information; analyzing the user demand information to obtain an information resource table; according to the information resource table, automatically crawling a science and technology intelligence text to obtain an intelligence text database; the information text database comprises N information texts, and N is a positive integer; performing theme extraction on the intelligence text to obtain a keyword list of each theme cluster; and processing new demand information of the user according to the keyword list of each theme cluster to obtain a theme test result. According to the method, the webpage context can be constructed according to the webpage link relationship, the link relationship and the webpage content are integrated to automatically discover and analyze the information source, and the labor cost and the time cost are reduced. Through comparative analysis and identification of unknown entity characteristic values, a new information source theme can be found, the comprehensiveness of information is enhanced, and the information collection efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge discovery, and in particular to an intelligent processing method and device for source discovery. Background Art

[0002] Text mining is essentially a process of deeply exploring and utilizing text data, aiming to extract those implicit, previously unknown, but potentially valuable information and knowledge from text data. Traditional topic mining methods, such as LDA, etc., often rely on fixed probability models and are difficult to adapt to changing text data and complex semantic structures. The BERTopic model can model the topics of short text content and display the potential hierarchical structure of topics in a clustering manner. However, BERTopic can only mine topics in a fixed dimension, resulting in insufficient universality, and is affected by word-level popularity bias, leading to the problem of homogeneous topic results. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an intelligent processing method and device for source discovery, which can perform intelligent discovery and analysis of information sources based on user needs. By introducing a dynamic document embedding optimizer and a popularity correction regularization term, the topic consistency, topic diversity, and comprehensive quality indicators are improved, effectively solving the challenge of difficult timely identification of text topics in the prior art, realizing efficient and accurate identification of topics in a large amount of network data, and significantly improving the effect of text topic mining.

[0004] To solve the above technical problem, in the first aspect of the embodiments of the present invention, an intelligent processing method for source discovery is disclosed, and the method includes:

[0005] S1, obtaining user demand information;

[0006] S2, parsing the user demand information to obtain an information resource table;

[0007] S3, automatically crawling scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, and N is a positive integer;

[0008] S4, performing topic extraction on the intelligence texts to obtain a keyword list for each topic cluster;

[0009] S5, processing the new user demand information according to the keyword list of each topic cluster to obtain a topic test result.

[0010] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the parsing the user demand information to obtain an information resource table includes:

[0011] S21. Extract keywords from the user requirement information to obtain key requirement information;

[0012] S22. Process the key requirement information to obtain a dynamic Prompt;

[0013] S23. Use a large model to process the dynamic Prompt to obtain an information resource table.

[0014] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the extracting keywords for each topic cluster from the intelligence text includes:

[0015] S41. Perform document embedding on the intelligence text to obtain an embedding vector;

[0016] S42. Perform dimensionality reduction processing on the embedding vector to obtain a dimensionality-reduced embedding vector;

[0017] S43. Perform clustering processing on the dimensionality-reduced embedding vector to obtain class vectors of K topics, where K is a positive integer;

[0018] S44. Perform topic representation on the class vectors of each topic to obtain a keyword list for each topic cluster.

[0019] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the performing document embedding on the intelligence text to obtain an embedding vector includes:

[0020] Use a BERT document embedding processing model to perform document embedding on the intelligence text to obtain an embedding vector;

[0021] The expression of the BERT document embedding processing model is:

[0022] Embedding(d) = BERT(d)

[0023] Where d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

[0024] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the performing dimensionality reduction processing on the embedding vector to obtain a dimensionality-reduced embedding vector includes:

[0025] S421. Process the embedding vector to obtain 8 candidate embedding dimension vectors;

[0026] S422. Use a dimension selector based on the attention mechanism to process the 8 candidate embedding dimension vectors to obtain a dimensionality-reduced embedding vector.

[0027] As an alternative implementation, in the first aspect of the embodiments of the present invention, the process of performing topic representation on the class vectors of each topic to obtain a keyword list for each topic cluster includes:

[0028] S441, calculating the importance degree of each topic's class vector to obtain the importance degree score of each topic;

[0029] The expression for the importance degree score is:

[0030]

[0031] where F c is the importance degree score of topic cluster c, w represents a candidate word, C represents the set of all topic clusters, |c′∈C:w∈c| represents the number of other topic clusters containing word w, and |C| represents the number of topic clusters;

[0032] S442, according to the importance degree scores of each topic, selecting the top N words as the topic keywords of each topic cluster to obtain the keyword list of each topic cluster.

[0033] As an alternative implementation, in the first aspect of the embodiments of the present invention, the process of processing the user's new demand information according to the keyword list of each topic cluster to obtain a topic test result includes:

[0034] S51, using a large language model to parse the user's new demand information to obtain the user's intent keyword vector;

[0035] S52, using a similarity calculation model to calculate the similarity between the keyword list of each topic cluster and the user's intent keyword vector to obtain a similarity value;

[0036] The expression for the similarity calculation model is:

[0037]

[0038] where S(A,B) is the similarity value, A is the user's intent keyword vector, B is the keyword list of each topic cluster, A i and B i are the values of the corresponding vectors in the i-th dimension;

[0039] S53, comparing the similarity value with a preset threshold. When the similarity value is lower than the preset threshold, the topic test result is that a new topic is discovered and the new topic keywords are stored; when the similarity value is greater than the preset threshold, the topic test result is that no new topic is discovered.

[0040] In the second aspect of the embodiments of the present invention, an intelligent processing device for source discovery is disclosed, and the device includes:

[0041] An information acquisition module, configured to acquire user demand information;

[0042] A demand analysis module, configured to analyze the user demand information to obtain an information resource table;

[0043] An intelligence text database construction module, which automatically crawls scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, and N is a positive integer;

[0044] A theme extraction module, configured to extract themes from the intelligence texts to obtain a keyword list for each theme cluster;

[0045] A theme testing module, configured to process new user demand information according to the keyword list of each theme cluster to obtain a theme test result.

[0046] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the analyzing the user demand information to obtain an information resource table includes:

[0047] S21, extracting keywords from the user demand information to obtain demand key information;

[0048] S22, processing the demand key information to obtain a dynamic Prompt;

[0049] S23, using a large model to process the dynamic Prompt to obtain an information resource table.

[0050] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the extracting themes from the intelligence texts to obtain a keyword list for each theme cluster includes:

[0051] S41, performing document embedding on the intelligence texts to obtain embedding vectors;

[0052] S42, performing dimensionality reduction processing on the embedding vectors to obtain dimensionality-reduced embedding vectors;

[0053] S43, performing clustering processing on the dimensionality-reduced embedding vectors to obtain class vectors for K themes, where K is a positive integer;

[0054] S44, performing theme representation on the class vectors of each theme to obtain a keyword list for each theme cluster.

[0055] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the performing document embedding on the intelligence texts to obtain embedding vectors includes:

[0056] Use the BERT document embedding processing model to perform document embedding on the intelligence text to obtain an embedding vector;

[0057] The expression of the BERT document embedding processing model is:

[0058] Embedding(d) = BERT(d)

[0059] where d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

[0060] As an alternative implementation, in the second aspect of the embodiments of the present invention, the processing of dimension reduction on the embedding vector to obtain a dimension-reduced embedding vector includes:

[0061] S421. Process the embedding vector to obtain 8 candidate embedding dimension vectors;

[0062] S422. Use a dimension selector based on the attention mechanism to process the 8 candidate embedding dimension vectors to obtain a dimension-reduced embedding vector.

[0063] As an alternative implementation, in the second aspect of the embodiments of the present invention, the processing of the class vector of each theme to obtain a keyword list for each theme cluster includes:

[0064] S441. Calculate the importance degree of each theme's class vector to obtain the importance degree score of each theme;

[0065] The expression of the importance degree score is:

[0066]

[0067] where F c is the importance degree score of the theme cluster c, w represents a candidate word, C represents the set of all theme clusters, |c′∈C:w∈c| represents the number of other theme clusters containing the word w, and |C| represents the number of theme clusters;

[0068] S442. According to the importance degree scores of each theme, select the top N words as the theme keywords of each theme cluster to obtain the keyword list of each theme cluster.

[0069] As an alternative implementation, in the second aspect of the embodiments of the present invention, the processing of the user's new demand information according to the keyword list of each theme cluster to obtain a theme test result includes:

[0070] S51. Parse the new requirement information of the user using a large language model to obtain the intent keyword vector of the user.

[0071] S52. Use a similarity calculation model to calculate the similarity between the keyword list of each topic cluster and the intent keyword vector of the user to obtain a similarity value.

[0072] The expression of the similarity calculation model is:

[0073]

[0074] where S(A, B) is the similarity value, A is the intent keyword vector of the user, B is the keyword list of each topic cluster, A i and B i are the values of the corresponding vectors in the i-th dimension.

[0075] S53. Compare the similarity value with a preset threshold. When the similarity value is lower than the preset threshold, the topic test result is that a new topic is discovered, and the new topic keywords are stored; when the similarity value is greater than the preset threshold, the topic test result is that no new topic is discovered.

[0076] The third aspect of the present invention discloses another intelligent processing device for source discovery, and the device includes:

[0077] A memory storing executable program code;

[0078] A processor coupled to the memory;

[0079] The processor calls the executable program code stored in the memory and executes some or all of the steps in the intelligent processing method for source discovery disclosed in the first aspect of the embodiments of the present invention.

[0080] The fourth aspect of the present invention discloses a computer-readable storage medium, and the computer-readable storage medium stores computer instructions, which are used to execute some or all of the steps in the intelligent processing method for source discovery disclosed in the first aspect of the embodiments of the present invention when called.

[0081] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0082] The design of the present invention realizes an intelligent processing method for source discovery, which can construct web page context based on web page link relationships, and comprehensively analyze the link relationships and web page content for automatic discovery and analysis of information sources; it reduces labor costs and time costs; through comparative analysis and identification of characteristic values of unknown entities, it can discover new source themes and enhance the comprehensiveness of information; it improves the efficiency and accuracy of information collection: it overcomes the lack of universality caused by BERTopic's ability to mine themes only in fixed dimensions and the problem of theme result homogenization caused by the influence of word-level popularity bias, and realizes the automatic selection of the optimal theme clustering vector dimension and the effective correction of popular words. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0084] Figure 1 It is a schematic flowchart of an intelligent processing method for source discovery disclosed in an embodiment of the present invention;

[0085] Figure 2 It is a schematic flowchart of another intelligent processing method for source discovery disclosed in an embodiment of the present invention;

[0086] Figure 3 It is a schematic flowchart of the theme extraction process disclosed in an embodiment of the present invention;

[0087] Figure 4 It is a schematic structural diagram of an intelligent processing device for source discovery disclosed in an embodiment of the present invention;

[0088] Figure 5 It is a schematic structural diagram of another intelligent processing device for source discovery disclosed in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0089] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0090] The terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or equipment.

[0091] Reference to "embodiment" herein means that a particular feature, structure or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0092] The present invention discloses an intelligent processing method and device for source discovery. The method includes: obtaining user demand information; parsing the user demand information to obtain an information resource table; automatically crawling scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, where N is a positive integer; extracting themes from the intelligence texts to obtain a keyword list for each theme cluster; and processing new user demand information according to the keyword list for each theme cluster to obtain a theme test result. The method of the present invention can construct a web page context according to the web page link relationship, and comprehensively analyze the link relationship and web page content for automatic source discovery, reducing the labor cost and time cost. Through comparative analysis and identification of the feature values of unknown entities, new source discovery themes can be found, enhancing the comprehensiveness of information and improving the efficiency and accuracy of information collection. The following will be described in detail respectively.

[0093] Embodiment 1

[0094] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of an intelligent processing method for source discovery disclosed in an embodiment of the present invention. Among them, Figure 1 the described intelligent processing method for source discovery is applied to the field of knowledge discovery technology, and the embodiments of the present invention do not make limitations. As Figure 1 shown, the intelligent processing method for source discovery may include the following operations:

[0095] S1. Obtain user demand information;

[0096] S2. Parse the user demand information to obtain an information resource table;

[0097] S3. Automatically crawl scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, where N is a positive integer;

[0098] According to the content of the information resource table, use an automatic crawling tool, such as a crawler tool, to extract relevant useful information from the web page, including the web page title, body content, publication time, author, etc., and store the extracted data in the Redis intelligence text database in JSON format;

[0099] S4. Perform topic extraction on the intelligence texts to obtain a keyword list for each topic cluster;

[0100] S5. Process the user's new demand information according to the keyword list of each topic cluster to obtain a topic test result.

[0101] Optionally, the parsing of the user demand information to obtain an information resource table includes:

[0102] S21. Extract keywords from the user demand information to obtain demand key information;

[0103] The following method is used for keyword extraction:

[0104] Construct a lattice network for the user demand information. The construction of the lattice network is an existing technology in the art. The set of nodes of the constructed lattice network is N = {n0, n1,...}, and the set of arcs is E = {e0, e1,...}. Each node n i ∈N contains a time marker t(n i ), and the arc e u→v represents the arc connecting node u to node v. Let the nodes in the keyword network be NS = {N0, N1,...}, representing a set of arcs connecting node N i to node N j in the keyword network.

[0105] 1. For any n i ∈N i , n j ∈N i , from t(n i ) < t(n j ), it can be obtained that i ≤ j.

[0106] 2. For any n i ∈N i , n j ∈N i , from t(n i ) = t(n j ), it can be obtained that i = j.

[0107] 3. Any e u→v ∈ E, if for u ∈ N i and v ∈ N j , e u→v is equivalent to a group of then i ≤ m ≤ n ≤ j. (Here n = m + 1, and the two endpoints of e u→v belong to N m and N n .)

[0108] Convert the lattice network into a keyword network, and the generation algorithm is as follows:

[0109] 1. Convert the keyword into the corresponding syllable string: K1…K M (M is the number of keyword syllables). Traverse all nodes N in the Lattice network and find the node n k that matches K1

[0110] 2. Using as the end point, search backward for similar nodes n k-1 in reverse chronological order. If there is no arc between n k-1 and , then n k-1 also belongs to Continue to search for the previous node of n k-1 . Otherwise, the search stops.

[0111] 3. Using as the starting point, search forward for similar nodes n k+1 in chronological order. If there is no arc between n k+1 and , then n k+1 belongs to Continue to search for the next node of n k+1 . Otherwise, the search stops.

[0112] 4. For e k→k+1 belonging to n k+1 belonging to

[0113] 5. All arcs e contained in k→k+1 form the keyword confusion network.

[0114] 6. For each arc If u belongs to N s , n i belongs to N t . When t = s + 1, belong to Otherwise belong to (s + 1 ≤ n ≤ t).

[0115]

[0116] Wherein, w(l) and w(e) are the words corresponding to arcs l and e, and sim(·, ·) refers to the similarity between two words refers to the arc and the normalized time overlap of e. Additionally, each node n in the Lattice i ∈ N contains a time stamp t(n i ) which will be used as a constraint condition in judging the termination of the search

[0117] The method of data normalization is adopted to confirm the keywords. First, normalize the linguistic probability likelihood scores for each candidate:

[0118]

[0119] Where, Value is the set of linguistic score values in the keyword network; minValue and maxValue are respectively the minimum and maximum values in the set; x is the score before normalization; y is the score after normalization. Then, calculate the sum of the scores and mark it as the final score of the arc in the keyword network

[0120] Select the arc candidate with the highest score in each keyword network and judge whether it is a keyword. If so, output the keyword and the time stamp; if not, skip it. Move on to the next keyword network. In this way, the detection of keywords is completed

[0121] S22. Process the key information of the demand to obtain a dynamic Prompt (prompt word);

[0122] S23. Use the large model to process the dynamic Prompt to obtain an information resource table

[0123] Specifically: Call the Web search engine in the way of Agent through the large model, send a request to the large model using the generated Prompt, obtain the information source data related to the demand, and the result returns the information resource table in JSON format, including the website name, corresponding link, preliminary introduction, etc

[0124] The large model first conducts pre-training on a large amount of data through self-supervised learning or semi-supervised learning, and then further optimizes its performance and capabilities through methods such as instruction fine-tuning and human alignment. The large model has characteristics such as a large number of parameters, a large amount of training data, and a large amount of computing resources, and has the capabilities of solving general tasks, following human instructions, and performing complex reasoning

[0125] Optionally, the topic extraction of the intelligence text to obtain a keyword list for each topic cluster includes:

[0126] S41. Perform document embedding on the intelligence text to obtain an embedding vector;

[0127] S42. Perform dimensionality reduction processing on the embedding vector to obtain a dimensionality-reduced embedding vector;

[0128] S43. Perform clustering processing on the dimensionality-reduced embedding vector to obtain class vectors for K topics, where K is a positive integer;

[0129] Optionally, the clustering method is:

[0130] (1) Randomly select a point from the initial data points X = {x1, x2,..., x n} as the cluster center, and set a threshold, where n is the number of data;

[0131] (2) Calculate all the points within the threshold distance range of the cluster center, record them in the set M, and classify them into the cluster c. The distance metric formula is:

[0132]

[0133] Z(c k ) is the distance value of the cluster c k , x i is the i-th data point, and c j is the j-th cluster;

[0134] (3) Calculate the distance vector between each element in the set M and the cluster c and accumulate them to obtain the offset vector M h (x):

[0135]

[0136] (4) The cluster center moves along the direction of the offset vector by a distance equal to the modulus of the offset vector;

[0137] (5) Loop through steps (2), (3), and (4) until the offset vector value reaches the threshold set by the termination condition, and record the cluster center at this time;

[0138] (6) Loop through steps (1) to (5) until all data points are traversed;

[0139] (7) Finally, calculate the number of times each element is accessed by each class, and classify the element into the cluster class with the most access times.

[0140] S44. Perform topic representation on the class vectors of each topic to obtain a keyword list for each topic cluster.

[0141] Optionally, the document embedding of the intelligence text to obtain an embedding vector includes:

[0142] Using a BERT document embedding processing model to perform document embedding on the intelligence text to obtain an embedding vector;

[0143] The expression of the BERT document embedding processing model is:

[0144] Embedding(d) = BERT(d)

[0145] where d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

[0146] Optionally, the dimensionality reduction processing of the embedding vector to obtain a dimensionality-reduced embedding vector includes:

[0147] S421. Process the embedding vector to obtain 8 candidate embedding dimension vectors;

[0148] The specific method is:

[0149] Process the embedding vector x = {x1, x2…, x n} to obtain a connection structure diagram:

[0150] N(x i ) = {x j |s = 1, 2,…, n, s ≠ j, d(x i , x j ) ≤ d(x s , x j )}

[0151] n is the vector dimension, N(x i ) is the connection structure diagram corresponding to x i , and d(x i , x j ) represents the Euclidean distance between x i , x j ; i = 1, 2,…, n, j = 1, 2,…, n;

[0152] Calculate the connection probability p i , x j between two points x ij :

[0153]

[0154] σ i is the scale parameter corresponding to the point x i , which is a pre-set parameter;

[0155] Assume the low-dimensional vector is Y = {y1, y2, …, y m}, where m is the dimension;

[0156] Construct the similarity coefficient q ij as:

[0157] q ij = (1 + a||y i - y j || 2b ) -1

[0158] where a and b are hyperparameters;

[0159] Construct the objective function:

[0160]

[0161] In the formula, η is the learning rate, which controls the step size of each update.

[0162] By randomly initializing Y and continuously updating and iterating, the objective function is minimized, and at this time, the structure in the high-dimensional space is retained in the low-dimensional space, obtaining the final dimensionality reduction result Y.

[0163] Process the dimensionality reduction result Y to generate 8 different-dimensional vectors from 32, 64 to 1024 as 8 candidate embedding dimension vectors.

[0164] S422. Use a dimension selector based on the attention mechanism to process the 8 candidate embedding dimension vectors to obtain the dimensionality reduction embedding vector.

[0165] Specifically, determine the best candidate dimension by comparing the classification accuracy rates of the candidate dimensions in terms of the issuing agencies and types of policy texts.

[0166] Extract information through the self-attention mechanism, and connect a fully connected network after the self-attention layer as the output layer to output the probability of the corresponding category. Select the best candidate dimension by evaluating the accuracy rates of different candidate dimensions in the classification task.

[0167]

[0168] out = softmax(MLP(x hid ))

[0169] Among them, q, k, and v are 8 candidate embedding dimension vectors corresponding to each document, and Q, K, and V are embedding dimension vectors composed of all corresponding texts; Attention is the attention mechanism, softmax represents the Softmax function, T represents transpose, out represents the output vector, that is, the dimensionality-reduced embedding vector, and d k = q 2 + k 2 + v 2 , and MLP represents a multi-layer perceptron.

[0170] Optionally, representing the class vector of each theme to obtain a keyword list for each theme cluster, including:

[0171] S441, calculating the importance degree of the class vector of each theme to obtain the importance degree score of each theme;

[0172] The expression of the importance degree score is:

[0173]

[0174] Among them, F c is the importance degree score of theme cluster c, w represents a candidate word, C represents the set of all theme clusters, |c′∈C:w∈c| represents the number of other theme clusters containing word w, and |C| represents the number of theme clusters;

[0175] S442, according to the importance degree scores of each theme, selecting the top N words as the theme keywords of each theme cluster to obtain the keyword list of each theme cluster.

[0176] Optionally, processing the new demand information of the user according to the keyword list of each theme cluster to obtain a theme test result, including:

[0177] S51, using a large language model to parse the new demand information of the user to obtain the intent keyword vector of the user;

[0178] The large language model is a type of large model.

[0179] S52, using a similarity calculation model to calculate the similarity between the keyword list of each theme cluster and the intent keyword vector of the user to obtain a similarity value;

[0180] The expression of the similarity calculation model is:

[0181]

[0182] Among them, S(A,B) is the similarity value, A is the intent keyword vector of the user, B is the keyword list of each theme cluster, Ai and B i is the value of the corresponding vector in the i-th dimension;

[0183] S53. Compare the similarity value with a preset threshold. When the similarity value is lower than the preset threshold, the theme test result is that a new theme is discovered, and the new theme keywords are stored; when the similarity value is greater than the preset threshold, the theme test result is that no new theme is discovered.

[0184] It can be seen that the present invention designs and implements an intelligent processing method for source discovery, which can construct web page context according to web page link relationships, and comprehensively perform automatic discovery and analysis of information sources based on link relationships and web page content; reduces labor costs and time costs; through comparative analysis and identification of unknown entity feature values, new source themes can be discovered, enhancing the comprehensiveness of information; improves information collection efficiency and accuracy: overcomes the lack of universality caused by BERTopic's ability to only mine themes in fixed dimensions and the problem of theme result homogenization caused by being affected by word-level popularity bias, and realizes the automatic selection of the optimal theme clustering vector dimension and the effective correction of popular words.

[0185] Embodiment 2

[0186] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another intelligent processing method for source discovery disclosed in the embodiments of the present invention. Among them, Figure 2 The described intelligent processing method for source discovery is applied to the field of knowledge discovery technology, and the embodiments of the present invention do not make limitations. As Figure 2 shown, the intelligent processing method for source discovery may include the following operations:

[0187] 1. Parse the user input, extract the key information in the requirements, then generate a dynamic Prompt according to the extracted key information, and then call the Web search engine in the form of an Agent through a large model, send a request to the large model using the generated Prompt, and obtain information source data related to the requirements. The result returns the website name, corresponding link, and preliminary introduction in JSON format.

[0188] 2. Obtain source information: Obtain content according to relevant source links, including web page title, web page content, publication time, author, link, and store the data in the redis database.

[0189] 3. Theme extraction

[0190] The BertTopic model is used for topic extraction. BertTopic is a powerful tool that combines the pre-trained model BERT and topic modeling. Its core idea is to capture the semantic information of documents through the BERT model, and use topic modeling techniques to cluster these semantic information to derive topics. BertTopic uses BERT embedding and c-TF-IDF method (an improved TF-IDF representation that can more accurately represent the frequency of words in clusters) to create clusters, allowing easy-to-interpret topics to be generated while maintaining important vocabulary in topic descriptions. The main steps of the BertTopic algorithm are as follows:

[0191] 3.1 Document Embedding

[0192] The BERT document embedding processor inputs the document text in the database into the pre-trained BERT model to generate the document's embedding vector. This process can be expressed as:

[0193] The BERT document embedding processing model expression is:

[0194] Embedding(d)=BERT(d)

[0195] Among them, d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

[0196] 3.2 Dimensionality reduction

[0197] Through dimensionality reduction, the embedding vector is generated into vectors of 8 different dimensions, ranging from 32, 64 to 1024. In this process, the document embedding dimension optimizer DDEO is proposed to enhance the dimensionality reduction process, and the dimension selection is performed through an automated method. This embodiment designs two methods for dimension selection: downstream tasks and word vector evaluation.

[0198] The specific method of dimensionality reduction is:

[0199] For the embedding vector x = {x1, x2…, x n} to process and obtain the connection structure diagram:

[0200] N(x i )={x j |s=1,2,…,n,s≠j,d(x i ,x j )≤d(x s ,x j )}

[0201] n is the vector dimension, N(x i ) is x i The corresponding connection structure diagram, d(xi , x j ) represents x i , x j The Euclidean distance between; i = 1, 2, …, n, j = 1, 2, …, n;

[0202] Calculate the connection probability p between two points x i , x j and x ij :

[0203]

[0204] σ i is the scale parameter corresponding to point x i , which is a pre-set parameter;

[0205] Assume the low-dimensional vector is Y = {y1, y2, …, y m} and m is the dimension;

[0206] Construct the similarity coefficient q ij as:

[0207] q ij = (1 + a||y i - y j || 2b ) -1

[0208] where a and b are hyperparameters;

[0209] Construct the objective function:

[0210]

[0211] In the formula, η is the learning rate, which controls the step size of each update.

[0212] By randomly initializing Y and continuously updating and iterating, the objective function is minimized, and at this time, the structure in the high-dimensional space is retained in the low-dimensional space, obtaining the final dimensionality reduction result Y.

[0213] 1) The dimension selector based on the downstream task determines the best candidate dimension by inputting the dimensionality-reduced embedding vector into the classification task and comparing the classification accuracies of the candidate dimensions on the issuing agency and type of the policy text. The dimension selector based on the downstream task is divided into two types: the dimension selector based on K-Means (KDS) and the dimension selector based on the attention mechanism (ADS).

[0214] a. The dimension selector based on K-Means (KDS):

[0215] The K-Means-based Dimension Selector (KDS) first reduces the dimensionality of the document vectors to different candidate embedding dimensions; then it clusters the low-dimensional vectors of all documents in each reduced candidate dimension and calculates the accuracy rate of each candidate dimension respectively. Finally, the candidate dimension with the highest accuracy rate under each dataset is the optimal embedding dimension OND.

[0216] b. Attention-based Dimension Selector (ADS):

[0217] Process the dimensionality reduction result Y through an embedding method to obtain the document embedding x;

[0218] q, k, v = linear qkv (x)

[0219]

[0220] out = softmax(MLP(x hid ))

[0221] Process x to obtain q, k, v, where q, k, v are the vectors corresponding to 8 candidate embedding dimensions for each document, and Q, K, V are the embedding dimension vectors composed of the corresponding entire text; Attention is the attention mechanism, softmax represents the Softmax function, T represents the transpose, out represents the output vector, that is, the dimensionality reduction embedding vector, d k = q 2 + k 2 + v 2 , and MLP represents the multi-layer perceptron.

[0222] Based on the trained ADS model, calculate the classification accuracy rate of different candidate dimensions on the test set, and output the candidate dimension with the high classification accuracy rate as the optimal embedding dimension.

[0223] 2) Word Vector-based Evaluation Dimension Selector (WDS):

[0224] This embodiment examines the dimension selection problem from the perspective of word vectors and designs an evaluation method based on analogical reasoning and document vector contrast. The analogical reasoning method in vector evaluation refers to mining the linear relationship of the text itself. Classify the text and obtain the embedding vectors. Subsequently, calculate the similarity ratio of the document embeddings with analogical relationships.

[0225] The evaluation method for the document vector contrast in this embodiment is as follows:

[0226]

[0227] The embedding quality is evaluated by comparing the similarity of document embeddings on the publishing institutions: for documents belonging to the same publishing institution, the higher the vector similarity ES of their embeddings, the better; for documents from different publishing institutions, the lower the vector similarity ES of their embeddings, the better. NESD is the normalized difference in vector similarity. The larger the difference in vector similarity, the better the embedding quality. At the same time, the embedding vectors with a difference in vector similarity less than the preset threshold are discarded. Then, the dimensionality reduction process in 3.2 is performed. s The higher, the better. For documents from different publishing institutions, the vector similarity ES d should be as low as possible. NESD is the normalized difference in vector similarity. The larger the difference in vector similarity, the better the embedding quality. At the same time, the embedding vectors with a difference in vector similarity less than the preset threshold are discarded. Then, the dimensionality reduction process in 3.2 is performed.

[0228] Furthermore, to improve the simplification metric, two metrics, ARS and NESD, are fused into ARS - NESD. ARS is the cosine similarity, with a value range between 0 and 1, where the larger the number, the better the evaluation of the candidate dimension in the word vector.

[0229] 3.3 Clustering Analysis

[0230] In this embodiment, the PDR (Popularity Debias Regularization) module is introduced to optimize the clustering matching relationship between word vectors and topic vectors through a probability clustering module with regularization, improving the clustering quality. The idea of the probability clustering regularization module is as follows: regarding the topic vectors as clustering centers and the word vectors as clustering samples; making the topic vectors more evenly and dispersedly distributed in the semantic space through constraint conditions. Specifically, the two core constraints of PDR are: 1) the distance between topic vectors is as large as possible; 2) each word vector belongs to only one topic vector as much as possible. To achieve the above goals, first, the constraint on the size of the word vector clustering cluster corresponding to each clustering center (topic vector) is defined to avoid the occurrence of empty clusters or duplicate clusters. Suppose there are K topics and V words, and the size of the topic vector cluster is n k , then the clustering cluster size can be expressed as s k = n k / v. The clustering size is set to be uniformly distributed, that is, the size of each topic vector cluster is 1 / K. The goal of PDR is to learn a sparse soft assignment relationship between topics and word vectors, which exactly conforms to the optimal transport problem: generating a sparse and optimal solution. This embodiment uses the optimal transport problem to measure and optimize the clustering matching relationship between word vectors and topic vectors to achieve the goal of PDR, that is, to learn a sparse soft assignment relationship between topics and word vectors. This embodiment first defines the discrete distributions of topic t k and word vector W i where φ is the value of the Dirac function at point x, and gives the expression of the entropy - regularized optimal transport problem:

[0231]

[0232] Among them, the latter formula can be expressed as:

[0233]

[0234] The first term π jk ||W j -t k || 2 is the expression of the original optimal transport, where ||W j -t k || 2 is the Euclidean distance between the topic t k vector and the word vector, which is also the transport cost. The remainder is the entropy regularization term, where π jk is the transport weight between the topic vector and the word vector, representing the relationship between the word vector and the topic vector. By minimizing L OT (Y, φ) to solve the parameter π. At this time, it is assumed that π ε is the parameter value corresponding to the optimal solution. At this time, π ε is the allocation relationship between the word vector and the topic vector. The objective of the clustering regularization method PDR proposed in this embodiment is to minimize the total distance between the topic vector and the word vector weighted by the soft allocation parameter π ε . In this way, the average distance between the clustering center and the clustering samples can be reduced, making the clustering result compact, and the word vectors within the cluster have a high similarity to their center. The objective of PDR is as follows:

[0235]

[0236] Use the word frequency ratio of the vocabulary to replace the popularity tendency score. In order to make the PDR regularization optimization objective plug-and-play on BERTopic, a simple VAE model is used to complete this task. VAE will receive the vector after dimensionality reduction and then learn the topic and word vector representations by reconstructing the input. In this process, it will be constrained by PDR, so that the learned topics and word vectors meet the constraints.

[0237] 3.4 Topic Representation

[0238] For each cluster (topic), the c-TF-IDF method is used to evaluate the importance of each candidate topic word in the class. All the documents ClusterDocuments(c) of each cluster c are regarded as an overall document set. Calculate the term frequency (TF) and the inverse document frequency (IDF). The calculation formula of c-TF-IDF is:

[0239] The expression of the importance degree score is:

[0240]

[0241] Among them, F c is the importance score of the theme cluster c, w represents a candidate word, C represents the set of all theme clusters, |c′∈C:w∈c| represents the number of other theme clusters containing the word w, and |C| represents the number of theme clusters;

[0242] According to the calculated c-TF-IDF value, all candidate words in the theme cluster c are sorted, and the words with higher c-TF-IDF values are selected as the keywords of this theme. The keyword list TopicWords(c) of each theme cluster generated through the above steps can be expressed as:

[0243] TopicWords(c) = Top-N(c-TF-IDF(ClusterDocument(c)))

[0244] where Top-N means selecting the top N words with c-TF-IDF rankings as the final theme words, and ClusterDocument(c) represents each theme cluster.

[0245] 4. Theme Testing

[0246] The user inputs requirements through natural language. The large language model understands the intention of the input requirements and extracts restrictive conditions such as the user's purpose and theme features. The large language model calls the Web search engine through the Agent mechanism according to the extracted requirement information to obtain source data related to the requirements, including website names, links, and brief introductions. Subsequently, the system extracts relevant useful information from the source web page content, including the web page title, body content, publication time, author, etc., and stores the extracted data in the Redis database in JSON format. Next, the system generates a theme word list based on the content of the source article and calculates the similarity with the intention keywords extracted from the user's requirements. The formula is:

[0247]

[0248] where S(A,B) is the similarity value, A is the user's intention keyword vector, B is the keyword list of each theme cluster, A i and B i are the values of the corresponding vectors in the i-th dimension;

[0249] By calculating the similarity value between the theme word list and the user's intention keyword vector, it is compared with a set preset threshold (set by historical data statistics or experience). If the similarity value is lower than the preset threshold, it is considered that a new theme is discovered and the new theme keywords are stored; otherwise, if the similarity value is higher than or equal to the preset threshold, it is determined that no new theme is discovered. Figure 3 is the schematic diagram of the theme extraction process disclosed in the embodiments of the present invention.

[0250] It can be seen that the design of the present invention realizes an intelligent processing method for source discovery, which can construct web page context according to web page link relationships, and comprehensively perform automatic discovery and analysis of information sources based on link relationships and web page content; reduces labor costs and time costs; through comparative analysis and identification of characteristic values of unknown entities, new source discovery topics can be found, enhancing the comprehensiveness of information; improves information collection efficiency and accuracy: overcomes the lack of universality caused by BERTopic's ability to mine topics only in fixed dimensions and the problem of topic result homogenization caused by being affected by word-level popularity bias, and realizes automatic selection of the optimal topic clustering vector dimension and effective correction of popular words.

[0251] Embodiment III

[0252] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of an intelligent processing device for source discovery disclosed in an embodiment of the present invention. Among them, Figure 4 The described intelligent processing device for source discovery is applied to the field of knowledge discovery technology, and the embodiments of the present invention do not make limitations. As Figure 4 shown, the intelligent processing device for source discovery may include the following operations:

[0253] S301, an information acquisition module, for acquiring user demand information;

[0254] S302, a demand analysis module, for analyzing the user demand information to obtain an information resource table;

[0255] S303, an intelligence text database construction module, automatically crawling scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, and N is a positive integer;

[0256] S304, a topic extraction module, for extracting topics from the intelligence texts to obtain a keyword list for each topic cluster;

[0257] S305, a topic testing module, for processing the user's new demand information according to the keyword list of each topic cluster to obtain a topic test result.

[0258] Embodiment IV

[0259] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of another intelligent processing device for source discovery disclosed in an embodiment of the present invention. Among them, Figure 5 The described intelligent processing device for source discovery is applied to the field of knowledge discovery technology, and the embodiments of the present invention do not make limitations. AsFigure 5 As shown, the intelligent processing device for source discovery may include the following operations:

[0260] A memory 401 storing executable program code;

[0261] A processor 402 coupled to the memory 401;

[0262] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the intelligent processing method for source discovery described in Embodiment 1 and Embodiment 2.

[0263] Embodiment 5

[0264] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the steps in the intelligent processing method for source discovery described in Embodiment 1 and Embodiment 2.

[0265] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0266] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0267] Finally, it should be noted that: The intelligent processing method and device for source discovery disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention. It is only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent processing method for information source discovery, characterized in that: The method comprises: S1, obtain user demand information; S2, parsing the user demand information to obtain an information resource table; S3, automatically crawling scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, where N is a positive integer; S4, extracting topics from the intelligence text to obtain a keyword list for each topic cluster; S5, processing the new demand information of the user according to the keyword list of each topic cluster to obtain a topic test result.

2. The intelligent processing method for information source discovery according to claim 1, characterized in that: The user demand information is parsed to obtain an information resource table, including: S21, extracting keywords from the user demand information to obtain key demand information; S22, processing the key information of the requirement to obtain a dynamic prompt; S23, using the large model to process the dynamic prompt to obtain an information resource table.

3. The intelligent processing method for information source discovery according to claim 1, characterized in that: The subject extraction of the intelligence text to obtain a keyword list for each subject cluster includes: S41, performing document embedding on the intelligence text to obtain an embedding vector; S42, performing dimensionality reduction processing on the embedding vector to obtain a reduced-dimensionality embedding vector; S43, performing clustering processing on the dimension-reduced embedding vector to obtain class vectors of K topics, where K is a positive integer; S44, performing topic representation on the class vector of each topic to obtain a keyword list for each topic cluster.

4. The intelligent processing method for information source discovery according to claim 3, characterized in that: The step of embedding the intelligence text into a document to obtain an embedding vector includes: Using the BERT document embedding processing model, the intelligence text is embedded in the document to obtain an embedding vector; The BERT document embedding processing model expression is: Embedding(d)=BERT(d) Among them, d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

5. The intelligent processing method for information source discovery according to claim 3, characterized in that: The performing dimensionality reduction processing on the embedding vector to obtain the reduced dimensionality embedding vector includes: S421, processing the embedding vector to obtain 8 candidate embedding dimension vectors; S422, using a dimension selector based on an attention mechanism, the eight candidate embedding dimension vectors are processed to obtain a reduced-dimensional embedding vector.

6. The intelligent processing method for information source discovery according to claim 3, characterized in that: The class vector of each topic is represented by a topic, and a keyword list of each topic cluster is obtained, including: S441, calculating the importance of the class vector of each topic to obtain the importance score of each topic; The importance score expression is: Among them, F c is the importance score of topic cluster c, w represents a candidate word, C represents the set of all topic clusters, |c′∈C:w∈c′| represents the number of other topic clusters containing word w, and |C| represents the number of topic clusters; S442, according to the importance score of each topic, select the top N words as the topic keywords of each topic cluster to obtain a keyword list of each topic cluster.

7. The intelligent processing method for information source discovery according to claim 1, characterized in that: The method of processing the new demand information of the user according to the keyword list of each topic cluster to obtain the topic test result includes: S51, using the large language model to analyze the user's new demand information and obtain the user's intention keyword vector; S52, using a similarity calculation model, performing similarity calculation on the keyword list of each topic cluster and the user's intended keyword vector to obtain a similarity value; The similarity calculation model expression is: Among them, S(A,B) is the similarity value, A is the user's intended keyword vector, B is the keyword list of each topic cluster, A i and B i is the value of the corresponding vector in the i-th dimension; S53, compare the similarity value with a preset threshold. When the similarity value is lower than the preset threshold, the topic test result is that a new topic is found, and the new topic keywords are stored; when the similarity value is greater than the preset threshold, the topic test result is that no new topic is found.

8. An intelligent processing device for information source discovery, characterized in that: The device comprises: Information acquisition module, used to obtain user demand information; A demand analysis module, used to analyze the user demand information to obtain an information resource table; An intelligence text database construction module automatically crawls scientific and technological intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, where N is a positive integer; A topic extraction module is used to extract topics from the intelligence text to obtain a keyword list for each topic cluster; The topic test module is used to process the new demand information of the user according to the keyword list of each topic cluster to obtain the topic test result.

9. An intelligent processing device for information source discovery, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the intelligent processing method for information source discovery as described in any one of claims 1-7.

10. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the intelligent processing method for information source discovery according to any one of claims 1-7.

Images