Intelligent processing method and device for source discovery

Abstract

The application discloses a kind of intelligent processing method and device for source discovery, the method comprises: obtaining user demand information;The user demand information is parsed, and information resource table is obtained;According to the information resource table, automatically crawl scientific and technical information text, and get intelligence text database;The intelligence text database includes N intelligence texts, and N is positive integer;The theme of the intelligence text is extracted, and the keyword list of each theme cluster is obtained;According to the keyword list of each theme cluster, the new demand information of user is processed, and the theme test result is obtained.The method can construct webpage context according to webpage link relationship, and the information source is automatically discovered and analyzed by comprehensive link relationship and webpage content, which reduces the labor cost and time cost.Through comparative analysis and the identification of unknown entity characteristic value, new source theme can be found, the comprehensiveness of information is enhanced, and the information collection efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge discovery, in particular to an intelligent processing method and device for source discovery. BACKGROUND

[0002] Text mining, in essence, is a process of deep exploration and utilization of text data, aiming to mine out those implicit, previously unknown, but extremely potential value information and knowledge from text data. Traditional topic mining methods, such as LDA, often rely on fixed probability models, which are difficult to adapt to variable text data and complex semantic structures. The BERTopic model can model the topics of short text content and display the potential hierarchical structure of topics in a clustering manner. However, BERTopic can only mine topics in a fixed dimension, resulting in insufficient universality, and is affected by word-level popularity bias, leading to homogenization of topic results. SUMMARY

[0003] The technical problem to be solved by the present application is to provide an intelligent processing method and device for source discovery, which can intelligently discover and analyze information sources based on user needs. By introducing a dynamic document embedding optimizer and a popularity bias correction regularization term, the consistency, diversity and comprehensive quality of the topics are improved, effectively solving the challenge of timely identifying text topics in the prior art, achieving efficient and accurate identification of topics in massive network data, and significantly improving the effect of text topic mining.

[0004] To solve the above technical problems, the first aspect of the embodiment of the present application discloses an intelligent processing method for source discovery, which comprises:

[0005] S1, obtaining user demand information;

[0006] S2, analyzing the user demand information to obtain an information resource table;

[0007] S3, automatically crawling sci-tech intelligence text according to the information resource table to obtain an intelligence text database; the intelligence text database comprises N intelligence texts, and N is a positive integer;

[0008] S4, performing topic extraction on the intelligence text to obtain a keyword list of each topic cluster;

[0009] S5, processing new user demand information according to the keyword list of each topic cluster to obtain a topic test result.

[0010] As an optional implementation manner, in the first aspect of the embodiment of the present application, the analyzing the user demand information to obtain an information resource table comprises:

[0011] S21, keyword extraction is performed on the user demand information to obtain demand key information;

[0012] S22, the demand key information is processed to obtain a dynamic prompt;

[0013] S23, the dynamic prompt is processed by using a large model to obtain an information resource table.

[0014] As an optional implementation, in the first aspect of the embodiment of the application, the subject extraction on the intelligence text to obtain a keyword list of each subject cluster comprises:

[0015] S41, document embedding is performed on the intelligence text to obtain an embedding vector;

[0016] S42, the embedding vector is processed by dimension reduction to obtain a dimension-reduced embedding vector;

[0017] S43, the dimension-reduced embedding vector is processed by clustering to obtain a class vector of K subjects, K being a positive integer;

[0018] S44, the class vector of each subject is represented by a subject to obtain a keyword list of each subject cluster.

[0019] As an optional implementation, in the first aspect of the embodiment of the application, the document embedding on the intelligence text to obtain an embedding vector comprises:

[0020] The intelligence text is processed by using a BERT document embedding processing model to obtain an embedding vector;

[0021] The expression of the BERT document embedding processing model is:

[0022] Embedding(d)=BERT(d)

[0023] Wherein, d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

[0024] As an optional implementation, in the first aspect of the embodiment of the application, the dimension-reduced embedding vector obtained by processing the embedding vector by dimension reduction comprises:

[0025] S421, the embedding vector is processed to obtain eight candidate embedding dimension vectors;

[0026] S422, the eight candidate embedding dimension vectors are processed by using a dimension selector based on an attention mechanism to obtain a dimension-reduced embedding vector.

[0027] As an optional implementation, in the first aspect of the embodiment of the present application, the class vector of each topic is subjected to topic representation to obtain a keyword list of each topic cluster, which includes:

[0028] S441, the class vector of each topic is subjected to importance degree calculation to obtain an importance degree score of each topic;

[0029] The importance degree score expression is:

[0030]

[0031] Wherein, F c is the importance degree score of the topic cluster c, w represents a candidate word, C represents a set of all topic clusters, |c'∈C:w∈c| represents the number of other topic clusters containing the word w, and |C| represents the number of topic clusters;

[0032] S442, according to the importance degree score of each topic, the top N words are selected as the topic keywords of each topic cluster to obtain a keyword list of each topic cluster.

[0033] As an optional implementation, in the first aspect of the embodiment of the present application, the new demand information of the user is processed according to the keyword list of each topic cluster to obtain a topic test result, which includes:

[0034] S51, the new demand information of the user is parsed by using a large language model to obtain an intention keyword vector of the user;

[0035] S52, a similarity calculation model is used to calculate the similarity between the keyword list of each topic cluster and the intention keyword vector of the user to obtain a similarity value;

[0036] The similarity calculation model expression is:

[0037]

[0038] Wherein, S(A,B) is the similarity value, A is the intention keyword vector of the user, B is the keyword list of each topic cluster, A i and B i are the values of the corresponding vectors in the i-th dimension;

[0039] S53, the similarity value is compared with a preset threshold value, when the similarity value is lower than the preset threshold value, the topic test result is that a new topic is found, and the new topic keyword is stored; when the similarity value is greater than the preset threshold value, the topic test result is that no new topic is found.

[0040] The second aspect of the embodiment of the present application discloses an intelligent processing device for source discovery, and the device comprises:

[0041] An information acquisition module is configured to acquire user demand information.

[0042] A demand analysis module is configured to analyze the user demand information to obtain an information resource table.

[0043] An intelligence text database construction module is configured to automatically crawl scientific intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database comprises N intelligence texts, and N is a positive integer.

[0044] A theme extraction module is configured to extract themes from the intelligence texts to obtain a keyword list of each theme cluster.

[0045] A theme test module is configured to process new user demand information according to the keyword list of each theme cluster to obtain a theme test result.

[0046] As an optional implementation, in the second aspect of the embodiment of the present application, the analyzing the user demand information to obtain an information resource table comprises:

[0047] S21, keyword extraction is performed on the user demand information to obtain demand key information.

[0048] S22, the demand key information is processed to obtain a dynamic prompt.

[0049] S23, the dynamic prompt is processed by using a large model to obtain an information resource table.

[0050] As an optional implementation, in the first aspect of the embodiment of the present application, the extracting themes from the intelligence texts to obtain a keyword list of each theme cluster comprises:

[0051] S41, document embedding is performed on the intelligence texts to obtain embedding vectors.

[0052] S42, dimension reduction processing is performed on the embedding vectors to obtain dimension-reduced embedding vectors.

[0053] S43, clustering processing is performed on the dimension-reduced embedding vectors to obtain class vectors of K themes, and K is a positive integer.

[0054] S44, theme representation is performed on the class vectors of each theme to obtain a keyword list of each theme cluster.

[0055] As an optional implementation, in the second aspect of the embodiment of the present application, the performing document embedding on the intelligence texts to obtain embedding vectors comprises:

[0056] performing document embedding on the intelligence text by using a BERT document embedding processing model to obtain an embedding vector;

[0057] The BERT document embedding processing model expression is:

[0058] Embedding(d)=BERT(d)

[0059] wherein d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

[0060] As an optional implementation, in the second aspect of the embodiment of the present application, the dimension reduction embedding vector is obtained by performing dimension reduction processing on the embedding vector, including:

[0061] S421, performing processing on the embedding vector to obtain eight candidate embedding dimension vectors;

[0062] S422, performing processing on the eight candidate embedding dimension vectors by using a dimension selector based on an attention mechanism to obtain a dimension reduction embedding vector.

[0063] As an optional implementation, in the second aspect of the embodiment of the present application, the keyword list of each topic cluster is obtained by performing topic representation on the class vector of each topic, including:

[0064] S441, performing importance degree calculation on the class vector of each topic to obtain an importance degree score of each topic;

[0065] The importance degree score expression is:

[0066]

[0067] wherein F c is the importance degree score of the topic cluster c, w represents a candidate word, C represents a set of all topic clusters, |c'∈C:w∈c| represents the number of other topic clusters containing the word w, and |C| represents the number of topic clusters.

[0068] S442, selecting the top N words as the topic keywords of each topic cluster according to the importance degree score of each topic to obtain a keyword list of each topic cluster.

[0069] As an optional implementation, in the second aspect of the embodiment of the present application, the topic test result is obtained by performing processing on the new demand information of the user according to the keyword list of each topic cluster, including:

[0070] S51, the new demand information of the user is parsed by using a large language model to obtain an intention keyword vector of the user;

[0071] S52, similarity calculation is performed on the keyword list of each topic cluster and the intention keyword vector of the user by using a similarity calculation model to obtain a similarity value;

[0072] The similarity calculation model expression is:

[0073]

[0074] Wherein, S(A, B) is the similarity value, A is the intention keyword vector of the user, B is the keyword list of each topic cluster, A i and B i are the values of the corresponding vectors in the ith dimension;

[0075] S53, the similarity value is compared with a preset threshold value, when the similarity value is lower than the preset threshold value, the topic test result is that a new topic is found, and the new topic keyword is stored; when the similarity value is greater than the preset threshold value, the topic test result is that no new topic is found.

[0076] The third aspect of the application discloses another intelligent processing device for source discovery, the device comprises:

[0077] A memory storing executable program codes;

[0078] A processor coupled with the memory;

[0079] The processor calls the executable program codes stored in the memory to execute part or all steps of the intelligent processing method for source discovery disclosed in the first aspect of the application.

[0080] The fourth aspect of the application discloses a computer storage medium, the computer storage medium stores computer instructions, when the computer instructions are called, part or all steps of the intelligent processing method for source discovery disclosed in the first aspect of the application are executed.

[0081] Compared with the prior art, the embodiments of the application have the following beneficial effects:

[0082] The application discloses an intelligent processing method for information source discovery. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to make the technical solution in the embodiments of the application clearer, the following briefly introduces the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained based on the drawings without creative labor.

[0084] Figure 1 FIG. 1 is a flowchart of an intelligent processing method for information source discovery disclosed by an embodiment of the application;

[0085] Figure 2 FIG. 2 is a flowchart of another intelligent processing method for information source discovery disclosed by an embodiment of the application;

[0086] Figure 3 FIG. 3 is a subject extraction flowchart disclosed by an embodiment of the application;

[0087] Figure 4 FIG. 4 is a structural diagram of an intelligent processing device for information source discovery disclosed by an embodiment of the application;

[0088] Figure 5 FIG. 5 is a structural diagram of another intelligent processing device for information source discovery disclosed by an embodiment of the application. DETAILED DESCRIPTION

[0089] In order to make the technical solution in the embodiments of the application clearer, the following briefly introduces the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained based on the drawings without creative labor.

[0090] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used to distinguish different objects, and are not used to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or equipment including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product, or equipment.

[0091] Reference herein to "embodiments" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0092] The application discloses an intelligent processing method and device for source discovery, which comprises the following steps: obtaining user demand information; analyzing the user demand information to obtain an information resource table; automatically crawling sci-tech intelligence text according to the information resource table to obtain an intelligence text database; the intelligence text database comprises N intelligence texts, and N is a positive integer; subject extraction is performed on the intelligence texts to obtain a keyword list of each subject cluster; and new user demand information is processed according to the keyword list of each subject cluster to obtain a subject test result. The method can construct a webpage context according to a webpage link relationship, comprehensively analyze information source automatic discovery according to the link relationship and the webpage content, and reduce labor cost and time cost. Through comparative analysis and identification of unknown entity characteristic values, new source subjects can be discovered, the comprehensiveness of information is enhanced, and information collection efficiency and accuracy are improved. The following will be described in detail.

[0093] Embodiment one

[0094] Please refer to Figure 1 , Figure 1 is a flowchart of an intelligent processing method for source discovery disclosed by the embodiments of the application. Wherein, Figure 1 The intelligent processing method for source discovery described herein is applied to the technical field of knowledge discovery, and the embodiments of the application are not limited. As shown in Figure 1 The intelligent processing method for source discovery can comprise the following operations:

[0095] S1, obtaining user demand information;

[0096] S2, analyzing the user demand information to obtain an information resource table;

[0097] S3, automatically crawling the scientific information text according to the information resource table to obtain an information text database; the information text database includes N information texts, and N is a positive integer;

[0098] According to the content of the information resource table, related useful information in a webpage is extracted by using an automatic crawling tool, such as a crawler tool, including a webpage title, a text content, a publishing time, an author and the like, and the extracted data is stored in a Redis information text database in a JSON format;

[0099] S4, subject extraction is performed on the information text to obtain a keyword list of each subject cluster;

[0100] S5, according to the keyword list of each subject cluster, new demand information of a user is processed to obtain a subject test result.

[0101] Optionally, the user demand information is parsed to obtain an information resource table, including:

[0102] S21, keyword extraction is performed on the user demand information to obtain demand key information;

[0103] The keyword extraction is performed by using the following method:

[0104] The user demand information is subjected to lattice network construction, which is prior art in the field. A node set of the lattice network construction is N={n0, n1, …}, an arc set is E={e0, e1, …}, each node n i ∈N contains a time mark t(n i ), and an arc e u→v represents an arc of a node u connected to a node v. It is assumed that nodes in a keyword network are NS={N0, N1, …}, represent a set of arcs of a node N i connected to a node N j in the keyword network.

[0105] 1. For any n i ∈N i , n j ∈N i , i≤j can be obtained from t(n i )<t(n j ).

[0106] 2. For any n i ∈N i , n j ∈N i , i=j can be obtained from t(n i )=t(n j ).

[0107] 3. Any e u→v ∈E, if for u∈N i and v∈N j e u→v Equivalent to a group Then i ≤ m ≤ n ≤ j. (Here, n = m + 1, e u→v The two endpoints belong to N respectively m and N n 。 )

[0108] The algorithm for transforming the word lattice network into a keyword network is as follows:

[0109] 1. Convert keywords into corresponding syllable strings: K1…K M (M is the number of keyword syllables). Traverse all nodes N in the Lattice network and find the node n that matches K1. k , set as

[0110] 2. with Starting from the endpoint, search backwards in reverse chronological order for the nearest neighbor node n. k-1 If n k-1 and If there is no arc between them, then n k-1 Also belongs to Continue searching n k-1 The previous node. Otherwise, the search stops.

[0111] 3. With Starting from the nearest node n, search forward in chronological order for the nearest neighbor n. k+1 If n k+1 and If there is no arc between them, then n k+1 belong Continue searching n k+1 The next node. Otherwise, the search stops.

[0112] 4. For e k→k+1 belong n k+1 belong

[0113] 5. All included arc e k→k+1 This creates a keyword obfuscation network.

[0114] 6. For each arc If u belongs to N s n i Belongs to N t When t = s + 1, belong otherwise, belong (s+1≤n≤t).

[0115]

[0116] In the formula, w(l) and w(e) are the words corresponding to arcs l and e, and sim(·,·) refers to the similarity between the two words. It refers to the arc The normalization time overlaps with that of e. Additionally, each node n in Lattice... i The time stamp t(n) contained in ∈N i This will be used as a constraint to determine when the search terminates.

[0117] Keywords were identified using data normalization. First, the linguistic probability likelihood score of each candidate was normalized:

[0118]

[0119] Where Value is the set of linguistic scores in the keyword network; minValue and maxValue are the minimum and maximum values ​​in the set, respectively; x is the score before normalization; and y is the score after normalization. Then, the sum of the scores is calculated and marked as the final score of the arc in the keyword network.

[0120] In each keyword network, select the candidate arc with the highest score and determine if it is a keyword. If it is, output the keyword and time stamp; otherwise, skip it and proceed to the next keyword network. This completes the keyword detection process.

[0121] S22, Process the key information of the requirements to obtain a dynamic Prompt;

[0122] S23, The dynamic Prompt is processed using a large model to obtain an information resource table.

[0123] Specifically, the process involves using a large model to call a web search engine via an agent, sending a request to the large model using the generated Prompt, obtaining information source data relevant to the requirements, and returning an information resource table in JSON format, including website names, corresponding links, and preliminary introductions.

[0124] Large models are first pre-trained on massive datasets using self-supervised or semi-supervised learning, and then their performance and capabilities are further optimized through methods such as instruction fine-tuning and human alignment. Large models are characterized by a large number of parameters, large training data, and large computational resources, and possess the ability to solve general tasks, follow human instructions, and perform complex reasoning.

[0125] Optionally, the step of extracting topics from the intelligence text to obtain a keyword list for each topic cluster includes:

[0126] S41, perform document embedding on the intelligence text to obtain an embedding vector;

[0127] S42, perform dimensionality reduction processing on the embedding vector to obtain a dimensionality-reduced embedding vector;

[0128] S43, perform clustering processing on the dimensionality-reduced embedding vector to obtain K class vectors of topics, where K is a positive integer;

[0129] Optional clustering methods include:

[0130] (1) Randomly select initial data points X = {x1, x2, ..., x...} n Select a point as the cluster center in the data, and set a threshold, where n is the number of data points;

[0131] (2) Calculate all points within the threshold distance range of the cluster center, record them in set M, and classify them into cluster c. The distance measurement formula is:

[0132]

[0133] Z(c k ) is cluster c k The distance value, x i For the i-th data point, c j For the j-th cluster;

[0134] (3) Calculate the distance vector between each element in set M and cluster c and sum them to obtain the offset vector M. h (x):

[0135]

[0136] (4) The cluster center moves along the direction of the offset vector by a distance equal to the magnitude of the offset vector;

[0137] (5) Repeat steps (2), (3), and (4) until the offset vector value reaches the threshold set by the termination condition, and record the cluster center at this time.

[0138] (6) Repeat steps (1) to (5) until all data points have been traversed;

[0139] (7) Finally, calculate the number of times each element is accessed by each class, and classify the element into the cluster class that is accessed the most.

[0140] S44. Represent the class vector of each topic as a topic to obtain a list of keywords for each topic cluster.

[0141] Optionally, the step of document embedding the intelligence text to obtain an embedding vector includes:

[0142] The BERT document embedding processing model is used to perform document embedding on the intelligence text to obtain an embedding vector;

[0143] The expression for the BERT document embedding processing model is:

[0144] Embedding(d) = BERT(d)

[0145] Where d represents the intelligence text, Embedding(d) represents the embedding vector, and BERT represents the BERT document embedding processing model.

[0146] Optionally, the step of reducing the dimensionality of the embedding vector to obtain a dimensionality-reduced embedding vector includes:

[0147] S421, The embedding vector is processed to obtain 8 candidate embedding dimension vectors;

[0148] The specific method is as follows:

[0149] For the embedding vector x = {x1, x2, ..., x...} n After processing, the connection structure diagram is obtained:

[0150] N(x i )={x j |s=1,2,…,n,s≠j,d(x i ,x j )≤d(x s ,x j )}

[0151] n is the dimension of the vector, N(x) i ) is x i The corresponding connection structure diagram, d(x) i ,x j ) represents x i ,x j The Euclidean distance between them; i = 1, 2, ..., n, j = 1, 2, ..., n;

[0152] Calculate x at two points i ,x j The connection probability p between ij :

[0153]

[0154] σ i Let x be the point i The corresponding scale parameters are preset parameters;

[0155] Assume that the low-dimensional vector is Y = {y1, y2, …, y m}, m is the dimension;

[0156] The similarity coefficient q is constructed ij as follows:

[0157] q ij = (1 + a||y i -y j || 2b ) -1

[0158] where a and b are hyperparameters;

[0159] The objective function is constructed as follows:

[0160]

[0161] where η is the learning rate, controlling the step size of each update.

[0162] By randomly initializing Y, the iteration is constantly updated to minimize the objective function, at which time the structure in the high-dimensional space is preserved in the low-dimensional space, and the final dimension reduction result Y is obtained.

[0163] The dimension reduction result Y is processed to generate vectors of 8 different dimensions from 32, 64 to 1024 as 8 candidate embedding dimension vectors.

[0164] S422, using a dimension selector based on an attention mechanism, processes the 8 candidate embedding dimension vectors to obtain a dimension reduction embedding vector.

[0165] Specifically, the best candidate dimension is determined by comparing the classification accuracy of the candidate dimension on the publishing agencies and types of policy texts.

[0166] Information is extracted through a self-attention mechanism, and a fully connected network is connected after the self-attention layer as an output layer, outputting the probability of the corresponding category. The best candidate dimension is selected by evaluating the accuracy of different candidate dimensions on the classification task.

[0167]

[0168] out = softmax (MLP (x hid ))

[0169] wherein q, k, v are the 8 candidate embedding dimension vectors corresponding to each document, Q, K, V are the embedding dimension vectors corresponding to the entire text group; Attention is the attention mechanism, softmax represents the Softmax function, T represents transposition, out represents the output vector, i.e. the dimension reduction embedding vector, d k = q 2 + k 2 + v 2 , and MLP represents a multi-layer perception machine.

[0170] Optionally, the class vector of each topic is subjected to topic representation to obtain a keyword list of each topic cluster, including:

[0171] S441, the class vector of each topic is subjected to importance degree calculation to obtain an importance degree score of each topic;

[0172] The importance degree score expression is:

[0173]

[0174] wherein F c is the importance degree score of the topic cluster c, w represents a candidate word, C represents a set of all topic clusters, |c'∈C:w∈c| represents the number of other topic clusters containing the word w, and |C| represents the number of topic clusters;

[0175] S442, according to the importance degree score of each topic, the top N words are selected as the topic keywords of each topic cluster to obtain a keyword list of each topic cluster.

[0176] Optionally, the new demand information of the user is processed according to the keyword list of each topic cluster to obtain a topic test result, including:

[0177] S51, the new demand information of the user is parsed by using a large language model to obtain an intention keyword vector of the user;

[0178] The large language model is a kind of large model.

[0179] S52, the keyword list of each topic cluster and the intention keyword vector of the user are subjected to similarity calculation by using a similarity calculation model to obtain a similarity value;

[0180] The similarity calculation model expression is:

[0181]

[0182] wherein S(A, B) is the similarity value, A is the intention keyword vector of the user, B is the keyword list of each topic cluster, and Ai and B i is the value of the corresponding vector in the i-th dimension;

[0183] S53, compare the similarity value with a preset threshold value, when the similarity value is lower than the preset threshold value, the theme test result is that a new theme is found, and the new theme keyword is stored; when the similarity value is greater than the preset threshold value, the theme test result is that no new theme is found.

[0184] It can be seen that the application discloses an intelligent processing method for source discovery, which can construct a webpage context according to webpage link relationships, comprehensively analyze information sources through link relationships and webpage content, and automatically discover information sources. The method reduces labor cost and time cost, can discover new source themes through comparative analysis and unknown entity characteristic value identification, enhances information comprehensiveness, improves information collection efficiency and accuracy, overcomes the problem of insufficient universality caused by BERTopic topic mining in fixed dimensions and the problem of theme result homogenization caused by popularity bias of word level, and realizes automatic selection of optimal theme clustering vector dimensions and effective correction of popular words.

[0185] Embodiment two

[0186] Please refer to Figure 2 , Figure 2 is a flowchart of another intelligent processing method for source discovery disclosed by the embodiment of the application. Wherein, Figure 2 The intelligent processing method for source discovery described above is applied to the technical field of knowledge discovery, and the embodiment of the application is not limited. As Figure 2 shown, the intelligent processing method for source discovery can include the following operations:

[0187] 1. Analyze the user input, extract the key information in the demand, then generate a dynamic Prompt according to the extracted key information, and then call a Web search engine in the form of an Agent through a large model, send a request to the large model using the generated Prompt, obtain information source data related to the demand, and return the website name, corresponding link and preliminary introduction in JSON format.

[0188] 2. Obtain source information: obtain content according to related source links, including webpage title, webpage content, publication time, author, link, and store the data into a redis database.

[0189] 3. Theme extraction

[0190] The BertTopic model is used for topic extraction. BertTopic is a powerful tool that combines the pre-trained BERT model with topic modeling. Its core idea is to capture the semantic information of documents using the BERT model and then use topic modeling techniques to cluster this semantic information to derive topics. BertTopic utilizes BERT embeddings and the c-TF-IDF method (an improved TF-IDF representation that more accurately represents word frequencies in clusters) to create clusters, allowing for the generation of easily interpretable topics while preserving important vocabulary in topic descriptions. The main steps of the BertTopic algorithm are as follows:

[0191] 3.1 Document Embedding

[0192] The BERT document embedding processor takes document text from the database and feeds it into a pre-trained BERT model to generate document embedding vectors. This process can be represented as:

[0193] The expression for the BERT document embedding processing model is:

[0194] Embedding(d) = BERT(d)

[0195] Where d represents the intelligence text, Embedding(d) represents the embedding vector, and BERT represents the BERT document embedding processing model.

[0196] 3.2 Dimensionality Reduction

[0197] The embedding vectors are reduced to eight different dimensions, ranging from 32, 64 to 1024. In this process, the Document Embedding Dimension Optimizer (DDEO) is proposed to enhance dimensionality reduction. Dimension selection is performed automatically, and this embodiment employs two methods for dimension selection: downstream tasks and word vector evaluation.

[0198] The specific method for dimensionality reduction is as follows:

[0199] For the embedding vector x = {x1, x2, ..., x...} n After processing, the connection structure diagram is obtained:

[0200] N(x i )={x j |s=1,2,…,n,s≠j,d(x i ,x j )≤d(x s ,x j )}

[0201] n is the dimension of the vector, N(x) i ) is x i The corresponding connection structure diagram, d(x)i x j ) represents the Euclidean distance between x i x j ; i = 1, 2, …, n, j = 1, 2, …, n;

[0202] The connection probability p i x j between two points x ij :

[0203]

[0204] σ i is the corresponding scale parameter of the point x i , which is a pre-set parameter;

[0205] Assume that the low-dimensional vector is Y = {y1, y2, …, y m}, and m is the dimension;

[0206] The similarity coefficient q ij is constructed as follows:

[0207] q ij = (1 + a||y i -y j || 2b ) -1

[0208] where a and b are hyperparameters;

[0209] The objective function is constructed as follows:

[0210]

[0211] where η is the learning rate, which controls the step size of each update.

[0212] By randomly initializing Y, the iteration is constantly updated to minimize the objective function, so that the structure in the high-dimensional space is preserved in the low-dimensional space, and the final dimension reduction result Y is obtained.

[0213] 1) The dimension selector based on the downstream task is determined by inputting the embedding vector after dimension reduction into the classification task, and comparing the classification accuracy of the candidate dimension on the publishing agencies and types of policy texts to determine the best candidate dimension. The dimension selector based on the downstream task is divided into two types: the K-Means-based dimension selector (KDS) and the attention mechanism-based dimension selector (ADS).

[0214] a. K-Means-based dimension selector (KDS):

[0215] K-Means-based Dimension Selector (KDS) first reduces the document vectors to different candidate embedding dimensions; then clusters all the low-dimensional vectors of documents in each candidate dimension, and calculates the accuracy of each candidate dimension respectively. Finally, the candidate dimension with the highest accuracy under each dataset is the optimal embedding dimension OND.

[0216] b. Attention-based Dimension Selector (ADS):

[0217] The reduced dimension result Y is processed by the embedding method to obtain the document embedding x;

[0218] q, k, v = linear qkv (x)

[0219]

[0220] out = softmax (MLP (x hid ))

[0221] x is processed to obtain q, k, and v, where q, k, and v are 8 candidate embedding dimension vectors corresponding to each document, Q, K, and V are embedding dimension vectors corresponding to all text groups; Attention is an attention mechanism, softmax represents a Softmax function, T represents transposition, out represents an output vector, i.e., a reduced embedding vector, d k = q 2 +k 2 +v 2 , and MLP represents a multi-layer perceptron.

[0222] Based on the trained ADS model, the classification accuracy of different candidate dimensions on the test set is calculated, and the candidate dimension with high classification accuracy is output as the optimal embedding dimension.

[0223] 2) Word Vector Evaluation-based Dimension Selector (WDS):

[0224] This embodiment examines the dimension selection problem from the perspective of word vectors and designs an evaluation method based on analogy reasoning and document vector contrast. The analogy reasoning method refers to mining the linear relationship of the text itself in vector evaluation. The text is classified and the embedding vector is obtained. Then the similarity ratio of the document embedding with analogy relationship is calculated.

[0225] The evaluation method of document vector contrast in this embodiment is:

[0226]

[0227] The embedding quality is evaluated by comparing the similarity of the document embeddings across the publishing agencies: for documents belonging to the same publishing agency, the vector similarity ES s The higher the better, for documents belonging to different publishing agencies, the vector similarity ES d The lower the better. NESD is the normalized vector similarity difference. The larger the vector similarity difference represents the better embedding quality, and the embedding vectors with a vector similarity difference less than a preset threshold are discarded. Then, the dimension reduction processing of 3.2 is performed.

[0228] Further, in order to improve the simplification index, the two indexes of ARS and NESD are fused into ARS-NESD, ARS is the cosine similarity, the value range is between 0 and 1, and the larger the number represents that the candidate dimension is better in the evaluation of the word vector.

[0229] 3.3 Clustering analysis

[0230] This embodiment introduces a PDR (Popularity Debias Regularization) module to optimize the clustering matching relationship between the word vectors and the topic vectors through a probabilistic clustering module with regularization, and improve the clustering quality. The idea of the probabilistic clustering regularization module is: regarding the topic vectors as clustering centers and the word vectors as clustering samples; through the constraint condition, the topic vectors can be distributed more uniformly and dispersedly in the semantic space. Specifically, the two core constraints of PDR are: 1) the distance between the topic vectors is as large as possible; and 2) each word vector belongs to only one topic vector as possible. In order to achieve the above-mentioned goal, first, the constraint of the size of the word vector clustering cluster corresponding to each clustering center (topic vector) is defined, in order to avoid the occurrence of empty clustering or repeated cluster. Assuming that there are K topics and V words, the size of the topic vector cluster is n k , then the size of the clustering cluster can be expressed as s k = n k / v. The clustering size is set to be uniformly distributed, that is, the size of each topic vector cluster is 1 / K. The goal of PDR is to learn a sparse soft assignment relationship between the topic and the word vector, which is exactly consistent with the optimal transport problem: to generate a sparse and optimal solution. This embodiment adopts the optimal transport problem to measure and optimize the clustering matching relationship between the word vector and the topic vector to achieve the goal of PDR, that is, to learn a sparse soft assignment relationship between the topic and the word vector. This embodiment first defines the discrete distribution of the topic t k and the word vector W i where φ is the value of the Dirac function at x, and the expression of the entropy-regularized optimal transport problem is given as:

[0231] ​

[0232] where the latter can be expressed as:

[0233]

[0234] The first term π jk ||W j -t k || 2 is the expression of the original optimal transport, where ||W j -t k || 2 is the topic t k The Euclidean distance between the topic vector and the word vector is also the transport cost. The remainder is the entropy regularization term, where π jk is the transport weight between the topic vector and the word vector, which represents the relationship between the word vector and the topic vector. By minimizing L OT (Y, φ) to solve the parameter π, at this time, it is assumed that π ε is the parameter value corresponding to the optimal solution, at this time, π ε is the allocation relationship between the word vector and the topic vector. The goal of the clustering regularization method PDR proposed in this embodiment is to minimize the total distance between the weighted topic vector and the word vector under the soft assignment parameter π ε . In this way, the average distance between the cluster center and the cluster sample can be reduced, so that the clustering result is compact, and the word vector in the cluster has a high similarity to its center. The goal of PDR is as follows:

[0235]

[0236] The word frequency proportion of the vocabulary is used instead of the popularity tendency score. In order to enable the PDR regularization optimization goal to be plug and play on BERTopic, a simple VAE model is used to complete this task. The VAE will receive the reduced dimension vector and then learn the topic and word vector representation by reconstructing the input. In this process, it will be constrained by PDR, so that the learned topic and word vector meet the constraints.

[0237] 3.4 Topic representation

[0238] For each cluster (topic), the importance of each candidate topic word in the cluster is evaluated using the c-TF-IDF method. All documents of each cluster c ClusterDocuments(c) are regarded as a whole document set. The term frequency (TF) and inverse document frequency (IDF) are calculated, and the calculation formula of c-TF-IDF is as follows:

[0239] The importance degree score expression is as follows:

[0240]

[0241] where F c is the importance degree score of the topic cluster c, w represents a candidate word, C represents the set of all topic clusters, |c'∈C:w∈c| represents the number of other topic clusters containing the word w, and |C| represents the number of topic clusters;

[0242] According to the calculated c-TF-IDF value, all candidate words in the topic cluster c are sorted, and words with higher c-TF-IDF values are selected as keywords of the topic. The keyword list TopicWords(c) of each topic cluster generated through the above steps can be represented as:

[0243] TopicWords(c)=Top-N(c-TF-IDF(ClusterDocument(c)))

[0244] where Top-N represents selecting the top N words in the c-TF-IDF ranking as the final topic words, and ClusterDocument(c) represents each topic cluster.

[0245] 4. Topic test

[0246] The user inputs the demand through natural language, and the large language model understands the intention of the input demand, extracts the purpose, theme features and other limiting conditions of the user. The large language model calls the Web search engine according to the extracted demand information in the Agent mechanism, obtains the source data related to the demand, including website name, link and brief introduction. Subsequently, the system extracts the relevant useful information in the web page content, including web page title, text content, publishing time and author, etc., and stores the extracted data in JSON format to the Redis database. Next, the system generates a topic word list based on the article content of the source, and calculates the similarity with the intention keywords extracted in the user demand, the formula is:

[0247]

[0248] where S(A,B) is the similarity value, A is the intention keyword vector of the user, B is the keyword list of each topic cluster, A i and B i are the values of the corresponding vectors in the i-th dimension;

[0249] By calculating the similarity value of the topic word list and the user intention keyword vector, and comparing it with the preset threshold value (set by historical data statistics or experience), if the similarity value is lower than the preset threshold value, it is considered that a new topic is found, and the new topic keywords are stored; otherwise, if the similarity value is higher than or equal to the preset threshold value, it is determined that no new topic is found. Figure 3 is the schematic diagram of the topic extraction process disclosed in the embodiments of the present application.

[0250] It can be seen that the application designs an intelligent processing method for source discovery, can construct webpage context according to webpage link relationship, and comprehensively analyzes link relationship and webpage content for automatic discovery and analysis of information sources; reduces labor cost and time cost; through comparative analysis and identification of unknown entity characteristic values, new source topics can be discovered, and the comprehensiveness of information is enhanced; improves information collection efficiency and accuracy; overcomes the problem of insufficient universality caused by BERTopic only mining topics in fixed dimensions and the problem of topic result homogenization caused by the influence of word-level popularity bias, realizes automatic selection of optimal topic clustering vector dimension and effective correction of popular words.

[0251] Embodiment three

[0252] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an intelligent processing device for source discovery disclosed by the embodiment of the application. Wherein, Figure 4 The intelligent processing device for source discovery described is applied to the technical field of knowledge discovery, and the embodiment of the application is not limited. As shown in Figure 4 , the intelligent processing device for source discovery can include the following operations:

[0253] S301, an information acquisition module, configured to acquire user demand information;

[0254] S302, a demand analysis module, configured to analyze the user demand information to obtain an information resource table;

[0255] S303, an intelligence text database construction module, configured to automatically crawl scientific intelligence text according to the information resource table to obtain an intelligence text database; the intelligence text database includes N intelligence texts, and N is a positive integer;

[0256] S304, a topic extraction module, configured to extract topics from the intelligence texts to obtain a keyword list of each topic cluster;

[0257] S305, a topic test module, configured to process new user demand information according to the keyword list of each topic cluster to obtain a topic test result.

[0258] Embodiment four

[0259] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of another intelligent processing device for source discovery disclosed by the embodiment of the application. Wherein, Figure 5 The intelligent processing device for source discovery described is applied to the technical field of knowledge discovery, and the embodiment of the application is not limited. As shown inFigure 5 As shown, the intelligent processing device for source discovery can include the following operations:

[0260] a memory 401 storing executable program codes;

[0261] a processor 402 coupled with the memory 401;

[0262] The processor 402 invokes the executable program codes stored in the memory 401 for executing the steps in the intelligent processing method for source discovery described in Embodiment I and Embodiment II.

[0263] Embodiment V

[0264] The embodiments of the present application disclose a computer readable storage medium storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the steps in the intelligent processing method for source discovery described in Embodiment I and Embodiment II.

[0265] The above-described device embodiments are only schematic, and the modules illustrated as separate components can or can not be physically separate, and the components illustrated as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0266] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the above specific description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other computer readable medium that can be used to carry or store data.

[0267] Finally, it should be noted that: the intelligent processing method and device for source discovery disclosed by the embodiments of the present application are only the preferred embodiments of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for intelligent processing of source discovery, characterized in that, The method comprises: S1, obtaining user demand information; S2, analyzing the user demand information to obtain an information resource table; S3, automatically crawling a science and technology information text according to the information resource table to obtain an information text database; the information text database comprises N information texts, and N is a positive integer; S4, performing theme extraction on the information text to obtain a keyword list of each theme cluster, comprising: S41, performing document embedding on the information text to obtain an embedding vector; S42, performing dimension reduction processing on the embedding vector to obtain a dimension-reduced embedding vector, comprising: S421, processing the embedding vector to obtain eight candidate embedding dimension vectors; S422, using a dimension selector based on an attention mechanism to process the eight candidate embedding dimension vectors to obtain a dimension-reduced embedding vector; Specifically, the best candidate dimension is determined by comparing the classification accuracy of the candidate dimension in the policy text in terms of the publishing agency and type; Information is extracted through a self-attention mechanism, and a fully connected network is connected after the self-attention layer as an output layer to output the probability of the corresponding category, and the best candidate dimension is selected by evaluating the accuracy of different candidate dimensions in the classification task: out = softmax(MLP(x hid )) where q, k, v are the 8 candidate embedding dimension vectors corresponding to each document, Q, K, V are the embedding dimension vectors corresponding to the entire text composition; Attention is the attention mechanism, softmax represents the Softmax function, T represents transposition, out represents the output vector, i.e., the dimension-reduced embedding vector, d k = q 2 + k 2 + v 2 , and MLP represents a multi-layer perceptron; S43, performing clustering processing on the dimension-reduced embedding vector to obtain a class vector of K themes, and K is a positive integer; The clustering method is: (1) Randomly select a point as the cluster center in the initial data points X = {x1, x2, …, xn} and set a threshold value, n is the number of data; n} (2) Calculate all points within the threshold distance range of the cluster center, record them in set M, and classify them as cluster c. The distance measurement formula is: Z(c k ) is the distance value of cluster c k , x i is the i-th data point, c j is the j-th cluster; (3) Calculate the distance vector between each element in the set M and the cluster c and accumulate it to obtain the offset vector M h (x): (4) The cluster center moves along the direction of the offset vector by a distance length equivalent to the modulus of the offset vector; (5) Repeat steps (2), (3), and (4) until the offset vector value reaches the threshold value set by the termination condition, and record the cluster center at this time; (6) Repeat steps (1) to (5) until all data points are traversed; (7) Finally, calculate the access frequency of each element to each class, and classify the element into the cluster class with the highest access frequency; S44, performing theme representation on the class vector of each theme to obtain a keyword list of each theme cluster, comprising: S441, calculating the importance degree of the class vector of each theme to obtain an importance degree score of each theme; The importance degree score expression is: where F c is the importance score of topic cluster c, w represents a candidate word, C represents the set of all topic clusters, |c'∈C:w∈c'| represents the number of other topic clusters containing word w, and |C| represents the number of topic clusters. S442, selecting the top N words as the theme keywords of each theme cluster according to the importance degree score of each theme to obtain a keyword list of each theme cluster; S5, processing new user demand information according to the keyword list of each theme cluster to obtain a theme test result, comprising: S51, using a large language model to analyze the new user demand information to obtain an intent keyword vector of the user; S52, using a similarity calculation model to calculate the similarity between the keyword list of each theme cluster and the intent keyword vector of the user to obtain a similarity value; The similarity calculation model expression is: where S(A, B) is a similarity value, A is a user's intent keyword vector, B is a keyword list of each topic cluster, A i and B i are values of corresponding vectors in the i-th dimension; S53, compare the similarity value with a preset threshold value, when the similarity value is lower than the preset threshold value, the theme test result is that a new theme is found, and the new theme keyword is stored; when the similarity value is greater than the preset threshold value, the theme test result is that no new theme is found.

2. The intelligent processing method for source discovery according to claim 1, wherein, The information resource table is obtained by analyzing the user demand information, and the information resource table comprises: S21, keyword extraction is performed on the user demand information to obtain demand key information; S22, the demand key information is processed to obtain a dynamic prompt; S23, the dynamic prompt is processed by using a large model to obtain an information resource table.

3. The intelligent processing method for source discovery according to claim 1, wherein, The document embedding of the intelligence text is performed to obtain an embedding vector, and the document embedding of the intelligence text comprises: The document embedding of the intelligence text is performed by using a BERT document embedding processing model to obtain an embedding vector; The BERT document embedding processing model expression is: Embedding(d)=BERT(d) wherein, d is the intelligence text, Embedding(d) is the embedding vector, and BERT represents the BERT document embedding processing model.

4. An intelligent processing device for source discovery, characterized by The device comprises: an information acquisition module configured to acquire user demand information; a demand analysis module configured to analyze the user demand information to obtain an information resource table; an intelligence text database construction module configured to automatically crawl scientific intelligence texts according to the information resource table to obtain an intelligence text database; the intelligence text database comprises N intelligence texts, and N is a positive integer; a theme extraction module configured to extract themes from the intelligence texts to obtain a keyword list of each theme cluster, and the keyword list comprises: S41, document embedding of the intelligence text is performed to obtain an embedding vector; S42, dimension reduction processing is performed on the embedding vector to obtain a dimension reduction embedding vector, and the dimension reduction processing comprises: S421, the embedding vector is processed to obtain eight candidate embedding dimension vectors; S422, a dimension selector based on an attention mechanism is used to process the eight candidate embedding dimension vectors to obtain a dimension reduction embedding vector; Specifically, the best candidate dimension is determined by comparing the classification accuracy of the candidate dimensions in the publishing agencies and types of the policy texts; The information is extracted through the self-attention mechanism, and a fully connected network is connected after the self-attention layer as an output layer to output the probability of the corresponding category, and the best candidate dimension is selected by evaluating the accuracy of different candidate dimensions in the classification task: out = softmax(MLP(x hid )) where q, k, v are the 8 candidate embedding dimension vectors corresponding to each document, Q, K, V are the embedding dimension vectors corresponding to the entire text composition; Attention is the attention mechanism, softmax represents the Softmax function, T represents transposition, out represents the output vector, i.e., the dimension-reduced embedding vector, d k = q 2 + k 2 + v 2 , and MLP represents a multi-layer perceptron; S43, the dimension reduction embedding vector is clustered to obtain a class vector of K themes, and K is a positive integer; The clustering method is: (1) Randomly select a point as the cluster center in the initial data points X = {x1, x2, …, x n} and set a threshold, n is the number of data; (2) all points within a threshold distance range of the cluster center are recorded in a set M and classified into a cluster c, and the distance measurement formula is: Z(c k ) is the distance value of cluster c k , x i is the i-th data point, c j is the j-th cluster; (3) Calculate the distance vector between each element in the set M and the cluster c and accumulate it to get the offset vector M h (x): (4) the cluster center is moved along the direction of the offset vector by a distance length equivalent to the modulus of the offset vector; (5) steps (2), (3) and (4) are repeated until the offset vector value reaches a threshold value set as a termination condition, and the cluster center at this time is recorded; (6) steps (1) to (5) are repeated until all data points are traversed; (7) Finally, the number of accesses of each element by each class is calculated, and the element is classified into the cluster class with the most accesses; S44, subject representation is performed on the class vector of each subject to obtain a keyword list of each subject cluster, including: S441, importance degree calculation is performed on the class vector of each subject to obtain an importance degree score of each subject; The importance degree score expression is: where F c is the importance score of topic cluster c, w represents a candidate word, C represents the set of all topic clusters, |c'∈C:w∈c'| represents the number of other topic clusters containing the word w, and |C| represents the number of topic clusters. S442, according to the importance degree score of each subject, the top N words are selected as the subject keywords of each subject cluster to obtain a keyword list of each subject cluster; A subject test module is configured to process new demand information of a user according to the keyword list of each subject cluster to obtain a subject test result, including: S51, a large language model is used to analyze the new demand information of the user to obtain an intent keyword vector of the user; S52, a similarity calculation model is used to calculate the similarity between the keyword list of each subject cluster and the intent keyword vector of the user to obtain a similarity value; The similarity calculation model expression is: where S(A, B) is a similarity value, A is a user's intent keyword vector, B is a keyword list of each topic cluster, A i and B i are values of corresponding vectors in the i-th dimension; S53, the similarity value is compared with a preset threshold value, when the similarity value is lower than the preset threshold value, the subject test result is that a new subject is found, and the new subject keyword is stored; when the similarity value is greater than the preset threshold value, the subject test result is that no new subject is found.

5. An intelligent processing device for source discovery, characterized in that, The device includes: a memory storing executable program codes; a processor coupled with the memory; The processor invokes the executable program codes stored in the memory to execute the intelligent processing method for source discovery according to any one of claims 1-3.

6. A computer storable medium, characterized by The computer storage medium stores computer instructions, which are invoked to execute the intelligent processing method for source discovery according to any one of claims 1-3.

Images